Embrace the mess if you want to do better knowledge work

I’ve been deeply immersed in the recent profusion of new ideas, apps, and initiatives in the knowledge work  space. I’ve been working to make sense of a host of terms and concepts and discern their relevance to my own work. A partial list of those concepts (with some pointers to good entry points) includes:

There’s also a recent uptick in applications and services offering a path to implementing these ideas. These new apps are also fighting for mindshare with a set of existing apps. A very partial list (basically those apps I have experimented with or use with some regularity) includes:

Software developers, entrepreneurs, and evangelists of all stripes have to make the spine of their application, service, or approach clear and compelling. You’ve got to be a believer if you’re going to put in the time and effort to build something new. Early adopters also tend to be believers.

I tend to be an early adopter in many settings. But I’m also an old fart, so I’ve been jilted many times. Scar tissue provides perspective.

One of the drivers behind this surge in new work is the inexorable shift to knowledge work. Knowledge work is different from so much of the work that organizations have learned to manage and control. No matter what the bean-counters and compliance managers would like, knowledge work is inherently messy.

There’s a distinction in the world of early AI research that is useful in this context. The early world of AI research broke into two camps on the nature of intelligence; the “neats” and the “scruffies.” I took a look at this argument a number of years back in an earlier blog post on the realm of knowledge work–Knowledge management: the latest battle between the neats and the scruffies.

I once aspired to being a “neat”–business school is fundamentally targeted towards those who cherish and desire to impose order. The reality, linked no doubt to my ADD, is that I will always be a “scruffie.”

Fortunately, the world now aligns more closely with my “disorder.” You can’t get to “neat” without traveling through “scruffie.”

The challenge is that nearly all of the evangelizing and advice about new ideas is packaged as though that journey is over or, at least, easy. We get a “neat” picture of the destination. The journey is left as an exercise for the reader.

Even if the developers and early adopters acknowledge that there is a journey to be made, they gloss over the messy parts. If they share any details of the necessary hero’s journey, they offer just enough of the ugly parts to burnish their story. Preparing you for what you will encounter just gets in the way of the next chapters of their stories.

The absolutely essential step if you want to travel the path to being more effective as a knowledge worker is to accept that you have to walk the path for yourself. Seeking out more honest accounts of those who have traveled before you can help. Finding guides who can walk with you and help you avoid the quicksand and tar pits is even better.

But you’re still going to get dirty.

Choosing to say yes inside the organizational hairball

MacKenzie, Gordon. 1997. Orbiting the Giant Hairball : A Corporate Fool’s Guide to Surviving With Grace. US: Viking Pr.

I first encountered Gordon MacKenzie’s Orbiting the Giant Hairball during my time growing Diamond Technology Partners. I decided to revisit it recently. I’ve picked it up from time to time over the intervening 23 years (we’ll return to that notion in a bit). It wasn’t something that neatly fit my typical reading patterns. But it deals with a problem that has been central to my work. How do you reconcile creativity and organizations?

Maybe it was the image of large organizations as “hairballs.” MacKenzie’s point about larger organizations was that they accrete rules and standard operating procedures over time. Mostly, that is a good and necessary thing. You don’t want every worker on the line to decide which tires to attach to the axles today. But if you spend much time inside organizations, you learn that rules and rule-making can impose their own perverse logic. How many items in the standard operating procedures or the employee manual reduce to “never, ever, let Joe do that again?” Organizations are suspicious of creativity and people are full of it. A volatile combination.

MacKenzie worked inside Hallmark; he was a creative leader in an organization that depended on its creativity. Even there, the forces working to suppress creativity were powerful. MacKenzie puts it succinctly, ” it is not the business of authority figures to validate genius, because genius threatens authority.” His solution as he gained authority within Hallmark was appropriately creative. Genius is hard to discern before the fact. MacKenzie’s answer was to default to saying “yes.” If someone or some team approached MacKenzie with a proposal to try something, he gave them permission. He didn’t ask or worry whether he had any authority to do so. This mildly subversive act was often just enough to shake loose resources or convince others to let an experiment proceed. I adopted this strategy to good effect.

Fundamentally, Orbiting the Giant Hairball is about this tension between order and chaos. The proponents of order are loud and powerful. But the world depends on the balance between yin and yang. There are more than enough forces working against creativity in organizations; the creative spirit needs champions too.

One final observation. I pointed out at the start that I had elected to reread this provocative little book, Lately, I’ve been encountering a lot of well-argued, and certainly well-intentioned, advice about the importance of improving efficiency in processing new information. We’re assaulted with new inputs. But efficiency is the wrong metric. Managing inputs effectively should dominate efficiency. But effectiveness is much harder to assess. Many items on my reading list are worth no more than a single reading; some turn out to not be worth that. But the other tail of the distribution also matters. Certain books yield new insights when you revisit them with further experience. Don’t let a quest for temporary efficiencies block access to that additional value.

Building knowledge work toolsets

Swiss Army KnifeI first encountered the notion of “affordances” in Don Norman’s excellent The Psychology of Everyday Things (which was renamed The Design of Everyday Things a couple of years later).  From the world of design, an “affordance” is a characteristic of an object that offers clues about how to use or interact with that object; the design of a chair tells us it is something for sitting on, a pitcher is for holding and pouring things. Affordances can be difficult to design in a world of more complex physical objects; they can be nigh on impossible in the design of software tools and applications. One of the things that made early computer systems so confounding was that they offered no affordances to latch onto. Graphical user interfaces were a huge step forward in making software user friendly (or at least not overtly user hostile).

Affordances and the design thinking that goes into crafting good ones are rooted in the physical world; dexterity, reach, visual acuity, strength, all factor in to design. This is the world of ergonomics or human factors. Transferring that knowledge to the abstract world of software and the objects in our environment with significant software elements is no simple task. You need only think of the remote control for your DVR and set top box or the user interface of your bank’s ATM to appreciate how difficult this design work can be.

One failure (limitation?) of modern application design is affordances that run out of steam before users grasp the full capabilities of an application. If all chairs looked like a Shaker chair, how would we ever figure out what to do with a BarcaLounger; or the pilot’s cockpit seat in an F-16? The menus and window layouts of your typical desktop application (Word, Excel, Powerpoint, Outlook) get everyone up and running quickly, but they do nothing to hint at, much less reveal, any deeper capabilities or opportunities. So, for most users, most of the time, those opportunities go unrecognized and the capabilities never get exercised.

What separates power users from the rest of us, then, is a willingness to dig into users manuals (RTFM as a career advancement strategy) and to experiment and play with their tools to figure out what is possible. Otherwise smart people take marketing hype about “easy and intuitive” software seriously and judge themselves deficient when good software tools are often neither. The opportunity wasted is tremendously sad.

So, what do we do to help more people get more value out of their software? My specific interests happen to be about knowledge work. What can we do to help knowledge workers push their tools farther and better? Put another way, why do we settle for knowledge workers leveraging such limited subsets of the tools they already use or avoiding tools or techniques that might unlock significant new insights and outputs?

Although I’ve been talking about affordances I don’t think the blame or the answer lies exclusively with software designers. More realistic claims from software marketers wouldn’t hurt although I’m not holding my breath. As software users, one thing we can do is invest time to suss out more of the design models in the heads of the software developers who build the tools we are seeking to leverage.

Word processing software offers examples of what I’m thinking about.

Microsoft Word is likely the dominant word processor on the planet. You can find it installed on most any desktop or laptop computer you encounter. It ushered in the world of WYSIWYG computing where one goal of the interface was to represent your work in as close to final form as possible. Implicit in that design was that the “final form” was a printed page.

A consequence of that design choice was that you now had one tool that spanned what had once been a series of discrete stages in the process of bringing an idea to the printed page. Writing, editing, design, and layout are quite different cognitive activities. In a pre-WYSIWYG world, each of these steps had its own set of professionals, its own set of standards and practices, and its own set of tools. With Word you now have a single tool that can serve at all stages (at least for 80% of cases).

Whether that is a good thing is another question. Having one tool makes it more difficult to see that the activities in each stage are quite different. When you’re working out the structure of an argument, there’s little point to be worrying about what typeface and font size works best for a section heading. But a single tool blurs that distinction and boundary. You can be enticed into playing with those problems or thinking they are important to deal with now while your fundamental argument or storyline is still a mess.

A tool that is suitable across all these steps may be valuable within an organization. But at the price of obscuring the differences between different process stages. You could manage that problem if you made an effort to make the different stages of the process more explicit. But now the organization and its people need to work at cross purposes to the tools. You have a single tool obscuring the differences in the process while you try to highlight those same differences as you manage the development and evolution of any particular deliverable.

Scrivener is a popular word processing tool, especially among authors dealing with longer form writing projects. Its user experience embodies a very different model of the writing process than Microsoft Word. Scrivener makes the various stages of writing–drafting, editing, design, layout–visible and identifiable in its user interface. In particular, WYSIWYG is a secondary or even tertiary goal in the user experience.

Moreover, Scrivener was designed in a time when the printed page is only one of many target forms for final deliverables. It also supports multiple formats for electronic books, PDF files, and the web. To that end, the design of Scrivener separates the activities for structuring a draft output from formatting it for final output.

For users accustomed to the WYSIWYG model embodied in Microsoft Word this is a source of frequent confusion. Learning how to invoke specific functions and features in Scrivener isn’t terribly helpful until new users grasp the fundamentally different process model embedded in the design of the application.

Software marketers don’t like to acknowledge that software users require multiple individual software tools to handle the complexities of modern knowledge work.It is not their job to figure out how to fit multiple tools together to get work done.

This is an element of your job description that hasn’t been acknowledged or addressed in the average organization. You will likely be issued a starter set of basic tools. But don’t expect useful guidance on how to use them in concert to accomplish the work expected of you. It is yours if you hope to be an effective knowledge worker.

Review – Intertwingled: Information Changes Everything

Image of Intertwingled CoverMorville, Peter. 2014. Intertwingled: Information Changes Everything. US: Semantic Studios.

Intertwingled is not a practical book. It is not intended to be. Unfortunately, that will discourage those who could most benefit from its message and perspective. It’s an extended reflection by its author, Peter Morville, on the deep challenges of making information more easily accessible and impactful within organizations. Morville has written several of the essential books in the world of information design and architecture (Information Architecture,  Ambient Findability). Intertwingled is a more reflective exercise; an attempt to pick a broader vantage point on understanding both the technological and organizational environments we inhabit. Morville opens with an observation from Ted Nelson:

People keep pretending they can make things hierarchical, categorizable, and sequential when they can’t. Everything is deeply intertwingled.  – Theodor Holm Nelson

I’ve never been a huge fan of the term “intertwingled” – probably because I am generally suspicious of coining “cute” new terms. Ted Nelson, on the other hand, seems drawn to the practice. Nelson’s books, Computer Lib/Dream Machines and Literary Machines, were hugely influential in luring me into the field and Morville seems to have been similarly enticed.

I don’t think Morville makes me any more comfortable with the term, but the book and his arguments and observations are still  worth the time.

It’s beyond cliche that we live in an information economy. It’s easy to be distracted by the shiny stuff– apps, devices,  platforms, technology. Morville’s focus is on the peculiar notion of information. Information is a surprisingly slippery term and we are all well served by efforts such as Intertwingled that make us think about that slipperiness. Morville observes that

Access to massive amounts of conflicting information from myriad sources creates filter failure. We don’t know what to believe. So we fall back on simple ways of knowing. We trust experts and those in authority. We follow doctor’s orders. Or we reject expertise completely.

Lately, it seems that rejection has become the go to strategy in many settings. This is not, in fact, a new problem. Morville, for example, offers the following observation from computer scientist Calvin Mooers published in 1959:

Many people may not want information, and they will avoid using a system precisely because it gives them information. Having information is painful and troublesome. We have all experienced this. If you have information, you must first read it, which is not always easy. You must then try to understand it. Understanding the information may show that your work was wrong, or may show that your work was needless. Thus not having and not using information can often lead to less trouble and pain.

Morville’s goal with Intertwingled is to make you think. He succeeds admirably. Enough so that this will be a book I expect to return to.

Competitive strategy and market failures

Mike Porter's 5-forces modelI finished my MBA 40 years ago. Courtesy of the current pandemic, I won’t be having the reunion I was looking forward to. But, I have been thinking back to lessons from those days.

One of the hottest courses my second year was “Industry and Competitive Analysis.” Professor Mike Porter was putting the final touches on what would become Competitive Strategy : Techniques for Analyzing Industries and Competitors and we were the beta testers for what would become the bible of corporate strategy. Because Harvard is committed to the case method, there wasn’t much explicit discussion of theory as theory. You absorbed the theoretical essentials in the course of applying the tools and techniques to the case examples.

Fast forward several years. I’ve left the world of consulting and come back to school one more time. I’m now writing cases instead of reading them. And I’m soaking up formal theory in multiple subjects. One of the courses I take is a course in Industrial Economics from Richard Caves. Industrial Economics is the study of markets and competition from an economist’s perspective. Caves was Porter’s thesis advisor when Porter was getting his Ph.D.

When economists study markets and competition, their driving question is “why do markets fail?” Economists believe in markets and their theories presume perfect competition. Deviations from perfect competition are anomalies to be understood and explained. What gets in the way of the theoretical ideal of Adam Smith where sellers compete on price and features to meet the needs of well-informed buyers. The catalog of things that cause markets to be less than perfectly competitive is long: monopoly power, monopsony power, barriers to entry, barriers to exit, price fixing, collusion, are only a partial list. Economists then seek potential solutions to prevent or eliminate these sources of market failure.

Midway through the course it dawns on me. Porter’s genius in competitive strategy is to recognize that an economist’s market failure is a CEO’s strategic coup. Your goal as a CEO is pick your markets and shape your organization’s behaviors to maximize the probabilities of market failures that work in your favor. I’ve found that to be a very powerful lens for understanding how business strategy plays out in practice.

Learning to See-Improving Knowledge Work Capabilities

My wife is a photographer. Quite a good one, in fact. One sure way to annoy her is to ask what kind of camera she uses after admiring one of her photos.. It’s her eye, not the camera, that recognizes the perfect shot. The tool may well be the least important element in the mix.

My own photography has gotten better courtesy of time spent in apprentice mode by her side. Photography is also an example of a knowledge work capability that can shed light on performance improvement in a knowledge context. The primary performance metric is whether you can capture the image you envision. Secondary metrics might include meeting time, budget, and other constraints on the image. In some settings, you may also need to be able to articulate the logic for why the image you eventually capture meets the criteria set.

If your goal, for example, is to capture a simple selfie to demonstrate that you were there at Mt. Rushmore, anything with both you and the mountain in frame and in focus will suffice. As you goals evolve, you also acquire new concepts and vocabulary; composition, depth of field, light conditions, focal length. exposure.

Meeting those goals may lead you to exploring and adopting new tools. A better camera might well enable you to capture images that weren’t possible with starter tools. But the functions and features of more sophisticated tools might just as well not exist if you don’t have the corresponding concepts to work with.

These concepts and the tools all need to be in service to creating the images you imagine. You don’t learn them in theory or in isolation. You learn them by doing the work and getting feedback. Over time, you also learn to give yourself better feedback.

Ira Glass has an excellent series of short videos on storytelling that fit here and fit knowledge work in general. The whole series is worth your time and attention–Ira Glass on Storytelling – This American Life. The nut graf, however, is something to keep close at hand as you work at your craft:

Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.

There is craft behind all knowledge work. You get better at craft work by being intentional about getting better. And by accepting that craft lies in the mix of tools, techniques, practices, mentors, and peers. It’s a mistake to remain wed to the first tool you pick up, It’s equally a mistake to confuse changing tools with improving your craft.

Working in the boundaries – making the pieces fit together

“Specialization is for insects” Robert A. Heinlein

Between can be a difficult location. Cast or crew. Analog or digital. Quant or sales. Worker bee or management. Head or heart.

If you choose to reject the standard either/or logic and opt to stake out territory at a boundary or junction, the most likely result is ridicule and rejection. No matter how continuous reality may be, we insist on carving it up.

When I first moved out of the tech crew and into the stage manager’s shoes, I thought I was taking a small step. What I was doing was choosing to live in a DMZ. I became the referee when the lighting director and the choreographer both wanted the stage at the same time.

There was a time in one especially difficult production when I had had it with both the Producer and the Director. I can’t even remember what the argument was about. I left a note in the office that I was going back to school and they could finish the show without me.

A friend in the cast found me ten hours later in the one place I figured no one would think to look. I was in the third subbasement of the university library. Probably the first time I had set foot there since September and it was now March. I was coaxed into going back to the production, after apologies from the Producer and Director.

This was the earliest incident I can recall working to reconcile conflicting perspectives and demands to pull off a vision. Carpenters wanted to build, electricians wanted to light, dancers wanted to stretch, and actors wanted to run lines. All of it had to come together for the curtain to go up on opening night.

Over time, my interests have migrated to how to balance technology opportunities and organizational limits. But the reluctance of most to look outside the boundaries of their playgrounds remains. My head likes the challenge of figuring out the differing sets of details and their interactions.

Most players pick a side. They choose to belong to one organizational clan or the other. They commit to being management or to being a machine learning expert. Fewer choose to work as simultaneous translators; to learning the language and theories of new technology and of deep strategy.

Our systems are built around slotting people into speciality roles. What often gets lost is that someone has to work at being the glue fitting key elements together.

Getting Outside Your Head – Managing the Mess

Managers do not solve problems, they manage messes
– Russell Ackoff

I’ve long been a fan of the late Russell Ackoff. This was one of his observations that continues to stick with me. As much as we like to think of the world as a set of discrete problems to be solved, reality insists on  being interconnected and messy. As I seek to understand and improve my own knowledge work practices, this is one of the pieces of wisdom I try to keep in mind.

One of the sources of mess that has been on my mind lately is scale. Things that seem obvious and simple always get messy as they get bigger. For example, I’ve just wrapped up a course on how to do requirements analysis. We run it as a field based course which means we work with local organizations to tackle a real problem. But it has to be a small enough problem that we can fit it into a semester’s worth of work.

I can require student teams to prepare the same work products that would be expected of them on the job but the problems aren’t big enough to make the need for some of those work products to be evident. Students do the work I ask of them, but they don’t really believe me when I assert that all of those products are relevant and important.

I’m probably irretrievably tainted by my early days in public accounting when I hung out with auditors. Now, this was so long ago that spreadsheets were still actual sheets of paper with rows and columns preprinted on them. Audits generated piles and piles of paper. The most natural thing in the world for an auditor faced with managing a stack of spreadsheets was to prepare another spreadsheet to serve as an index into the stack.

The physical scale of stacks of paper made this solution fairly obvious. Turn those stacks into bits, however, and the need to manage that scale problem disappears or, at least, fades into the background. If you can no longer see the mess, it won’t occur to you that it needs to be managed.

On a project team, there is often enough friction in dealing with multiple team members trying to coordinate their work that you can impose some level of control over the growing collection of digital materials being produced. But the usual pressures to “get to done” work against efforts to manage the mess. Persuading team members to give some thought to what they name that Excel file gets lost in the rush to work on what’s inside the file.

A sufficiently OCD project manager might prevail over a project team. Certain regulated environments can force the mess to be managed. But for individual knowledge workers, there are few incentives to deal with, or even recognize, the problems of managing the mess that is a digital work environment.

Getting work outside of your head is only a first step. Doing so gives you the capacity to take on problems that are too big to fit inside your head. Your reward for increasing your capacity is a set of bigger messes to manage.

Getting Outside Your Head

I’ve been on a quest over the past year or so to understand the importance of getting outside of your head if you want to be more effective as a knowledge worker. The inciting incident for this quest was reading How to Take Smart Notes by Sonke Ahrens (my review is at Unexpected Aha Moments – Review – How to Take Smart Notes). I think I’m past the “refusal of the call” but I don’t know that there is a mentor to be found, although there do seem to be many others walking similar paths. Ahrens tells a story about Nobel physicist Richard Feynman that I traced back to James Gleick’s biography of Feynman (Genius: The Life and Science of Richard Feynman). Gleick tells it this way:

[Feynman] began dating his scientific notes as he worked, something he had never done before. Weiner once remarked casually that his new parton [In particle physics, the parton model is a model of hadrons] notes represented “a record of the day-to-day work,” and Feynman reacted sharply. “I actually did the work on the paper,” he said. “Well,” Weiner said, “the work was done in your head, but the record of it is still here.” “No, it’s not a record, not really. It’s working. You have to work on paper, and this is the paper. Okay?”

This is what my math teachers would label a “non-trivial” insight. However, if they made that point when I was studying math, it sailed right past me. Sure, you could sometimes salvage credit on a problem set by “showing your work” but it never occurred to me that “showing” and “doing” your work was the same thing. I always felt that the work was supposed to be going on inside my head, that the goal was to get everything inside my head before exam time rolled around. Certainly the testing and evaluation systems reinforced the notion that you were supposed to keep the important stuff in your head; storing it elsewhere was likely to land you in serious trouble if you got caught referring to that external storage during the exam.

Some of this is the problem of “toy problems.” In teaching settings, you need to work with problems that can fit into class sessions and semester-long projects. With most of these you can get away with lazy practices; you can manage it all in your head. If you’re lucky, the course designer may try to force you to follow good practices above and beyond simply finding the “right answer.” As a student, you’re still likely to miss the point of learning the supporting practices. [As an aside, this is now something I’m working on improving in my course design and delivery]

Once you start to look for it, you do see that smart people have been offering good advice about how to deal with the limitations of your unaided memory and brain. Think of Anne Lamott’s advice to write “shitty first drafts,”  Peter Elbow’s practice of “freewriting,”  Tony Buzan’s advocacy of “mind mapping,” or John McPhee’s ruminations on “Structure.” All of these are the kinds of techniques and practices that can make us more effective at creating quality knowledge work artifacts. But it isn’t clear that we encounter this advice as early or effectively as we should.

If we do stumble across this category of advice and fold it into our work practice, we can gain a meaningful edge. We’ve taken elements of the work out of our heads and into our extended work environment. We’ve increased the range and complexity of material we can now draw on to create better deliverables.

I’m in the midst of working this out for myself. I actually think that this is something that each knowledge worker is going to have to design for themself. I’m suspicious of claims that someone’s new tool or application contains the secret answer. Right now, I’m investigating various sources with an eye toward identifying design principles and ideas worth extracting or reverse engineering.

Some of the more interesting trains of thought include:

Task Zero – Well Begun is Half Done

The late Peter Drucker continues to be a source of insight and inspiration for me.  In 1999, he published “Knowledge-Worker Productivity; The Biggest Challenge” in the California Management Review. I’ve written about it before (Knowledge work and productivity). I want to explore the following observation:

The crucial question in knowledge-worker productivity is: What is the task? It is also the one most at odds with manual-worker productivity. In manual work, the key question is always: How should the work be done? In manual work, the task is always given.

I’ve started to think of this question as “Task Zero.”

The invention of zero was one of the great advances in mathematics; perhaps we should respect that power. One of the curious things you learn as a computer programmer is to start counting from zero rather than one. Certain things become easier when you do.

One of the reasons I’m drawn to starting at zero is that it frees up my thinking. If you think of yourself as starting from zero on a map or a coordinate system, you are free to move in any direction. Starting at one immediately commits you to a direction.

That’s tempting because a direction gets you moving. There’s a great observation from Cory Doctorow that reflects this:

Start at the beginning,” he said. “Move one step in the direction of your goal. Remember that you can change direction to maneuver around obstacles. You don’t need a plan, you need a vector.

― Cory Doctorow, Homeland

We like movement; it feels like progress. But they’re not the same thing. Take a closer look at Doctorow’s quote; he’s advocating movement in “the direction of your goal.”

Clarifying the goal is Task Zero. It is the “what problem are we trying to solve” question that grizzled consultants pose to annoy eager young MBAs and impatient clients.

It’s worth giving this task an identity separate from the tasks that follow. It’s like that pregnant moment at the end of a countdown to launch just before movement begins.