Andy Matuschak
patreon
Liquid olives and iPhones; problem-solving and problem-finding; The Uncertainty Mindset
🕑 Added 2020-11-15 04:18:37 +0000 UTCGrowing up years ago in the midwest, my perception of a fancy restaurant was awfully simple. Firstly, fancy restaurants have fancy waiters who make you feel uncomfortable for using the wrong utensil. And secondly, fancy restaurants use fancier ingredients: the menu might include ribeye and lobster instead of hamburgers and barbecue. But then I moved to California, and—to make a long story short—a bowl of tapioca made me weep. I learned that great restaurants are rarefied institutions of extraordinary innovation and artistry.
Around this time, I was an intern at Apple by day, and I was devouring high-end cooking texts by night. I felt a great deal of connection between the two worlds. They shared an essential shokunin-style craftsmanship important to me at that stage of my career: yes, invention matters, but first we must render each pixel perfectly, cut the brunoise precisely. But—and this was fairly mysterious to me at the time—both worlds also housed practitioners who thought the unthinkable, who pushed their fields forward. How do those two worlds co-exist? How do these teams operate?
At the world’s best restaurants, it’s not enough to deliver a great service every night. It’s an incredibly competitive sphere. Sitting still means falling behind. Apple’s situation is quite similar: the iPhone might be one of the most astonishing consumer products ever designed, but it’s not enough to simply manufacture the same device each day. Likewise, the world’s best restaurants spend lavishly on dedicated R&D operations which develop new cooking methods, source and understand new ingredients, invent surprising process improvements, and evolve the restaurants’ unique culinary styles.
I’m quite envious of the years spent Vaughn Tan in preparation for his new book, The Uncertainty Mindset. He embedded himself in the culinary labs of some of the best restaurant groups in the world, studying how they opened new locations, solved problems, developed new dishes—and convened a major conference in the mud beneath a circus tent. From his observations, Vaughn distilled several shared but unusual traits in these teams’ organizational practices. I was struck, reading the book, by how similar these practices were to the best of my experiences at Apple. I’ll share some of those stories and how they relate to Vaughn’s stories of culinary R&D teams. As excited as I am by this book, I’ve noticed that its practices seem to align less well with my experiences as they’ve shifted increasingly towards the “R” side of R&D. I'll try to characterize that evolution too.
Continuously-negotiated roles
In startups, roles are fluid. Everyone wears many hats: what’s important isn’t your job description but the problems which need to be solved. As companies get larger, though, abstraction becomes important. It’s awfully hard to have thousands of employees without clear job titles and areas of responsibility. Somehow, though, at least in the early days of iOS, Apple managed this. Roles were, as Vaughn puts it, “modular and provisional.” We fluidly assembled for each new problem, composed to provide the skills needed for that particular project.
My title was “software engineer,” but that’s not a useful descriptor for assembling a team in practice. It’s both too broad and too narrow. First: what kind of software engineer? Initially my experience was mostly in interfaces and API design, but I’d eventually develop skills in architecture and systems engineering which would get me pulled into a completely different set of projects. More importantly, though, my role evolved to include meaningful design and product management work, partially in response to the demands of projects, and partially in response to my interest. This was great for me personally—it kept me interested—but also great for the organization, since projects like parallax required interdisciplinary teams with unusual combinations of skill sets.
Vaughn argues that this kind of fluidly evolving role definition is essential for innovation in teams because of the activity’s inherent uncertainty. Consider el Bulli’s famous liquid olive: an ovoid appearing to be an olive arrives on a spoon before you, but when you pop it into your mouth, you discover that it is in fact a paper-thin membrane surrounding an explosive olive juice. To make one of these spheres, chefs drip juice into a bath of sodium alginate which reacts to form the delicate skin. Imagine that you’re hiring for a culinary R&D team, and you want to hire the chefs who would invent this technique. What title do you put on the job req? What qualifications? Material science? Chemical engineering? Pastry cookery? This prompt assumes, of course, that you’re even able to specify “invent the liquid olive” as the problem statement—which you can’t. The activities of such a team are inherently uncertain. Not in that they’re risky: risk is something you can understand, plan for, manage. Instead, they’re just laden in unknown unknowns. Your team members’ roles must deal with this inherent uncertainty, which itself shifts over time.
Demo-driven development
Of course, not every role in a kitchen is so nebulous. You will need an army of prep cooks who are simply prep cooks. And likewise, the vast majority of Apple’s software engineers were much more straightforwardly software engineers. Apple needs most of their software engineers to be happy just being software engineers. The process which separated those more unusual engineers at Apple sounds a great deal like the one Vaughn describes culinary teams using.
My journey away from being a straightforward software engineer relied on the process of “demo-driven development” which my mentor Ken Kocienda described in Creative Selection (recommended!). Our work revolved around a regular cadence of demonstrations. When I was working on the swipe-left-to-right gesture for navigating backwards on iOS, I demoed my work every day or two to designers, engineers, and executives. Craig Federighi, head of Apple's software engineering, would swing by my office, play with the gesture, notice what was working and not working, make suggestions. All this helped make the gesture itself good. But it also served an important organizational purpose: these projects and demos functioned as ongoing public tests. Everyone got to see how I handled (or failed to handle) the various problems which came up. When the next project came around, this would help leadership put together better teams. Because my peers were also present for many of these demos, they could see for themselves where I was doing well or poorly, which would both inform their own work and reduce feelings of resentment or confusion when future project assignments were made.
If you’re going to have a team with continuously negotiated roles, you need a context for that continuous negotiation. These demos unified the “tests” with the real work. Vaughn describes culinary R&D teams following a similar path, testing team members by asking them to solve a wide variety of problems in different operational configurations. Each day ends in a group tasting, just as our days often ended in a group demo.
This project-driven approach also creates a fertile training ground for new employees. The constant feedback is a great way for new team members to learn the house “style.” Different mentorship collaborations can be tried and discarded. This makes training feel a bit haphazard—and sometimes quite stressful—but it’s difficult to create consistent formal training programs when pursuing open-ended goals, for the same reason that it’s difficult to precisely specify anyone’s job description on such teams.
Desperation projects
Another common practice for culinary R&D teams overlapped a great deal with my experience at Apple: the strategic use of desperation. Excerpting from Vaughn:
Every point in the pattern looked like this: commit to a project beyond the team’s ability, freak out individually and collectively, work like mad, somehow pull victory from the jaws of defeat, breathe a massive sigh of relief. I ran into people from each of the teams periodically. When they were in the middle of one of these projects, they seemed desperate: emotionally and psychologically exhausted, worried (slightly terrified was often a better description) that things wouldn’t work out or (worse) would be disastrous. The teams seemed unable to learn from their mistakes and avoid these desperation projects. In fact, they kept committing to doing them. They would heave a sigh of relief that they’d scraped by and then—the next month or the next year—find something else to do that would make them desperate again.
Eventually, I came to understand that they put themselves into these terrible situations as a way to force themselves to innovate, that the desperation was productive, not destructive. It was desperation, but by design.
In late 2012, I landed after a long flight and shared in a ceremonial moment as three hundred of us simultaneously disabled airplane mode. Immediately, my phone was flooded in notifications: Scott Forstall (the executive primarily responsible for the creation and first few iterations of the iPhone) had been fired. Jony Ive, formerly responsible only for hardware design, would take on software design as well. In late November, a truly desperate project was declared. Jony wanted to redesign the entire operating system and every app we shipped. We’d be putting it in developers’ hands in early June. This would have been an insane proposition even if we were only planning to change the OS’s outermost skin, but the initial ambitions, at least, went much beyond that.
Just for flavor, one project I pursued was based on the observation that white things in the physical world are never actually white. They always take on some character of their surrounding, shifting with perspective and external lighting conditions. Perhaps we could develop a special material—“digital white”—which would embody a similar subtle dynamism. My prototypes used the gyroscope to create a subtle living shimmer, almost like a blind debossing on a book cover. The idea was to use this to indicate interactive elements on the screen, since we were stripping away the skeuomorphic trappings of buttons. We ended up tossing this idea: it was too heavy in power consumption—and in Jony’s words, it was “a bit… carnival.”
But still, that was one of at least half a dozen similar major, system-wide projects I led in those seven months. My routine was simple: each day, I woke up, rolled over, grabbed my laptop, and didn’t put it down until I went to sleep, every day for seven months. We worked from deep desperation, but as Vaughn describes, it was absolutely one of the most exhilarating and dynamic periods of my life.
Problem-solving and problem-finding
While I read Vaughn’s book, I felt deep connection between my experiences at Apple and his stories of the culinary teams. But the connection was much weaker to my time at Khan Academy, and weaker still to my more recent research. I can’t characterize the difference completely, but I think it may rest on a distinction between innovation and invention, between problem-solving and problem-finding.
I joined Khan Academy with my friend May-Li Khoe, who had been heavily involved in innovative work at Apple. We started an R&D group together, hoping to bring this kind of exploratory work to Khan Academy. Its charter meandered substantially throughout the five years I was there, and in those meanderings I feel I can trace some limitations of the approaches Vaughn describes.
For some of our projects (like scaling open-ended problems online), our team acted almost like an in-house consultancy for the rest of the organization, creating solutions to some problem which could be at least vaguely defined. This was similar in many ways to my time at Apple: yes, we were creating interesting new interfaces, but in response to some sort of exogenous problem statement. In these situations, the problem statement is basically never fully defined—part of the project is negotiating what the problem to be solved really is, “ripping the brief” and so on. Part of the work between projects is lobbying for future problems-to-be-solved to executives. But the problem statement’s presence anchors the project and makes its edges finite. Regular demos make sense because you can evaluate them in the context of the evolving problem statement. Desperation can apply useful pressure because you can make pragmatic tradeoffs, perhaps sacrificing some bold artistry in pursuit of some practical solution.
But some of our projects (like Cantor) were more about problem-finding than problem-solving. There was no clear problem statement. There wasn’t really a client or a customer. In fact, the most important work to be done was in identifying a powerful problem statement—and for many of these efforts, we never did! The interesting parts of my work today are mostly of this kind. Yes, there are specific problems to be solved, many of which I’ve discussed here, but the most powerful forces in my work are about problem finding. This work felt much less connected to Vaughn’s descriptions than my other experiences had.
Why is it that knowledge workers seem so fundamentally unserious about improving their fundamental skills, compared to athletes and musicians? How might we create environments which do the job of a book, but which participates more actively in the impact the book hopes to have? These are already very unusual problems, and just posing them is a significant contribution. But these big-picture problem statements shatter fractally into a hundred sub-problems, and most of the progress in my work comes from identifying and improving articulations these sub-problems. Actually solving those problems is important, of course, but that’s downstream.
When problem-finding, theories and concepts are often the locus of activity. Rapidly iterative demos and prototypes become much harder to produce—and, at this stage, much less relevant. Desperation becomes, at least in my experience, a much less helpful force. When it’s not yet possible to sharply articulate a clear problem to be solved, pouring jet fuel on the fire just produces a haphazard eruption. Long walks and hours-long lunch discussions are the order of the day.
I confess I don’t understand this distinction very well. My experience suggests that a certain amount of “demoing” and a certain amount of desperation are actually quite helpful in research. But I notice that most stories Vaughn describes are of a culinary R&D team reacting to an externally-defined problem (though perhaps a vague one): a palate cleanser is needed on this tasting menu; these cannelloni are produced inefficiently; this restaurant needs help opening; and so on. There are some glimpses of problem-finding, like the liquid olives from el Bulli described above, but these aren’t documented as clearly as the others, and it’s not clear that the stories follow the same principles quite so sharply. After all, as I understand it was Ferran Adria himself (the head chef) who developed that technique, not some R&D team in his employ.
Is this the distinction between invention and innovation? Research and applied research? Academia and industrial R&D? The lines are fuzzy, and I don’t claim to understand them. Vaughn’s book is the most insightful treatment I’ve read of industrial R&D organizational practices—and now I’m hungry for more, focused more on the “R” side of the equation than the “D”!
My thanks to James Cham, both for recommending this book and for nudging me to compare its ideas to my own experiences.
Comments
Andy Matuschak
I haven't seen Gary's work, but it seems quite relevant. I've queued up "Seeing What Others Don't."
Your examples about how the mixed outcomes for the “desperate situation” technique to create innovation has mirrors with my own experience at hackathons / certain past projects. The Khan academy open ended scenario felt very relatable in the context of my being on a data visualization team that is occasionally builds tools in forms that the partner team hasn’t explicitly asked for (because it’s a new UI paradigm), so a big part of the project ends up being problem definition. Have you seen Gary Klein’s work on creativity? “Seeing what others don’t” was a neat read, he went through 100 odd examples of innovation and tried to apply a taxonomy to them (creative desperation, in some cases induced by being in an actual life of death situation, was a subcategory)
Andy Matuschak
I've really enjoyed rapid prototyping cycles like the ones your describe! Great surprises and energy.
Andrew Sutherland
I'm reminded of one of my most favorite times at Quizlet, when we were building Quizlet Live. We wanted to build a game, and after trying a few random things we found one we all (five of us) wanted to play another round of. We started scheduling weekly visits to schools around the Bay Area to test our prototype with students in person. After each visit, we'd go to a nearby restaurant for lunch and compare notes on what we saw the kids doing, what would make the learning stick better, what the kids' energy was as they played. When we got back to the office, we'd map out the next week's worth of changes and tests (i.e. let's try the kids being on teams), and we'd get them all done because we had a clear deadline of our next visit already scheduled. We kept scheduling school visits until we felt the feeling in the room created by the software was spot-on. The roles on the team were very fluid, like you described. I loved that high-energy development style, and the product we shipped was great -- it was our most successful product launch to date. It's a shame this researchy demo-driven development style isn't used more frequently. > When problem-finding, theories and concepts are often the locus of activity. Rapidly iterative demos and prototypes become much harder to produce—and, at this stage, much less relevant... My experience suggests that a certain amount of “demoing” and a certain amount of desperation are actually quite helpful in research Yes, I'd agree. I think it's worth trying to force yourself to create demos more often than is comfortable. Even if some of them will make no sense. I'm sure the patrons here would still enjoy it. When we were building Quizlet Live we had a lot of theories about how team assignment should work (e.g. kids type in a table number?) and I thought the simplest solution (random teams) would be terrible. But when we tested it, it was actually way more delightful and surprising than we all thought, since it forced kids to get out of their seats and call out to find their teams; doing that kick-started the energy of the game. We never would have discovered that now-key feature of the game. So ya, our theories about that were just totally wrong.
Andy Matuschak
Yes! Anders Ericsson wrote a lot about this. He suggested that deliberate practice was only possible in fields where progress was clearly evaluable; where we knew concrete actions to take to improve skills; where we'd developed a corpus and culture of coaching/training techniques; where feedback loops are tight; etc etc. It's interesting to ponder how to achieve some of those properties in knowledge work.
Andy Matuschak
In my experience, industrial R&D involves the same kind of technical and professional skills which would be involved in a "production" role in such an organization, plus research skills and mindsets: question-finding, literature reviewing, experiment design, and various cultural norms like expansiveness.
Thanks for the response Andy! Slightly different question: what skills do you think are needed for a job in industrial R&D?
Andy Matuschak
I really wish I did, but I don't! I don't know of any industrial R&D organization which is both hiring and about which I hear consistently positive news. This is a gloomy observation. Most of the interesting industrial R&D work I see is being done scrappily as part of some broader startup effort or as an "indie" sort of thing, like I'm doing.
Do you have any advice for looking for jobs in industrial R&D?
Late to this one but I also enjoyed it quite a lot. > Why is it that knowledge workers seem so fundamentally unserious about improving their fundamental skills, compared to athletes and musicians? I have experienced, at some level, being a knowledge worker, an athlete, and a musician, so this question particularly stood out to me. If I try to answer it for myself, it comes down to the fact that athletes (and musicians, to a lesser extent), have clear metrics for understanding their progress ("How fast can I run 100 meters?", "Can I play this piece flawlessly?", etc.). There is something exciting and deeply satisfying to me about measuring my progress by those metrics. When it comes to knowledge work, I still notice progress over time, but in a much less concrete way. Improvements in my ability to understand or communicate ideas are harder to notice as they happen. External rewards like promotions and raises or recognition from others are satisfying, but often don't feel strongly correlated with better knowledge work.
Andy Matuschak
Thanks, Jamie!
I really enjoyed this! Your essays are always excellent food for thought, and this was no exception.
Malcolm Ocean
Sounds good! I'm in the process of preparing to move across Canada by car, so while there may be a moment to connect in the next couple weeks, December is a more likely fit :)
Andy Matuschak
Thank you for these generous replies! Like you I'm super frustrated by the "standard solution." Will reach out to find time to jam on this problem together~~
Malcolm Ocean
(that idea led to me making this tweet. idk if you want to have the conversation there but I thought I'd let you know: https://twitter.com/Malcolm_Ocean/status/1328401920797663233)
Malcolm Ocean
(man Patreon's commenting feature is not ideal. maybe we want to reboot the convo, on twitter?)
Malcolm Ocean
The standard solution to this problem is to become a CEO of a company of hundreds of people, but this is such an utterly inadequate solution for at least 3 reasons: 1. requires a bunch of management skills that are not actually the problem-finding work you want to be doing 2. hard for one company to do a bunch of different only-vaguely-related things, so you end up needing to start multiple companies 3. you have to pay all of these people?!? why should that be your job? 4. due to various attractors, the company & teams tend to start trying to exist for its own sake rather than arising & dissolving with the problems they're trying to solve
Malcolm Ocean
And this also suggests that perhaps it would make sense for me to interview you as a prospective ~customer (to be clear I am not imagining that you will be paying me for this thing, I just mean customer in the sense of "person who has the problem I'm trying to solve")
Malcolm Ocean
Oh man, this is funny, I initially thought like "oh no, you have the same problem I have, we're gonna end up competing for people to implement our solutions" and then I remembered that one of the primary current problems I'm trying to solve is *generating meta-teams who form a readily tappable resource for people who have this problem of wanting to be able to get people to implement their solutions*. Which means that if I can solve that problem for me, I will also have solved it for you!
Andy Matuschak
Oddly, my answer is that I'm not even sure. High level, one framing this post suggests is that there's a sort of hierarchy of work in research: problem-finding, problem-solving, and implementing solutions. I spend more time on the third than I'd like. I have lots of ideas about problems which are worth solving, and lots of ideas about potential solutions, but because I'm also the one implementing the solutions, I move through that list very slowly! I'd love to parallelize some of the implementation and problem-solving work (and also, ideally, delegate more of it).
Andy Matuschak
I'm glad you enjoyed! I haven't: I'm deeply conditioned by the culture of secrecy, even years later. But I'm starting to feel like some of these stories can safely be told. More to come, perhaps!
Malcolm Ocean
Wow yeah this is awesome. I'd picked up a copy of Vaughn's book but hadn't really opened it yet. Feeling a lot of excitement reading this and it gets me wondering like, what are your current bottlenecks/contraints?
