Andy Matuschak
patreon
Revamping the mnemonic medium around reader control
🕑 Added 2021-08-25 02:44:13 +0000 UTCWhen you read non-fiction, you’re in the driver’s seat. You can skip to the last page and read only the conclusion. You can riffle through the pages, reading only the headings; or you can spend a week reading ten pages with extreme care. You don’t need to focus on just one text: you can compare one book’s ideas to another sitting by its side. Great non-fiction authors exert careful control over their prose, but once a book arrives in your hands, it becomes a tool in your service. You’ll use each book according to your own needs and interests.
This reader-centricity distinguishes non-fiction texts from other informational mass mediums, like videos and lectures. Those forms make it much more difficult for viewers to “drive” the experience, for instance to focus especially on the parts they find most interesting. In fact, abdication of control is part of the appeal. It’s fun to put yourself in the hands of a master explicator like 3blue1brown. I watch his videos when I don’t want to be in the driver’s seat; I want to sit back and enjoy seeing the topic through Grant’s eyes.
When the goal is enablement, though, reader centricity offers some important advantages. Authors model what their readers might already know and what they might be interested in, and then structure their texts accordingly. In most contexts, a one-size-fits-all (or even -fits-many) solution is impossible, but that’s okay: readers can collaborate with the author to mold the experience to their interests. In this way, texts enable readers in a wider range of contexts than author-centric mediums can reach. Perhaps more importantly, texts support readers in remaining relentlessly focused on their own sense of what’s meaningful.
All this said, I can now articulate a key problem for the mnemonic medium: it glues authors to the driver’s seat. Its key insight is combining spaced repetition memory prompts with narrative prose. Those author-provided memory prompts make it easy for people to remember what they read, but at the cost of sharply shifting control back to authors. Reading a mnemonic text is very unlike reading a normal text. The current interactions demand not only that you read a text in full, but that you repeatedly study—and commit to memory—whatever the author thinks is important, in whatever form the author chooses. The memory system isn’t “yours”; it’s on loan from the author, kept under glass. As Gary Wolf has pointed out to me, it’s an authoritarian medium.
Where today’s mnemonic medium succeeds and fails
Quantum Country succeeds despite these limitations because it’s a primer in a well-established technical field. Because it’s a primer, it can safely assume that most readers have little prior experience. They may be especially willing (and it may be especially appropriate) to defer to the author. Because Quantum Country is an introduction to a well-established field, there’s a set of topics it’s expected to cover. Its table of contents is partially a matter of authorial choice, but much is a reflection of general consensus. Readers who want to understand the field won’t commonly feel the need to pick and choose from these foundational concepts. And because quantum computing is a technical topic, the content of the prompts is less contingent on an author’s choice of metaphor or phrasing. It’s closer to capturing some standardized representation of physical law. So readers are more likely to be happy internalizing prompts as written, rather than wanting to rephrase them in terms which better match the way they think about the topic.
The mnemonic medium’s current design works much less well in contexts where these assumptions don’t hold. For instance, in my own mnemonic essay on How to write good prompts, many readers have substantial experience with prompt-writing, while others have never written one before. Readers’ areas of interest and levels of commitment vary widely. The essay’s topic is not well-established or standardized: I’m inventing my own abstractions, which resonate well with some readers’ experiences and poorly with others’. And because it’s a non-technical topic, the prompts are much more contingent on my authorial choices of metaphor, naming, phrasing, etc. Based on reader feedback, other authors’ similar mnemonic essays have experienced similar problems.
This isn’t just a problem with relatively informal topics, or with less-committed readers. One valuable application of the mnemonic medium is to augment important academic papers. Summer intern Ozzie Kirkby has been exploring topics in decentralized technologies, so as an experiment, I adapted the Interplanetary Filesystem (IPFS) paper into the mnemonic medium, and he logged his experience reading it. Many of my prompts worked as written, but others focused on aspects he didn’t personally care to explore. Ozzie found himself wanting to internalize much more detail in some sections, so he wrote additional prompts in his log (pretending he could add them to Orbit that way). A couple prompts felt too complex given his prior knowledge, so he split them into smaller prompts. I believe that issues like these would occur with most paper-reading experiences, since contexts and motivations vary so widely, and since papers are read much more tactically than essays.
Edit and delete buttons aren’t enough
Practically speaking, the reader feedback I get sounds like simple requests for control: can you add a button that lets me delete prompts? Can you let me edit the author’s text? I’ve been hesitant to implement these simple features because I think the mnemonic medium’s fundamental model needs re-shaping. A traditional text is a tool to be used as the reader sees fit. The reader’s in the driver’s seat. But edit and delete buttons aren’t enough to remove mnemonic medium authors from the driver’s seat. Even with those buttons, the reader would just be along for the ride, perhaps making adjustments along the way.
Part of the problem here is the medium’s positive starting assumption that the reader is expected to collect all the author’s prompts (except those they veto). This creates a school-like learning aesthetic, a sense that the author is assigning you to get X, Y, and Z out of the text. Some readers will respond to this with frustration and abandonment; others will respond by dutifully but passively studying things they don’t really care about, in practical misalignment with Orbit’s values (“helps you deepen your relationship with whatever you care about most … not for things you think you ‘should’ be engaged with … not ‘educational’ in tone”). This mismatch is really what keeps me from advancing the mnemonic medium (as it exists) only in Quantum Country-like “primer” contexts, contexts in which many readers don’t mind abdicating control: I want to promote a more active, less dutiful stance towards learning.
Another part of the problem is that an editing affordance, absent a more complete authoring experience for readers, would still strongly privilege the author’s writing. Such a medium would permit readers to adjust the author’s wording to better match their understanding, but wouldn’t permit readers to capture an interesting connection they noticed to some idea outside the text. Readers couldn’t capture a detail they found interesting, but for which the author didn’t provide a prompt. Readers couldn’t capture an idea the text inspired the next day. More philosophically, this asymmetry promotes the idea that prompts are something you consume, not something you create. Imagine if you were forbidden to use any writing materials of your own while carefully studying a book: you’re only permitted to write in the page’s gutter margins. Your thoughts would shrink accordingly.
Of course, a simple solution here is to add an authoring interface to Orbit. One possibility I’m excited about is that authors’ prompt-writing practices will scaffold readers’ own prompt-writing abilities. But this won’t happen if readers can’t fluidly write prompts when inspired to do so. People today can add their own prompts in Anki or SuperMemo while they read a mnemonic essay. But my instinct is that it’s harmful to create a strong separation between engaging with author-provided prompts, and writing prompts of your own—for instance by requiring readers to visit a separate app or page to write their own prompts while reading. If you’re looking at prompts, you should be able to write prompts. Author-provided prompts should feel like material in your hands, malleable to your needs and interests, not a different “kind” of thing from prompts you write for yourself. I want the fluidity of plaintext: copy and paste between documents, edit and combine coarsely or finely, bulk-manipulate, generate, grep, pipe, tweet, etc.
Memory fantasies
Let’s step back and examine the medium’s original goals. To indulge in science fiction fantasies for a moment, it’d be great to jack a plug into our neck, flip a switch, and deeply understand any topic. That’s not possible (for now)—but how close can we get?
Imagine that whenever you read—or thought—something interesting or important, you’d simply remember that idea. Learning wouldn’t be quite as easy as plug-and-play, but you could read a book (once) and remember every meaningful detail; you could tinker with those ideas on your workbench and remember everything you noticed as you put them into practice; you could discuss those ideas with a collaborator and remember all the implications that arose. To fend off dystopian objections, imagine that you can also effortlessly correct any memories which turn out to be false or unhelpful. All this is also impossible, but we can at least come somewhat closer.
Spaced repetition memory systems let us remember a specific detail at the cost of 20-60 seconds over the course of the first year, and < 10 seconds per year thereafter. Not quite effortless, but also not terribly onerous—if you have a good prompt already written. Writing good prompts requires much more effort than reviewing does, and it requires a skill few people have yet developed. This suggests a slightly more achievable version of our fantasy: a genie automatically creates spaced repetition prompts for everything interesting you read, think, or hear; you remember all those details at the cost of, say, ten minutes’ daily review.
Our genie would need two skills: it must read your mind (i.e. to notice what you find interesting, in what context, and understood in what way); and it must write effective spaced repetition prompts (i.e. which cue retrieval of the details which constitute understanding). As a more plausible approximation, perhaps you can imagine hiring someone to follow you around all day, to sit in on your meetings, to read what you’re reading, to listen as you think aloud—and to write memory prompts for everything which seems important. You could call them your “chief of memory,” a nod to your chief of staff. This would be awfully expensive (and intrusive), of course; and because this assistant can’t read your mind, their work would be imperfect. But it’s interesting to consider as a non-sci-fi model you could actually implement if you had the means. Can we approximate this model more affordably?
Books are surprisingly analogous to at least part of this situation. If you’re wealthy, you can hire a personal tutor to teach you about a topic. This is inconvenient in some respects, and of course, it’s quite expensive. Happily, thanks to the printing press and the internet, you have the option to read a book about the topic instead. In some respects the book will offer a better experience than you’d have with a tutor: when you buy a book, you can (indirectly, partially) hire the greatest domain expert in the world to teach you. The book’s prose will (hopefully) represent careful editing and sculpted narrative, rather than improvised explanation. And of course you can read much more quickly than you can listen. There’s no equivalent conversational analogue to “flipping through the pages” or “scanning the headings.”
We can apply a similar logic to our “chief of memory,” at least for the portion of your day spent reading. The idea here is that many of the details you’d find important in those texts would overlap with details other readers would find important. And so perhaps the work that your “chief of memory” would need to do as you read this text would overlap substantially with the work that others’ “chiefs of memory” would need to do. If there’s enough overlap, it might become a high-leverage opportunity for a domain expert—perhaps the author, perhaps a different expert—to write prompts covering these overlaps. Indeed, that domain expert may be able to write higher-quality prompts than a general-purpose “chief of memory” could for that specific text.
Now we arrive at an idea resembling the mnemonic medium—and still not the mnemonic medium. In our thought experiment, the idea is that you effortlessly remember everything you find interesting or meaningful. The author-provided prompts are just material for that process. More concretely, imagine that as you read, you talk aloud to your “chief of memory” by raising an eyebrow or gesturing at material you find important. If the author’s already done the work to write prompts about that bit, and the prompts match what your assistant thinks you found interesting, they can take a shortcut and use the author’s prompt. Otherwise, they have to take the slow path of new writing a prompt by hand. Perhaps in some cases you speak aloud: “this detail actually relates to the problem I’ve been having with my research project; it’s a reason to consider doing X instead of Y.” Unlike the mnemonic medium, this workflow is driven by the reader’s interest, rather than the author’s specifications. The author-provided prompts are a shortcut, not a list of expectations.
Okay, now imagine you have no “chief of memory.” I recognize that we’ve come awfully far from “I know Kung Fu” at this point, but if we restrict ourselves to just the portion of your day spent reading, how close can we get in software using author-provided prompts? Ozzie and I have been exploring some approaches along these lines. Our prototypes are quite nascent, but perhaps you can imagine highlighting details you find interesting while you read; if the author has provided relevant prompts, you’ll collect those; otherwise, we might provide a bulk-editing interface for shaping annotations into prompts, including those you might create from nothing.
Trading away effortlessness
There are many problems with this approach! One central tension is that in almost all cases, the correct amount of authority to assign to the author is not zero; the correct amount of control and responsibility to assign to the reader is not 100%. How should we negotiate the spectrum of control between authors and readers?
The author’s prompts are not in fact just a shortcut, as I’d described earlier. They also carry meaning. Prompts signal what the author finds important. They communicate a norm around what it means (or at least what the author believes it means) to understand a topic. They give authors the opportunity to communicate the prose’s ideas in a different way, or even to create conversation between the prompts and the prose—but that’s a topic for another essay. They cue attention and participation; when they’re working well, they create a feeling of support and safety.
And so maybe we should still present the author’s prompts, perhaps as annotations, but ask readers to “opt in” to each prompt they’d like to collect. The trouble here is that interaction is a cost center in interface design. If 80% of the time, 80% of the users want to collect 80% of the author’s prompts, a naive opt-in mechanic would require readers to perform a huge number of interactions to indicate the common case. Imagine reading through Quantum Country’s first chapter and clicking on 112 prompts in the margins to collect them all. You can perhaps improve the situation by batching those interactions in end-of-session “review areas,” akin to Orbit’s current review areas—but I don’t think that’s enough.
If we’re not careful, we won’t just require users to perform excessive interactions: we’ll also distract them and create decision fatigue. Imagine that as you’re reading a text, you’re constantly evaluating prompts that appear in the margin alongside the text: do I want to collect this prompt? Or do I want to write my own? Your eyes dart back and forth between the text and the sidebar; your attention is drawn away from the text. The frenzied nature of this design can be improved by bulk operations at the end of sections, but it’s still hard to imagine asking users to explicitly decide whether they’d like to keep each of the 112 prompts in Quantum Country’s first chapter. It’s already quite an imposition to ask them to try to remember the answers.
Another difficult problem is reconciliation. Imagine that you find a detail particularly interesting, so you highlight it. You see that the author’s provided a prompt about that passage—great. But now you need to decide: is the author’s prompt about the sense I found interesting? You need to form some picture of the prompts you would write, if you were to write prompts, then read the authors’ prompts and perform a sort of diff. Worse: if you don’t quite trust the author as a prompt writer, you also need to evaluate the quality of their prompts. My experiments with reading others’ mnemonic texts suggest that both these activities are quite taxing.
I think a successful approach here is likely to be more incremental, annealing the prompt set through a number of stages. Perhaps you mark passages you find interesting with some very coarse interaction. If a passage inspires some specific prompts you know the author won’t cover—for instance because they’re about a connection to your present project—you can write those inline as you read. Intermittently, at the end of sections, you review the author-provided prompts from the passages you’ve marked, both to reinforce your memory and to offer a lightweight opportunity to discard or edit those which obviously don’t work. Perhaps you notice at this point that the author didn’t provide prompts about some detail you found important, so you write some on the spot. Then, over the following weeks in review sessions, you refine the prompts from this text, modifying and trashing those which don’t inspire, removing inadvertent duplicates, filling in details with new prompts, adding connections you hadn’t noticed, and so on. But critically, the text’s prompts feel like yours; they’re co-mingled with your own prompts about your own ideas. Taylor Rogalski suggested a metaphor I like here: someone sent you their Google Doc, then you clicked File > Make a Copy so you could scribble all over it with impunity.
A postscript on machine learning and language models
I imagine that many of my readers have been chanting it this whole time. What about language models?! Why insist that authors (or those who adapt their texts to the mnemonic medium) do all this work? What about the long tail of texts which will never be adapted?
I know. I’m interested in these questions too. I’ve experimented with several approaches along these lines, and my impression so far is that some partial automation here is possible, for certain kinds of prompts, but that a decent solution will require a great deal of work and a great deal of insight. I don’t believe the simple approaches floating around are likely to represent viable paths. But I do think this work is worth doing; if you’re interested in and capable of taking it on as a research project, and you’d like me to supervise (and possibly help fund) or advise your work, please reach out.
My instinct is that we’ll be best off approaching this as an augmentation rather than an automation problem, at least in the near future. I have some specific workflow ideas along these lines, but they’ll have to wait for another essay.
————————————
I’d like to thank Ozzie Kirkby for prototyping ideas around this problem with me this summer; Nick Barr, Ty Jung, and Taylor Rogalski for extended discussions and whiteboarding; and Gary Wolf for valuable correspondence on this topic.
And I’d like to thank all of you, Patreon sponsors, for your kind support. It’s quite remarkable to have the opportunity to pursue open-ended exploration like this. Your contributions make it possible, and I’m grateful.
Comments
Ruben Artus
I wonder if a social layer of this could be that interaction between people of different levels of knowledge about a topic is facilitated. E.g. a person who read an interactive book should explain a topic from the first chapter to a person who is right now reading that part of the book. And there is some kind of back and forth happening to deepen the topic or clarify. Aligned with the idea that the best way to learn is to teach/explain others.
Ah yes, I see what you mean that it's tricky indeed to "diff intent." Even exact overlaps in highlighted text do not necessarily signify that the intent between the annotation of the author is the same as that of the reader's; it is even messier when considering partial overlaps.
Andy Matuschak
Thanks for this note! Ozzie prototyped something along the lines of this workflow. One surprising challenge was the need to "reconcile" or "diff," as I described in the post. Basically: many of the key bits you highlight/annotate are likely to have author-associated prompts with them. So now you're facing a list of annotations and author prompts, and you have to compute something like a set difference: which annotations *do not* have author prompts? Somewhat more subtly: which of the author-provided prompts intersecting my annotations capture the intent of my highlight, and which do not? Tricky.
The model that I came up with in my mind as I read this essay goes something like this: - Read through text without any embedded prompts - Highlight parts that I find relevant; annotate them (logged to Orbit) - At end of essay, click "Review" button - A full-window interface (preferably a same-site modal) pops up and shows me my highlights and annotations - AND it cleverly shows me the parts where the author chose to add prompts (so in this way, authors need to add prompts to specific parts of the text), next to my own highlights and annotations - (Perhaps, with some ML, I'm also shown prompts that others who are like me have saved. But I'd personally prefer if these were shown only when the model has high confidence, to reduce noise. I think of Kindle popular highlights) - I (reader) am given control to ultimately decide the prompt I want to keep for remembering via spaced repetition I think this model came together in my mind the moment I read about Taylor Rogalski's metaphor of the Google Doc copy-and-edit-with-impunity. That's a great metaphor! Thanks for writing this essay, Andy!
Andy Matuschak
Yes, crowdsourcing is quite an interesting approach on this problem. One angle I'm unclear about: how important is it to have a consistent author of the prompts for a given work (whether that's the original author or a single coherent "annotator"?) Is it more like Wikipedia or the Stanford Encyclopedia of Philosophy?
Andy Matuschak
Yes, it would certainly be nice to improve the complex workflow here. Thanks for sharing your experiences, Hana!
Besides authors and readers, another place to have some control over the prompts could be in a curated crowd mechanism. Maybe readers could place their own prompts (in addition to any the authors provides), and after a while passages that are linked to a lot of prompts get the attention of a curator who presumably sees a (or several) theme in the cards reader have been placing and crafts a one or more good prompts for that section (maybe he can also see the review performance of readers to see which prompts have been working well). Readers who had already made cards are alerted and able to swap their card for the curated one, while new readers are then offered the option to collect the curated card instead of their own (the curated prompt might replace an author prompt if that wasn't doing a good job). Then neither the reader nor the author needs to be particularly skilled at crafting prompts, they just need to provide a starting point for the curator to work from.
Andy Matuschak
Definitely! I think it'd be really interesting to accompany fine-grained memory prompts with broader "try to explain this concept" tasks.
Andy Matuschak
This really resonates, Chris. You're right that in practice, my reading is so often iterative, spirally…
Andy Matuschak
Yes, I think that's right. It's basically a very lightweight, generally embeddable ITS. Makes sense to try to increase that scope…
Andy Matuschak
Gosh, yes, this is absolutely right. It connects to the notion of "salience prompts," which I find fascinating… but which I haven't yet figured out how to characterize.
From my own work flow as a student researcher, I've found that tools attached to my reading all come from some kind of annotation of a digitial text that I can process later. They either take the form of 1) an anki prompt 2) a literature note with an associated excerpt that I'll later export to my Zettlekasten note library or 3) some kind of prompt for the Feynman technique. I think when it comes to generating user interaction with the interface, I think that it's worthwhile to give the user options with what to do with the prompts later. There can be a wider array of tasks that you can schedule in order to integrate an idea from a text into your own thinking. For now I use a cumbersome combination of several apps for different purposes when I am reading (iBooks to annotate, Zotero, Obsidian, Anki Notes, GoodNotes for Feynman). It would be helpful to have a centralized interface all these future activities can be efficiently prepared when one first encounters the text.
Perhaps a helpful reminder is to consider the underbelly of genuine “understanding”; viz., the Feynman Learning Technique. The facility to teach, simply; to apply, dynamically, and extend, contextually. If we cannot teach “it,” we cannot use “it”. And if we cannot use it, we aren’t learning, we are memorizing. Nothing new here, but sometimes the most powerful insights hide in plain sight. Cheers.
The way you discuss the process of entering things into Orbit implies to me that after you do it, you’ve sort of mined the source of what it has. I wonder what this process would look like if you emphasized provenance, and allowing users to return to sources from which they picked up one card in order to find a few more. Definitely as a reader, I would feel more free to filter out unimportant ideas if I thought I could always return to them.
I'm inclined to agree re ML. Hopefully it'll get there eventually, but for now. The comparison of tutors and books suggests to me that eventually an evolution of the mnemonic medium is a pedagogic medium broadly (What aspects of learning are not covered in the current system?). The initial iteration was "embedded curated flashcards", but once one opens the floodgates of returning control to the reader, and enable a more fluid conversation with the author via editing (And potentially, other users via suggestions), one can then expand to something that may well cover what a digital tutoring system may aim to do, if the system truly gets what a given reader needs.
This would indeed be ideal. Part of venturing into a new literature is figuring out what matters or not, and right now this works by either someone telling you or reading enough that you can patternmatch your way through. But at the same time it's something very personal! Different people have different goals for what they read. Following the CoM metaphor, the CoM would have a very detailed model of you and what you tend to find useful or not. What comes to mind is some form of social recommendation system where you get "users like you wanted to remember this or that" reminders.
Benjamin Reinhardt
The one piece I'd add to the "Chief of Memory" metaphor (which I love) is that you want the CoM to whisper in your ear whenever it's relevant for you to remember the things that you had come across previously. Wealthy and powerful people have advisors who whisper in their ears for a reason! This function is, of course, the goal of spaced repetition but I think it's important to call it out instead of seeing prompts as the final goal.
