Andy Matuschak
patreon
“Skip”: exponential-backoff deferral mechanisms and fuzzy inboxes
🕑 Added 2020-08-20 18:04:15 +0000 UTCIf you search Google Scholar for research on spaced repetition systems, you’ll find a sea of papers focused on optimization. They’re tuning algorithms for less forgetting, more stabilization, better scheduling. To be clear, these are worthy aspirations. If we can improve response accuracy even just a couple percentage points, the exponential schedule magnifies our gain, substantially increasing the capacity of our system. But this extreme focus on efficiency misses the highest-order bit: almost no one actually uses these systems, so their practical efficiency rounds to something near zero. As Gwern framed this problem: “If you’re so good, why aren’t you rich?”
Happily, memory systems are already extremely efficient. Having experimented with these systems for a few years, I’ve come to believe that the critical thing to optimize is emotional connection to the review session and its contents, and conversely, to ruthlessly minimize elements which provoke a sigh. It’s a continuous process. These systems, left to their natural inclinations, naturally decay to produce dutiful sessions which feel disconnected from anything that matters to you.
Some techniques can help up front: don’t “stockpile” material for some future day; avoid writing prompts in response to a feeling of “should”; write about connections, consequences, implications—not just about facts; avoid tangential “orphan questions”; etc. But this isn’t enough. Review sessions need ongoing grooming to stay interesting. If you don’t add anything new for a while, your sessions will feel stale. If you often forget an answer, you must refactor the question or risk future eye-rolls.
For me, the most important type of maintenance concerns my shifting interests. I might stumble on a fascinating paper and write a few dozen prompts to help me internalize the details. Now fast forward six months. In the happy case, I’m thrilled when these questions recur because they reconnect me to the ideas of this fascinating paper, which now appear richer through the lens of my intervening experiences. Much of the time, I simply feel grateful to still remember the details. But sometimes, I no longer find the paper very interesting; the questions just feel like a chore.
If these review sessions are to be a long-term habit, it’s important not to spend time with prompts about things that don’t matter to you. Without culling, sessions stop feeling valuable and start feeling like a chore. Long before sessions feel like a chore, unimportant questions will dull your focus on their neighbors.
The problem is that culling is hard. It’s not functionally hard: the app has a simple delete button. It’s emotionally hard. Much of the value of computerized memory systems is that they save you from having to make endless tiny decisions about what to do and when. I see memory systems as an example of a more general class of “programmable attention” systems. Making a decision to delete a prompt is a relatively weighty decision. It might be nominally undoable (e.g. from a trash bin) but in practical terms, you know that you're unlikely to restore a deleted question. The irreversibility of the decision feels misaligned with the slight emotional aversion the prompt produced. So most people (myself included) will tend not to delete as much material as they should, which means their sessions will be less emotionally connected than they should be.
I believe this problem arises not just in memory systems but in email inboxes, reading lists, overflowing browser tabs, to-do lists, and so on. I’ve been exploring a mechanic for a fuzzier, less-destructive alternative to “delete” operations.
Imagine that this is what you see when you’re reviewing a prompt:
Note the button in the bottom-left: “Skip.” We could also call it “Not now,” “Later,” “Defer,” or simply: “Nah.” When we press it, we simply move on to the next prompt. The current prompt will reappear in, say, a few weeks. Maybe then we’ll be more interested; if we decide to answer at that point, the review schedule will continue as normal. But if we hit “Skip” again, we’ll defer the prompt for a few months. At which point if we skip it again, we’ll defer it for a year or more. So after a few consecutive “skip” operations, we’ve effectively “archived” this prompt, but we never had to make the destructive, non-contiguous decision to remove it. Instead, we just respond to our emotional reaction in each moment. If a prompt elicits “nah,” this mechanism gives that emotion a safe outlet. It’s “fuzzy delete.”
I believe this mechanism can be usefully applied to anything shaped like an inbox. I’ve experimented with it for reading and writing lists, and I’d like to expand that. Here’s a springboard to more general working notes on the “fuzzy inbox” topic.
This kind of mechanism is important not just in review sessions but in the reading experience of the mnemonic medium. Quantum Country assumes a goal of completionism: readers are expected to answer every prompt in the essay. That’s certainly appropriate for some texts, but in more casual texts, readers may have a wider range of prior knowledge and interests. The completionism requirement also pushes authors around: it restrains them from writing prompts which many readers would likely find interesting, but which feel inappropriate to make mandatory.
But if we handled that by asking readers to pick and choose which questions they’d “bring into their Orbit,” we’d create much the same emotional friction as with deletion. It’s too weighty a decision, too early. Many people would feel hesitant to drop questions they don’t really care about. In many cases, since they’re still reading about the topic, they may not even know what’s important yet! So this same fuzzy mechanism seems useful in the context of the initial reading experience.
One challenge for this mechanism is that the feeling of aversion which accompanies material you don’t care about is somewhat similar to the feeling of aversion which accompanies challenging material. We don’t want to encourage people to skip questions for the latter reason: desirable difficulties are important to learning! I’m not sure how to address this conflict within the interface: at the moment, the point (like many others) will have to be communicated through culture and the halo of “canonical literature” around the tool. This challenge is more fundamental than it may seem. Memory systems achieve their remarkable efficiency by scheduling prompts for when they should feel hard to answer. This is probably a major barrier to adoption: review sessions focus on the material you have the most trouble remembering, which both makes them somewhat unpleasant and also makes it appear that the system’s not working!
Another, more parochial challenge, is that I’m not sure how to convey the state of a skipped question in the interface. If an article has 100 questions and you skip half of them, do we talk about your memory of the article in terms of the remaining half? Graphically, the rays of the starburst depict interval period. If I skip a prompt the first time I see it, does its ray jump to 1 month? This seems to convey a false sense of “progress”… but maybe it’s not false, since you don’t really intend to review that prompt. On the other hand, if we leave the ray at the minimum length, it’ll feel unpleasantly “left behind” as you review the rest of the article.
***
I think I was thirteen when I first read this quote on WikiWikiWeb:
The first ninety percent of the task takes ninety percent of the time, and the last ten percent takes the other ninety percent.
Why do I ever share estimates of project timelines? You’d think I’d have learned by now.
Anyway.
I’m deep in the second ninety percent of Orbit’s initial release. It’s a grab bag. I have replaced one binary serialization protocol for a different binary serialization protocol. I have patched a platform text renderer whose hinting was broken. I have gone cross-eyed staring at endless iterations of approaches to styles for embedded prompts.
We’re in the home stretch. Zeno keeps stretching the home stretch. More soon. In the meantime, I've created an Orbit account on Twitter which has been posting little visual flotsam that doesn't merit a full Patreon post; feel free to follow along. The good stuff will eventually end up here. Speaking of which, here's a fun little study I did recently on the starburst:
I’m giving a talk about Quantum Country at a small private conference this week. The occasion has pushed me to return to analysis of Quantum Country readers. One interesting finding to report: in How can we develop transformative tools for thought, Michael and I showed this pleasingly-exponential graph, which shows that readers reach around 54 days of demonstrated retention after six repetitions.
In the second half of 2019, we made a number of improvements to the platform, which have cumulatively resulted in compressing this curve by more than two repetitions. That is, readers now achieve a noticeably higher degree of demonstrated retention after four repetitions than readers did last year after six. Putting aside for the moment my hesitations about focusing on efficiency, there really does appear to be quite a lot of low-hanging fruit here.
As always, thank you for funding my burgeoning research grant. Relatedly, if you didn't see it, you might enjoy this little thread on Twitter expanding on some of the funding model ideas I brought up in the last post.
Comments
Andy Matuschak
I dig these ideas—thank you!
Malcolm Ocean
"Not interested"
Malcolm Ocean
"I don't care"
Malcolm Ocean
"Whatever"
Malcolm Ocean
"Boring"
Malcolm Ocean
There might be a way to rename "Skip" to better evoke that it's intended for disinterest rather than challenge. Ideas:
Andy Matuschak
Totally agreed! I have sketches for a workflow like this, and I think it could be very helpful.
Thanks for the writeup, this is very interesting! A related concept I've found myself wishing for in Anki is "mark this card as needing edits". I often go through multiple revisions of a card before it really works for me, and having to interrupt the flow of study to make specific edits when a card happens to come up is much worse than just having a button.
Andy Matuschak
That's interesting, or even—"think to yourself: why do you care about this? if you decide that you have no good answer, click here to snooze this prompt for a year"
I'm sure it could if a filter was set up to search for similar resources on the net. I'm surprised to hear that your other QC reviewers aren't furthering their knowledge past the articles in QC.
Andy, thank you for the interesting writeup. I wholeheartedly agree that "the critical thing to optimize is emotional connection to the review session and its contents." A potential optimization on top of the "Skip" concept would be to explicitly ask the user (after they've skipped a question a couple of times) if they are still interested in memorizing that bit of information. If they answer yes, that might actually strengthen the user's emotional connection to the material, since they just described it to themselves as important. If they answer no, then you can immediately archive the question.
Andy Matuschak
Good question. This type of interaction certainly distorts the data, particularly over long time periods. After many interviews with long-time QC reviewers, my impression is that the substantial majority aren't doing this, so I'm not inclined to be too worried about it—at least in terms of "data integrity." But it's interesting to think about opportunities in this regard. Can the system _help_ you with this "boosting"? Help you connect to outside sources which might facilitate deeper engagement?
Tangentially related to your comment, "Now fast forward six months. In the happy case, I’m thrilled when these questions recur because they reconnect me to the ideas of this fascinating paper, which now appear richer through the lens of my intervening experiences," I'm wondering if you've thought about how this impacts the data collected supporting spaced repetition. With Quantum Country I've gone on to other materials and instruction to further my knowledge of quantum computing / mechanics. So, when I come back and review do you find it material or immaterial that I have had my knowledge "boosted" by other sources? Does it matter whether the remembrance comes from the spaced repetition of the original article or from improvements made outside?
