Andy Matuschak
patreon
Fluid practice for fluid understanding
🕑 Added 2023-06-01 06:17:58 +0000 UTCEarly, rough thinking, but perhaps useful for others interested in the design of learning environments. Assumes detailed familiarity with memory systems.
When everything goes right, my memory practice totally transforms the experience of diving into a new topic. I’ll feel like I can keep turning a crank, and my understanding will just ratchet up and up, durably and inevitably. But quite a lot of the time, everything doesn’t go right. I’ll find that I’ve ended up with brittle, parochial understanding of a new topic. Review questions will strike me as boring or alienating; I’ll feel like I’m parroting, rather than really understanding.
One way to address this problem is to improve as a practitioner of memory systems. I grow my armory of prompt-writing strategies; I monitor my emotional connection more attentively; I intervene more aggressively when something’s not working. Over-simplifying, the quality of my memory practice depends on the quality of my prompts, and on the processes I’ve put into place for monitoring and revising them over time. If I’ve produced a brittle understanding, that means I need better prompts and prompt-writing processes.
Prompt-writing ability is a useful framing. I’ve improved enormously at writing and maintaining prompts over the years; my memory practice is much more versatile and reliable as a result. Yet it’s also worth thinking about the limitations of this framing. Aspirationally, I’d like a memory practice which not only helps me build and maintain a flexible, reliable understanding, but which actually deepens my understanding over time.
I’ve been wondering: to what extent is achieving that goal about learning to write better prompts? Are individual prompts the best primitives for that task? Generalized flashcards are a convenient representation for declarative knowledge, but I’d like to help people learn complex conceptual topics. If we step back and look at what’s actually necessary to produce that sort of understanding, and at the mechanisms of practice, could we find a better-matched primitive?
Learning from Alex
This exploration is motivated in part by my experiences with Alex, a motivated adult student whom I helped study physics earlier this year[1]. Among other experiments, I wrote hundreds of memory prompts for Alex to help him internalize his readings and tutoring sessions. I learned a great deal by watching him grapple with these concepts both inside and outside his memory system. Being an observer, rather than a learner, really helped: when I’m writing and reviewing my own prompts, I often find it tough to create reflective distance in the moment, to see what’s going on and what’s breaking.
One key issue I saw in Alex’s experience was trouble with transfer. Often, he learned to answer the prompts I’d written, but his understanding was brittle: he struggled to draw on what he’d learned in slightly different circumstances. In my experiments, it seemed that what Alex really needed was much more variation in his practice with each idea.
For some concepts, I tried the standard memory system practice, writing a dozen (or more!) prompts about this one idea, accessing it from many different angles and with many different surface features. This may have produced more flexible understanding, but the impact on review sessions was quite negative: even when prompts are shuffled, it’s obnoxious to answer many variations of the same question in a single review session. Because these variations are so burdensome, I didn’t write them for as many concepts as I probably should have. And from a cognitive perspective, retrieval will be much less effortful after the first question, so the subsequent questions will probably not produce as much reinforcement as they could.
Emotionally and cognitively, I believe it would be much better to “smear” these variations out across many review sessions. Conceptually, I thought of myself as writing many variations on the same Platonic idea, using various strategies to solicit that idea in different ways. These prompts were a single thing in my mind—a “prompt cluster”—but represented as an unrelated set in the memory system, despite the high cognitive overlap.
A related phenomenon: we saw a lot of pattern matching in Alex’s review experience. His comments would often include the telling phrase “this one”—i.e. “Oh, I know this one has a trick where…”. This is probably a sign of trouble for transfer, as we’ve discussed, but it’s also a marker of emotional friction. It feels bad to parrot. It makes you (rightly!) question whether your practice is a good use of time. It can create a feeling of rote drudgery. It seems to make Alex (and me) less likely to pay close attention, not only to that question but sometimes to the ones which follow. I believe that pattern matching creates emotional distance, a dulling of involved participation.
So it’s interesting to consider the design goal: how might we make pattern matching never occur? What if an idea is never accessed in exactly the same way twice? In fact, I have a strong instinct that this is a terrible idea as written. There’s something very powerful about the reflexive stimulus–response pattern that highly entrained memory practice establishes. And the feeling of flow in practice sessions is delicate: you don’t want rote boredom, but you also don’t want meaningless make-work. Still, I’d like to experience these tradeoffs more viscerally.
Alex also needed variation in scope of the prompts over time. Immediately after reading an explanation, he most needed prompts which acted like a simple reading comprehension check—in many cases, he hadn’t actually understood or absorbed some of the important statements on the page. Then he needed help building a durable memory for key details: terms, notation, equations, relationships, etc. But that’s not enough to actually understand the topic. We’d meet to discuss the concepts and to solve problems together, and I’d write more prompts for him each time, often about progressively higher-level details.
Once the fundamentals were in place, what he really needed was problem-solving practice. The problems were tough for him, so they needed to start quite simple and become more complex. Intermittently we’d retrace some worked examples in great detail. After days of problem-solving practice in one chapter, he’d want to move onto the next chapter to keep motivated, but we needed to ensure that he’d swing back to earlier chapters and continue to solve problems about their ideas over time. At this point, the “reading comprehension” prompts and some of the basic prompts on notation felt burdensome to review, but more complex declarative prompts, like those on equations, still seemed necessary to keep those details fresh.
Eventually, Alex and I got to the point where we were connecting the material we were learning with new research papers he found interesting. It’s conceivable to me that review sessions could be systematically orchestrated to include that sort of activity: he showed me the papers he wanted to understand, and I said things like “oh, once you’ve got some experience with electric flux, we’ll be able to make sense of this figure together.”
Alex’s study sessions needed to consist of different activities at different stages of his learning process. He needed more complex practice activities to really understand the material, but those couldn’t be introduced immediately. Conversely, trivial “bootstrapping” tasks could feel obnoxious when he’s just finished an activity which uses those same ideas in much more complex ways. If we want a memory practice to deepen our understanding over time, that will likely mean shifting the distribution of tasks over time to create progressively more complexity, elaboration, and creative opportunity.
In summary, my experience with Alex suggests that to produce fluid understanding, one would benefit from much more fluidity in review. A straightforward flashcard works if you want to remember the value of the electric constant. But to internalize complex conceptual matter, you want to be pushed to draw on each idea from different angles, with different wording, in different modalities, alongside different combinations of other ideas, at different levels of complexity over time.
Review sessions as generic vessel
Part of my hazy thesis here is that much of the potential of a memory practice is not about memory, per-se: it’s about the daily sessions. A steady Anki user has designated a ~5-15 minute slice in their day for systematically improving their knowledge and understanding. That’s a powerful vessel. It can be filled with rote recall flashcards, but more experienced memory system users discover that prompts can be used to stimulate a wider range of thought.
My contention is that if you’re trying to learn a complex conceptual topic, we should fill that daily vessel with whatever activities would most effectively and enjoyably produce the understanding we desire. If the material is new, it might be best to reinforce your memory of key details, and ask you to predict the next steps of some partially-worked examples. In the next session, you might be asked to solve some simple problems; in the next session, some different, more complex ones. One week later, you might be offered an alternative derivation of a theorem you’d studied, and some questions relating the two approaches. Perhaps it would be helpful to ask you to synthesize an explanation, or to brainstorm a list of questions you have about the concept. Each session’s contents are completely different, unless some activity seems particularly desirable to repeat verbatim.
In some sense, I’m arguing that “memory systems” can be thought of as “practice systems”. You can’t quite orchestrate what I’ve described for yourself with today’s tools. But experienced memory system users have internalized many strategies for filling their practice vessels—that is, for digesting ideas into interesting and effective practice tasks. These strategies are a reflection of several more fundamental (generally implicit) understandings:
- a theory of knowledge: what it means to understand something in a particular context;
- a theory of learning: how to construct concrete activities which produce and maintain those understandings; and
- a theory of the medium: how to instantiate those activities in an item as it will be presented by an existing memory system, and in the context of a review session (i.e. possibly on a phone, interleaved with many other tasks, often without pen and paper, etc)
As Michael Nielsen has pointed out, it’s tough to directly communicate these strategies or these understandings. But I suspect there is a kind of “pattern language” hiding within these strategies we develop. I've now written memory system prompts for quite a lot of technical material; it would be interesting to work through those prompts and their associated context, looking for "vocabulary" and "grammar".
Prompt primitives for conceptual learning
The flashcard primitive captures declarative knowledge quite directly. If you’re studying Italian and want to remember the word “carciofo”, you can write it on a flashcard and write “artichoke” on the back. If you want to learn anatomy, your flashcard can be an arrow pointing to a part of an illustration. You can learn lead’s atomic number by writing “lead’s atomic number” on a flashcard and “82” on the back. Context makes the task obvious; in each case, the implied task is to answer: “What is this?”
But when you’re learning about the relationship between electric potential and electric potential energy, you can’t put that concept directly into a memory system. It’s multifaceted. There are lots of tasks you should practice to understand that relationship. You should make sure you know the symbolic relationship, sure. But you should also be able to apply that relationship in various scenarios; you should be able to see when it makes sense to use one of these concepts rather than the other; you should see the parallel between this relationship and the relationship between electric field strength and electric force. You’ll want to form new connections to this concept over time, as you learn more. For example, once you’ve been introduced to capacitance, you’ll be able to work with a new facet of this relationship.
One way to model all this is as a large collection of primitive prompts, each reinforcing a different “atom” of knowledge related to that concept. But as we’ve discussed, they’re all so highly related that it would be better to smear them over many sessions. Feedback on their scheduling should be at least somewhat linked. You’d want many of the task details to vary with each presentation. And some of these tasks should be added later, once you’ve digested the earlier ones or learned more.
Practically speaking, when I write prompts for a concept like this, the way I’m thinking about it is that there’s a single thing—the concept—and I’m turning it in the light to see it from many angles, to see how it interacts with other objects in the space. My implicit prompt-writing “pattern language” suggests a family of tasks, and as I start pulling on those threads, more patterns suggest themselves. Sometimes it’ll feel like I’m working with the focal concept and another adjacent concept at once; or I’ll feel like I’m zooming in on some facet of the concept so far that the facet itself becomes the object of my attention. But the concept itself retains a sort of integrity in my mind. For the most part, mentally, I’m “putting that concept” into my memory system, in richer and richer ways. The concept is the primitive noun, and the verb (“putting it into my memory system”) is a messy, complex thing which depends on my pattern language and my prior knowledge.
I can put the word “carciofo” directly into my memory system. There’s almost no distance between the object itself (“carciofo = artichoke”) and its representation on a flashcard (“Q: carciofo? A: artichoke”). That flashcard basically is the declarative knowledge atom. That’s its strength as a primitive. Memory systems were developed for learning declarative knowledge, so maybe we shouldn’t be surprised that their core primitive has a natural representational unity with that kind of information. But now we’re trying to stretch memory systems to work well for internalizing complex concepts, too. Reframing “memory systems” as “practice systems” for a moment, is there some other kind of primitive waiting to be created—one which would let me “add the concept” as an elementary operation, in the same way I can “add a vocabulary word” to one of today’s memory systems?
Language models and ideas as primitives
In the past few months, lots of people have been trying to use large language models to automate the creation of memory system prompts from explanatory text. This might be close to tractable now (1, 2), for simple declarative knowledge. But I’m more interested in memory systems for their potential to help me learn complex conceptual material, and in deeply internalizing ideas that are relevant to my creative work. This is a very different task, not obviously amenable to the same kind of direct automation. The language model doesn’t seem to have the pattern language we’ve been discussing—in part because it’s still something being actively discovered at the frontier of the memory system practitioner community.[2]
But if I could externalize that pattern language into something the model understands, maybe we could produce a memory system in which ideas are a primitive, alongside concrete tasks. Here, I’m using the word “idea” very broadly to cover a multifaceted-but-distinctly-coherent element—like the relationship between electric potential energy and electric potential. In such a system, perhaps you could “add a concept” directly, optionally with a comment about the nature of your interest, and the system would use the pattern language to create novel practice activities on the fly in each session.
What does it mean to “add a concept”? What specifically would you be adding? One natural route would be to add a passage from a book, perhaps with some markup designating the central idea within its context. In this world, the memory system library would consist not (or not just) of static flashcards, but rather of a set of marked-up references to texts or personal notes. This would also mean that if a task gave you trouble, you could easily navigate to the writings which inspired it.
This conceptual design isn’t “use a language model to generate prompts, then add them to a library”; it’s “add ideas to a library, then use a model to generate activities dynamically.” Those activities would vary in each session to help you encode a fluid grasp of the concept. They can become more complex to help deepen your understanding over time—for instance, using the pattern language to combine multiple ideas. If the pattern language is concrete enough that it can be reified in the interface, we can give different kinds of activities different visual identities (e.g. problem-solving practice, visualization, generating examples, explanation synthesis) and allow users to quickly flip between alternative activities for a given idea.
My claim here is that much more fluid practice will produce much more fluid understanding. But idea-as-primitive is appealing from an interaction design perspective, too[3]. My concrete emotional experience as a memory system user is: I hear or read or think something that excites me. I think, “ah! I want to bring this into my practice!” But I can’t quite bring “this” into my memory practice; I have to find a way to transform it into some other object which I can bring into my practice. Sometimes, that transformation task is a welcome exercise. It brings me closer to the idea that stimulated me, helps me pick it apart and understand it better. But much of the time, it just feels like a burden. I want to emphasize that this impulse is not about efficiency or “lowering the floor.” It’s about making the core verb feel better—making it more naturally aligned with my internal intent and my emotional interest. I think this explains the attraction of cloze deletions: they’re closer to “adding the thing itself.” The trouble is that they don’t really work for complex material, at least not when used so directly.
What next
Language models are shiny, but I don’t think that the best way to explore the ideas I’ve articulated here is through a leap to systematization and automation. I can experiment with “practice systems”, variation, problem-solving, escalation, and “pattern languages” manually, working with another student or perhaps with my own studies. Intimacy and rapid experimentation still feel like the right properties to prioritize for now.
One serious reservation I have about these ideas is: by expanding the scope of memory systems to include such a wide range of practice and learning activities, I dilute the specific powerful idea that they represent (efficient, systematic retrieval practice using a simple primitive). It might be better to let memory systems remain tightly scoped, better to play to their natural strengths. What I’ve been describing is instead closer to an intelligent tutoring system. I have a lot of reservations about those systems, but the last time I grappled with that body of research was years ago when I was at Khan Academy. If I continue down this path I expect I’ll need to return to that literature with fresh eyes, and to draw some clearer delineation between my ideas and those of that field.
————————
[1] Alex’s professional life has been undergoing some significant upheaval, so his studies have been on pause recently. We may resume in time, and/or I may start working with another student.
[2] This relates to a gripe I have about the recent gush of “AI tutors”: they lack a clear theory of instruction. Some of them are told to quiz you with boring problems; others seem simply told to “be a world-class tutor and answer the student’s questions.” What does it mean to be a world-class tutor? That’s hard enough to answer; it seems clear the model doesn’t have a strong opinion. What does it mean in this person’s context, specifically? A better system needs to describe a theory of instruction—what is supposed to happen in the course of a session, what does that mean cognitively, and how should the tutor bring that about, in interaction with the student’s behavior? If there were millions of tokens in the training set with good answers to those questions, the model might not need more steering. But like the memory system pattern language, this sort of knowledge is mostly tacit; where it is written out, accounts differ so widely as to offer little guidance.
[3] There’s an extended discussion of “ideas as primitives” in last October’s letter (“Lessons from summer 2022’s mnemonic medium prototype”), mostly from an interaction design perspective.
Comments
Amrit Daswaney
Hi Andy, This turned out much much longer than I intended - sorry about that! Everything is in a very nascent stage so I thought it would be more effective to share the entire thought process with examples. I've broken down the response into 3 parts so you can skip around: 1. Original problem I tried to address with templates, 2. Original idea/inspiration behind templates, 3. Templates I used and issues/successes I've had with each of them. I've also moved most examples / comments to the footnotes so it is easier to read. The main issues I was (am) facing are pattern matching and (maybe an overblown worry about) parochial learning. A representative question for the pattern-matching issue is I try to prove a theorem, I do it incorrectly and want to remember my thought process (misconception) that led to the mistake. However, during review, I almost never have to read the question entirely and instead immediately recall "This is the one where..." [1]. It is at least somewhat true that I don't make the same mistake when proving other results (I catch myself thinking along the same incorrect lines before correcting myself), however, you don't know what you don't know and I'm not sure how to delineate between I understand the concept vs I know how to avoid a very specific version of a more general mistake. Along somewhat similar lines, I am more broadly worried about being unable to fluidly apply concepts I have "learned" and instead am only able to answer specific questions. The original inspiration for making template questions was (very loosely) the Factory Method (software) design pattern. The main idea was to have a small set of "well-tested" templates. First, using a small number of templates at least partially addresses the shape-of-question-related pattern-matching, i.e., it leads to many very different questions looking alike and therefore, I have to read the question more closely before answering [2]. Second, loosely inspired by [3], my initial idea was if I can tinker and through experimentation identify certain templates that "work" [4], I can hopefully start to identify feature(s) of templates that work and use that to build others. I haven't figured out rigid templates as of yet but below I sketch the rough ideas as of now. I have broadly used 4 different types of templates: I. Definition-related 1. Standard/boring. Define (in words only), Define (using only a mathematical expression), Formalize {{statement in words}}, Explain in words {{mathematical expression}}. 2. Stolen from Issa Rice [5]: Give an {obvious, surprising} x {non-example, example} of {concept}. This has been spectacularly useful. I am not sure about its impact on remembering but this has been one of the unexpected benefits of having templates. If I cover a concept and don't have a question from a template, asking myself why usually helps. Sometimes it is not necessary, at other times, like for this template, it has deepened my understanding of the concept [6]. 3. Inspired by [7], Define a concept X that meets {desiderata} / What would we need to do if we needed to satisfy {desiderata} / Does definition X meet desiderata Y? These are bad questions but see examples in [8], [9], and [10] respectively for the idea. This has also been useful for similar reasons - delineating each part of the definition and understanding the role of each part individually and how they work together [11]. II. Story time In concurrence with something you've mentioned in your writing, I find that stories / vignettes / anecdotes significantly increase joy / delight during the review session. In addition, I find that they are also one of the most effective ways to learn things [12, 13, 14]. In particular, I have found they are strong "anchors" to a network of other connected / associated material. I wanted to highlight this template because for example, in Quantum Country, I really enjoyed the part where both of you explain why we study the Hadamard matrix by connecting to taking a journey from North Africa to the Iberian peninsula and separately added the card to my own deck. III. Application prompts Let's say I want to practice applying a concept but I am worried about remembering the answer and therefore, don't need to apply the concept to answer the question correctly. So far, there are 3 ways I've handled this. Easy case. I only need the associated numbers to change so I don't remember the final number-related answer. For example, if I want to practice the law of total variance using an exercise from the textbook. Most of the procedure is independent of the specific probabilities used and the only requirement is the probabilities not be too unwieldy. I use a script that takes in relations between variables and gives different current small numbers so the question is different. "Variable name hack." I think it is better to show you the hack before explaining it. Which question is "harder": Assuming they exist, what are the solutions to the quadratic equation: (a) $ax^2 + bx + c = 0$, (b) $xa^2 + ya + z = 0$? In theory, both questions are exactly the same but (at least for me), I remember the answer to the first without even thinking about it. However, because (b) uses unusual variable names, it causes me to slow down and think. This tiny bit of slowdown is all I need to make go through the process of thinking through the question instead of blurting out the final answer. 3. Hard case. In software testing, a simple example of testing each case is testing a function that checks whether a year is a leap year. Every year can fall into one of 4 buckets: (a) Not a multiple of 4 or 100, (b) Multiple of 4 but not 100, (c) Multiple of 100 but not 400, (d) Multiple of 400. If we can write test cases for each of these cases, then we have effectively tested all possible years. I have been trying to identify such partitions for application questions and adding them. I have not made as much progress here. IV. Prove or give a counterexample There are 2 types of questions here: 1. "What was the key idea(s) behind the proof of {...}?" / "Can we prove X with {simpler/obvious approach}? Why or why not?" ~ Nothing, in particular, to report about these questions. I'm sure this is a fairly common approach. 2. "Prove or give a counterexample." The main approach of phrasing ALL questions like this instead of "Show that / Prove X" is that statements "in the wild" that I would like to prove may or may not be true. For this approach to work, I need to have a reasonable mix of statements that are true and false. I have run into a bit of an issue with this and wonder what you think. For example, consider two very similar questions [15]: (a) If there are n iid bidders for a single item with value distribution F , what is the revenue-optimal auction? (b) f there are n iid bidders for a single item with **regular** value distribution F , what is the revenue-optimal auction? (regular is highlighted for your reading ease but isn't highlighted in the card). It turns out the answer for (b) is "second-price auction with Myerson's price as reserve" and the answer is "it depends" for (a). The rule I've enforced is that if I don't read the question clearly (don't check that the distribution is regular) then I've answered the question incorrectly. This does feel like a "gotcha" question and is frustrating when I get wrong but I can't think of a better way to do this. In part, I think this speaks to the emotional connection during review that you've spoken about. However, as a counterpoint, I often wonder about an anecdote Ericsson [16] shares: when professionals / musicians aspiring to musicians finish classes, they are often exhausted because those are periods of hard, focused work. In contrast, people who take the class as a hobby often feel refreshed and relaxed after the class. There are times when my review practice seems difficult and hard and in general, this doesn't hamper my desire to do them. I do derive joy / delight from some questions during the review but for me, the core value proposition is more long-term in nature: (a) my significantly increased ability to remember material I've read and use them in a flexible fashion, and (b) as guardrails, I feel much more secure learning / reading new material because I know I have a strategy to remember the parts I need to so I can just lock-in. Hope this helps! [1] As an aside, I have tried writing multiple questions from different angles, both with and without the original (theorem-related) context. The main "issue" is the memory of the mistake is usually so strong that it "anchors" all the other questions, even if the question doesn't even indirectly reference the mistake. [2] Is this "better"? Do I gain anything from reading the question fully vs recognizing the shape of the question? To be honest, I don't know whether I think/feel so and/or I want it to feel that way. [3] Towards A Rigorous Science of Interpretable Machine Learning, Finale Doshi-Velez, Been Kim, https://arxiv.org/abs/1702.08608. [4] What does "work" mean? Initially, it made sense to consider memory-related metrics but this would take a very long time and at least looking at my own data, has very little variance. It is possible that this is because I almost always review on my phone and continuously sharpen and re-factor cards. Instead, I keep note of questions I like (subjective but has been useful - more on this below) and anytime I remember a card "out-of-context" (also subjective and maybe not even directly measuring the quality of the card but how widely applicable the card's content is generally or at least to the content I come across). [5] https://learning.subwiki.org/wiki/Examples_in_mathematics [6] I also learn better with examples so maybe this effect won't generalize. [7] https://cognitivemedium.com/trouble_with_definitions/index.html [8] For example, let's consider 2-sided matching where we want to match 2 sides each with full preferences over the other side (e.g. university admissions where universities have a preference ordering over students and vice-versa). For simplicity, let the number of universities be the same as the number of students. Design an algorithm that gives a valid matching. Any bijective function from universities to students (or vice-versa) works. Observe that we totally ignored the preferences of both sides! How do preferences fit in? Can we do "better" if we consider preferences? What does "better" mean? More people are happier (get their favourite or close to their favourite)? I can continue but we build up to Pareto-optimality --> Stability --> {Incentive-compatibility, university/student-{optimal, pessimal}} matchings. The details are not as important as having questions that isolate why we want certain conditions over others and how we got there. [9] For example, "Build an efficient data structure that supports only the insert operation." A data structure that does nothing suffices since we don't support a retrieval operation. [10] Consider the following data structure that supports insert and find_min operations: (skipping the formal description but it initializes an empty array, appends the element when you call insert, and importantly, for this question finds the minimum of the list only when find_min is called). Compare the runtime of this data structure to the more standard min-heap implementation (that always ensures that the root is the minimum element). When, if at all, is it better or worse? The answer depends on how often find_min is called. If it is called less than $\log_2 n$ times then the lazier version is better. (I am not sure if you have the initial same bias but when we were being taught this, the initial thought process was that this "lazier" version must be worse! This is related to amortized vs worst-case analysis and with more context I think helps understand how to evaluate data structures) [11] I don't have separate questions for "motivation" / "why do we care"? These are all framed as we were interested in X, how could you formalize / approach this? Separate questions cover "wrong" answers, routes not taken, and the correct answer. If during review, I think of something new - to me - this is only a good thing, it's something for me to consider why it wasn't done. This is a bit misleading, during review, I usually think: I remember the answer to this question is X but I don't know why it isn't Y. I know they also considered A, B, and C but why not Y? I then flag that card and write a note about Y, and review those cards separately on the weekend. [12] I can't find the essay but Michael writes about the different results when the Wason Selection Task is presented using coloured cards with numbers on one side and as an underage drinking problem. [13] In an interview, Maria Chudnovsky, a mathematics professor at Princeton, was asked how do you remember mathematics (other theorems) and she says, Oh, well I only remember the ones I can prove. The interviewer asks really and she says well you know I remember stuff like we remember anything through stories. For example, how do you remember how old your young cousin is, well you remember when his parents got married then maybe you remember where you were living when they first had kids and so you now roughly remember how old your cousin was. I realize she is talking about networked thought but I find it easier to form networks with stories. Interview: https://youtu.be/iDpcFwsVa0s?t=2163 [14] An example question would be: "We want to remember to be patient with our work and the progress we are making. What relevant story does George Saunders share with Story Club regarding this?" Someone asked him how he writes such wonderfully textured non-stereotypical characters. He said well they don't start out that way but you sit with a piece and edit it for months and months and suddenly the character settles in your mind and you are bored and the character asks is this all? What's next? They want to be more. However, this doesn't always happen and when it does, it doesn't until months into the process and I can't "fast forward" to that point. Story Club with George Saunders: https://georgesaunders.substack.com/ [15] I recognize this is not a "Proof or give a counterexample:" question. I am giving a simple example to illustrate my main concern. [16] K. Anders Ericsson, Peak, Secrets from the new science of expertise - I'm not sure where!
Andy Matuschak
I have tried something like that, using a special card type with multiple fields to hack together alternatives. And, of course, Quantum Country had "application prompts", which were along the same lines. But I haven't explored this concept very carefully at all—just noodling. I really enjoy your observation that a script has the advantage of being able to implement "rules". Do you have templates like the ones you sketch there? If so, I'd be quite curious to hear more about the templates you have and how they vary for different kinds of concepts.
Andy Matuschak
Ah, that's fascinating! Thank you very much for sharing. It's great to hear both that the ML-based generation eliminated pattern matching, and that my "artichoke=carciofo" claim is in fact over-simplified. Everything is fractal.
Andy Matuschak
There's a thousand words or so here: https://andymatuschak.org/prompts/#prompts-in-practice But unfortunately much is still difficult to render legible! It's a good nudge…
Amrit Daswaney
Hi Andy, With respect to repetitive (-ish) cards in the same review session, have you tried linking cards and burying all but one card? I use a custom field that is not displayed during review but is used to store IDs. All associated cards have the same ID (to ensure they are unique, I pick a parent card and give everyone that card's ID but that's not important). Every day an external script inspects the due cards, if more than one card has the same ID, it breaks ties according to a rule. In addition to using ease as a tiebreaker (e.g. prioritize more difficult cards), you can also implement rules like: "definition before application." If I write questions using a specific pre-defined template(s) (e.g. "Define: ", "Prove or give a counterexample: "; you can use a many-to-one mapping, i.e., n different but fixed templates correspond to one type of question) then the script can automatically classify questions for me. As an aside, this also helps with pattern matching because most questions have a similar "shape" since many of the questions have a similar structure.
Russel Simmons
A while back I built a prototype of a new tool for studying Japanese, that is a small step in the direction you're describing. TL;DR It works quite well. I think it's fair to say that "most" Japanese words don't have an exact English equivalent. Many seem to have an equivalent but it's far from exact, in that the usage/nuance is substantially different. You can use English translations as a temporary scaffold, but eventually you need to build a direct connection between the target-language word and internal meaning. So even with learning vocab, the canonical example of SRS, repping target-language -> native-language mappings ("carciofo = artichoke") has significant shortcomings. (This video section touches on this in more detail https://youtu.be/yBmWomPZzPc?t=386) For this reason, a lot of learners use "sentence cards", with a target-language sentence on the front (usu. with one "target" word for study, tho it is unmarked) and on the back any definitions needed to confirm that the target word etc. was understood. The test is "could I read+understand the entire sentence", not "did I regurgitate the target word definition". This has the obvious downside that over time, you tend to pattern match against that specific sentence, and the card becomes less useful. Adding multiple sentence cards for one target word helps, but is considerable work. In my prototype, the "atoms" tracked by the SRS system are Japanese words. To review a word, it searches a large database of sentences (that I pre-filtered via ML for quality for this purpose) to find sentences that use that word, and chooses a not-yet-seen one to show to the user. The user self-grades on whether or not they understood the whole sentence and remembered the target word. (I'm skimming over a number of details, e.g. filtering sentences to only contain other words you probably know, but that's the gist of it.) My experience in using it myself is that it completely solves the issue of pattern matching, and works very well for learning new vocab robustly, in varied contexts. It's also sort of inherently fun to be presented with random sentences that are around the level of my ability. The annoyances are mostly around "bug" cases e.g. where word occurrences are misidentified, etc. So FWIW I am strongly in support of this direction, and skeptical of your reservation. I think the simple primitive of "learn rote A->B mappings" is not a great fit for what most people need for most of their learning, and that there is no sense in staying too attached to it. After all, this new direction is still "spaced repetition", it's just repeating at the level of concepts, not cards!
Andrew Judson
I'm interested in "processes I’ve[you've] put into place for monitoring and revising them over time" - is this something you've addressed in a post or essay?
