Andy Matuschak
patreon
Discussion: AI-generated ad-hoc UIs and malleable software; Saturday, November 25th @ 9AM PST
🕑 Added 2023-11-21 18:13:46 +0000 UTCtl;dr: Join me (via Google Meet) this Saturday, November 25, at 9 AM PST [GCal] for a discussion of the intersections between two fascinating research agendas for the future of personal computing: malleable software, which aspires to let users compose and tailor interfaces; and AI-generated ad-hoc UIs, which aspires to let users generate interfaces anew, on-demand.
One of my favorite current lines of human-computer interaction research pursues the dream of malleable software. Rather than being "stuck" within the silos of traditional apps, these systems imagine that users could spontaneously recombine favorite parts of different pieces of software and, when necessary, modify those pieces to suit their idiosyncratic preferences. This dream stretches back to Smalltalk and the Dynabook, but Klokmose's Webstrates inspired much recent work. Here's a 2021 video overview of that lab's work; see also Potluck for related ideas in the context of text editors, Mirrorverse for newer work with malleable video interfaces, and Dynamicland for a physicalized approach.
Recently, though, large language models have focused new attention on another old idea: AI-generated ad-hoc UIs. If the central idea of malleable software is that different people in different situations need different software, one way to achieve that is through flexible composition and tailoring; but another way is to generate the appropriate software and UIs as needed. "Apps" become throwaway objects, like an afternoon's arrangements of papers on a table. For past work, see e.g. programming by example. More recently, we've seen ad-hoc UIs generated in the context of chats (Dot, at least aspirationally), search (Perplexity), and whiteboards (tldraw). None of these really substitutes for the full aspirations of the malleable software projects… yet.
Some questions I'd like to understand better:
- How might users refine and expand the behavior of AI-generated UIs?
- How might AI-generated UIs express their interpretation of user intent; or, how might users understand how the AI-generated UIs will behave?
- What unique challenges will we face in "mixed-initiative" UI programming, in which part of the UI is machine-generated, and part is hand-created or modified by an expert?
- Is the range of AI-generated ad-hoc UIs bounded by the intelligence of the underlying LLMs? Are there more fundamental limitations?
- If AI-generated UIs are successful, what happens to the malleable software agenda? Is it fully subsumed? Or do its goals shift?—how? Do its ideas somehow enable part of the AI-generated UI agenda?
Please read/view some of the links above, and bring your own favorites to share, as well as questions and ideas! I'll record the discussion and share it here afterward.
Comments
Jamie Dubs
couldn't join but also excited for the recording - this is an active area of work for me and my startup
Andy Matuschak
Gosh, I'm sorry, Gabriel! I didn't see a notification. Try again if you're game? https://meet.google.com/mun-xast-pva
Kronopath
Stuck around for a half hour but was never let into the meeting. I suppose I’ll have to catch up with the recording/notes.
