Andy Matuschak
patreon
(Recording) Discussion: AI-generated ad-hoc UIs and malleable software; Saturday, November 25th @ 9AM PST
🕑 Added 2023-11-26 19:33:45 +0000 UTCThank you all for a very interesting discussion! See original post for premise.
Please don't share this recording publicly.
And here's the chat transcript with some links which were mentioned:
00:20:19.962,00:20:22.962
Taylor Rogalski: quip had a nice chat-as-changelog-alongside-artifact pattern https://www.smartsheet.com/sites/default/files/2020-02/5cba0c3ca6e1a17be8045cc0.jpg
00:27:12.684,00:27:15.684
Andy Matuschak: This is my #1 favorite UI + AI paper! https://www.pnas.org/doi/10.1073/pnas.1807184115
00:27:18.199,00:27:21.199
Andy Matuschak: (the Jeff Heer one)
00:45:41.614,00:45:44.614
Maggie Appleton: I forget if mentioned in the call notes, but Adpet's latest release very relevant here: https://www.adept.ai/blog/experiments
00:46:44.369,00:46:47.369
Maggie Appleton: At the moment it's essentially keyboard maestro in the browser, but the unreleased stuff they're working on much more interesting and includes lots of on the fly UIs: https://www.youtube.com/watch?v=PAy_GHUAICw
00:46:55.387,00:46:58.387
Andy Matuschak: ^ this is new to meβthank you, will watch!
00:50:26.475,00:50:29.475
Andy Matuschak: https://dynamicland.org/links/2023-04-27
00:55:15.232,00:55:18.232
Tim Riherd: https://www.sidefx.com/tutorials/foundations-overview/
00:55:17.112,00:55:20.112
Taylor Rogalski: https://en.wikipedia.org/wiki/5D_Chess_with_Multiverse_Time_Travel
00:57:02.054,00:57:05.054
Maggie Appleton: @Geoffrey have you written anywhere about those hypothetical primitives of a world where we're generating UIs? Want to think about that more
00:57:30.416,00:57:33.416
Geoffrey Litt: hmm i dont think so; maybe soon :)
00:57:45.575,00:57:48.575
Maggie Appleton: π please do
00:57:51.926,00:57:54.926
Tim Riherd: +1
00:58:20.816,00:58:23.816
David D: https://github.com/inkandswitch/cambria-project
01:00:59.230,01:01:02.230
Maggie Appleton: Walking treadmill is a sad solution to that dream ;)
01:01:29.100,01:01:32.100
Tim Urian: for real!
01:01:53.136,01:01:56.136
Geoffrey Litt: Currently 1% of people make software; getting to 10% would be huge progress. 100% not necessary!
01:01:58.290,01:02:01.290
Andy Matuschak: ++
01:03:11.572,01:03:14.572
Gabriel Nunes: The problem of adapting data between interfaces for malleable software was the subject of my masterβs thesis (not AI though): https://www.kronopath.com/blog/atypical-a-differently-optimized-type-system/
01:03:17.299,01:03:20.299
Maggie Appleton: ++ thanks for hosting
01:03:18.279,01:03:21.279
Daniel Dosen: Thanks!
The org-mode-for-sermons video Taylor was referring to: https://www.youtube.com/watch?v=Q8AqHdZTgNI
Comments
Parker
I found this 'how to generate UIs with LLMs' video fascinating! Thought I'd share here: https://www.youtube.com/watch?v=8w0hUcQSDy8
Andy Matuschak
That does seem right! It reminds me the way that Bret's UI in "Drawing Dynamic Visualizations" needs to surface and disambiguate multiple possible interpretations: https://vimeo.com/66085662
Benjamin Wheeler
I missed this, but enjoyed the video! One element that I'd like to discuss more is the interplay between automation and inspectability. Looking at Adept Workflows demo videos, for example, I kept thinking about what the user might do when these go wrong. Of course, there might be a chat interface where you could say, for example, "No, no, Workflows -- when you are contesting an insurance claim rather than filing one, go to this URL instead!" But this brings you to the problem of poor addressability in verbal interfaces that several people named in this session. It seems to me like there needs to be a visual representation of the major steps and logical decision-making elements, with some ability to replay decisions in a sandbox environment, and intervene at the point where the agent should have behaved differently.
