Making is Decision Making

Abstract: Human creation of high-quality content requires making decisions - from coarse, high-level decisions about content and style, to precise low-level decisions about the color of an individual pixel. Modern generative AI promises to serve a collaborative assistant capable of executing design decisions specified in simple text prompts into high-quality content. Yet today's AI systems are poor collaborators. They frequently misinterpret user intent, while users lack a predictive conceptual model of how an AI will interpret a prompt or why it produces a particular result. Without shared conceptual grounding, collaboration devolves into trial and error, with users repeatedly rewriting prompts, using the AI to generate a result and then adjusting the prompt to try again, in the hope of obtaining the desired outcome. In this talk I'll argue that for generative AI to fulfill its promise we must develop techniques and interfaces that enable users and AI models to establish shared conceptual grounding. I'll outline two complementary research challenges; First, we must identify the concepts that human creators commonly use when reasoning about and communicating within a content creation domain. Here, I'll show how we might extend methods from cognitive psychology to elicit, represent, and analyze domain-specific conceptual structures. Second, we must develop interfaces for teaching these concepts to AI models. While machine learning is rapidly advancing methods for teaching AI new concepts, I'll show how adapting these techniques to the diverse domains of human content creation requires new interaction techniques that support the way people think, communicate, and create. Finally, I'll demonstrate a few implementations of these ideas that we have developed in our group at Stanford. Bio: Maneesh Agrawala is the Forest Baskett Professor of Computer Science and Director of the Brown Institute for Media Innovation at Stanford University. He is also a consulting AI Scientist at Roblox. He works on computer graphics, human computer interaction and visualization. His focus is on investigating how cognitive design principles can be used to improve the effectiveness of audio/visual media. The goals of this work are to discover the design principles and then instantiate them in both interactive and automated design tools. Honors include an Okawa Foundation Research Grant (2006), an Alfred P. Sloan Foundation Fellowship (2007), an NSF CAREER Award (2007), a SIGGRAPH Significant New Researcher Award (2008), a MacArthur Foundation Fellowship (2009), an Allen Distinguished Investigator Award (2014) and induction into the SIGCHI Academy (2021). He was named an ACM Fellow in 2022.