What needs to be learned in robot learning? A case study: learning battery insertion from a diagram

Abstract: Manual diagrams are a rich and common knowledge source humans use to learn new skills, but their use for robot learning is still underexplored. A challenge with instruction diagrams is that they communicate task progression in a sparse, qualitative visual format, relying on the learner's prior physical understanding to fill in unmentioned execution details. This thesis investigates the fundamental question of emph{what needs to be learned in robot learning} by exploring the interplay between explicit information extracted from instructions and implicit physical knowledge discovered through practice or prior knowledge, using the cylindrical battery insertion task as a case study. First, we present a pipeline that compiles static 2D instruction diagrams into metrically accurate 3D simulation environments. We use Vision-Language Models (VLMs) to extract qualitative scene topology and contact modes, followed by a geometric optimization solver that certifies and refines metric object dimensions and spatial subgoals. Second, using the reconstructed task keyframes, we manually designed a control strategy on physical hardware. This hardware deployment exposes the limitations of purely explicit instructions and reveals crucial implicit details such as tricks - open-loop primitives exploiting the task's physical properties that yield robustness gains even over naive closed-loop methods. Finally, we investigate whether these implicit physical behaviors can be discovered autonomously by reinforcement and imitation learning. In summary, this thesis demonstrates a paradigm for bridging human data such as is found on YouTube, as well as instructional text, diagrams, and explicit demonstration videos, showing a promising approach to make use of the vast knowledge base of humans. Committee: Christopher Atkeson (advisor) Shubham Tulsiani Yuemin Mao