Robust Visuomotor Policy Learning in Uncertain World Models

Abstract: World models have shown promise in robotics by imagining future outcomes of robot actions directly from high-dimensional sensor observations. However, learning visuomotor policies with world models still faces significant reliability challenges, as imagined futures from world models often diverge from the outcomes that may actually occur. This unreliability arises from uncertainty in learned world models, including epistemic uncertainty, where the model lacks sufficient knowledge in out-of-distribution regions, and aleatoric uncertainty, where intrinsic randomness in the system allows multiple plausible outcomes to arise under the same robot action. This thesis develops methods for learning robust visuomotor policies using uncertain world models by explicitly reasoning about and mitigating these uncertainties. We first introduce UNISafe, an uncertainty-aware latent safety filter for mitigating epistemic uncertainty in learned world models. We propose a principled framework for detecting out-of-distribution world model predictions by quantifying epistemic uncertainty and calibrating an uncertainty threshold with conformal prediction. Moreover, this out-of-distribution detection is incorporated into Hamilton-Jacobi reachability analysis, synthesizing the safety filter to proactively avoid regions where world model predictions are unreliable and thereby achieve robust, safe visuomotor control. We then introduce StressDream, an inference-time steering method for mitigating aleatoric uncertainty in world models. Instead of relying on nominal samples from the world model, StressDream actively steers the imaginations of video world models to expose plausible but critical outcomes of robot actions. This enables more robust policy evaluation by uncovering failure modes of robot actions, as well as improved policy optimization by training policies against challenging but realistic imagined futures. Together, these methods enable visuomotor policies relying on learned but uncertain world models to achieve robust control in complex, uncertain environments with high-dimensional sensor observations by explicitly reasoning about the uncertainties of the learned world model. Thesis Committee: Andrea Bajcsy (chair) Max Simchowitz Jeff Schneider Michelle Zhao