Arxiv paper – DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion

In this episode, we discuss DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion by Qitao Zhao, Amy Lin, Jeff Tan, Jason Y. Zhang, Deva Ramanan, Shubham Tulsiani. The paper introduces DiffusionSfM, a novel data-driven framework that directly infers 3D scene geometry and camera poses from multi-view images using a transformer-based denoising diffusion model. It represents scene geometry and camera parameters as pixel-wise ray origins and endpoints in a global coordinate frame and incorporates specialized mechanisms to handle challenges like missing data and unbounded coordinates. Experiments on synthetic and real datasets show that DiffusionSfM outperforms existing classical and learning-based SfM methods while effectively modeling uncertainty.