arxiv preprint – 3D-LLM: Injecting the 3D World into Large Language Models

In this episode we discuss 3D-LLM: Injecting the 3D World into Large Language Models
by Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan. The paper proposes a new model called 3D-LLMs that integrates the 3D physical world into language models, allowing them to perform various 3D-related tasks such as captioning, question answering, and navigation. The authors employ three prompting mechanisms to collect a large dataset of 3D-language data efficiently and use a 3D feature extractor and 2D VLMs as the backbone for training the model. The experimental results demonstrate that the 3D-LLMs outperform existing baselines in terms of performance and capabilities.