arxiv preprint – Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

In this episode, we discuss Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning by Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, Lijuan Wang. This paper introduces LRV-Instruction, a diverse dataset designed for visual instruction tuning with a focus on mitigating hallucination in large multi-modal models (LMMs). The dataset contains 400k visual instructions generated by GPT4 and includes negative as well as positive instructions to increase robustness, structured at different semantic levels of complexity. The authors propose GAVIE, an evaluation method that mimics human expert assessment without needing annotated ground truth, and demonstrate that training on the LRV-Instruction dataset, with an appropriate mix of positive and negative samples, reduces LMM hallucinations and improves performance across several tasks.