Arxiv Paper – Video Instruction Tuning With Synthetic Data

In this episode, we discuss Video Instruction Tuning With Synthetic Data by Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, Chunyuan Li. The paper proposes a high-quality synthetic dataset, LLaVA-Video-178K, to address the challenge of developing large multimodal video models by improving video instruction-following tasks through detailed captioning and question-answering. Using this dataset and existing tuning data, the authors develop a novel model, LLaVA-Video, which demonstrates strong performance across various video benchmarks. They plan to release the dataset, generation pipeline, and model checkpoints to the public.