arxiv preprint – Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

In this episode, we discuss Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity by Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, Joan Serrà. The paper introduces MaskVAT, a video-to-audio generative model that utilizes a masked generative model alongside a high-quality general audio codec to achieve superior audio quality, semantic matching, and temporal synchronization. MaskVAT effectively addresses the synchronization issues in previous V2A models without compromising on audio quality. Empirical results demonstrate its capability to generate well-synchronized and high-quality audio that aligns with visual actions, competing with state-of-the-art non-codec generative models.