arxiv preprint – Evaluating Text-to-Visual Generation with Image-to-Text Generation

In this episode, we discuss Evaluating Text-to-Visual Generation with Image-to-Text Generation by Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, Deva Ramanan. The paper introduces VQAScore, a novel metric for evaluating the alignment of generated images to text prompts, utilizing a visual-question-answering model to score the relevance of images to prompts based on a simple yes-or-no question. Unlike existing metrics, the proposed VQAScore effectively handles complex prompts, demonstrating superior performance across numerous benchmarks, even when compared to proprietary models like GPT-4V. Additionally, the paper presents GenAI-Bench, a challenging new benchmark consisting of compositional text prompts and human ratings, and provides open-source access to their data and models to facilitate further research in text-to-visual generation evaluations.