arxiv Preprint – On the Connection between Pre-training Data Diversity and Fine-tuning Robustness

In this episode we discuss On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
by Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ludwig Schmidt, Ali Farhadi. The paper investigates the impact of different factors in pre-training data on the robustness of fine-tuned models. The authors find that the primary factor influencing robustness is data quantity, whereas other factors like label space, image diversity, and data domains have limited significance. The study uses pre-training distributions from natural and synthetic data sources and focuses on the iWildCam-WILDS distribution shift to test downstream robustness.