arxiv preprint – Faith and Fate: Limits of Transformers on Compositionality

In this episode we discuss Faith and Fate: Limits of Transformers on Compositionality
by Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi. The paper examines the limitations of large language models (LLMs) like Transformers on compositional tasks. It explores their performance on three specific tasks and finds that Transformers tend to solve these tasks by reducing them to linearized subgraph matching rather than demonstrating systematic problem-solving skills. The paper also discusses how Transformers’ performance decreases as task complexity increases.