Category: Uncategorized

In this episode we discuss TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise by Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, Zhaohui Hou, Zhiyuan Huang, Shaoqing Lu, Ding Liang, Mingjie Zhan. The paper introduces TeacherLM, a series of…

In this episode we discuss MM-VID: Advancing Video Understanding with GPT-4V(ision) by Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, Ce Liu, Lijuan Wang. The paper introduces MM-VID, a system that incorporates GPT-4V with vision, audio, and speech experts to enhance video understanding.…

In this episode we discuss Zephyr: Direct Distillation of LM Alignment by Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf. The paper introduces ZEPHYR, a language model that focuses on aligning with user…

In this episode we discuss ControlLLM: Augment Language Models with Tools by Searching on Graphs by Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang. The paper introduces a framework called ControlLLM that enhances large language models (LLMs) by allowing them to use multi-modal…

In this episode we discuss Talk like a Graph: Encoding Graphs for Large Language Models by Bahare Fatemi, Jonathan Halcrow, Bryan Perozzi. The paper discusses the encoding of graph-structured data for use in large language models (LLMs). It investigates different graph encoding methods, the nature of graph tasks, and the structure of the graph, and…

In this episode we discuss AgentTuning: Enabling Generalized Agent Abilities for LLMs by Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, Jie Tang. AgentTuning is a method that enhances the agent abilities of large language models (LLMs) while maintaining their general capabilities. It introduces AgentInstruct, a lightweight instruction-tuning dataset, and combines…

In this episode we discuss Jailbreaking Black Box Large Language Models in Twenty Queries by Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong. The paper introduces an algorithm called Prompt Automatic Iterative Refinement (PAIR) that generates “jailbreaks” for large language models (LLMs) using only black-box access. PAIR leverages an attacker…

In this episode we discuss Matryoshka Diffusion Models by Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, Navdeep Jaitly. The paper introduces Matryoshka Diffusion Models (MDM) for high-resolution image and video synthesis. The authors propose a diffusion process that denoises inputs at multiple resolutions simultaneously. They also present a NestedUNet architecture that combines features and…

In this episode we discuss An Image is Worth Multiple Words: Learning Object Level Concepts using Multi-Concept Prompt Learning by Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, Philip Teare. The paper proposes a framework called Multi-Concept Prompt Learning (MCPL) to address the challenge of integrating multiple object-level concepts within one scene using prompt learning.…

In this episode we discuss Retrieval meets Long Context Large Language Models by Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro. This paper compares two methods for handling long context in large language models (LLMs): retrieval-augmentation and extending the context window. The study…

In this episode we discuss Contrastive Prefence Learning: Learning from Human Feedback without RL by Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, Dorsa Sadigh. Traditional approaches to Reinforcement Learning from Human Feedback (RLHF) assume that human preferences align with reward, but recent research suggests they align with regret under…

In this episode we discuss BitNet: Scaling 1-bit Transformers for Large Language Models by Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei. The paper introduces BitNet, an architecture for large language models that addresses concerns about energy consumption and deployment challenges. BitNet utilizes…

In this episode we discuss Automatic Prompt Optimization with “Gradient Descent” and Beam Search by Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, Michael Zeng. The paper introduces ProTeGi, a method for improving prompts used in large language models. It utilizes mini-batches of data to generate “natural language gradients” that provide feedback…

In this episode we discuss Understanding Retrieval Augmentation for Long-Form Question Answering by Hung-Ting Chen, Fangyuan Xu, Shane A. Arora, Eunsol Choi. This paper examines the impact of retrieval-augmented language models on long-form question answering. The authors compare the generated answers using the same evidence documents to analyze how retrieval augmentation affects different language models.…

In this episode we discuss On the Connection between Pre-training Data Diversity and Fine-tuning Robustness by Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ludwig Schmidt, Ali Farhadi. The paper investigates the impact of different factors in pre-training data on the robustness of fine-tuned models. The authors find that the primary factor influencing robustness is data quantity,…

In this episode we discuss Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness by Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, Kristian Kersting. The paper proposes a strategy called Fair Diffusion to address biases in text-to-image models after deployment. This approach allows users to adjust biases in any direction based…

In this episode we discuss In-Context Pretraining: Language Modeling Beyond Document Boundaries by Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, Mike Lewis. This paper introduces a new approach called IN-CONTEXT PRETRAINING for training large language models. It addresses the limitation of current LM…

In this episode we discuss Sigmoid Loss for Language Image Pre-Training by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. The paper introduces a pairwise Sigmoid loss for Language-Image Pre-training (SigLIP), which operates on image-text pairs and allows for scaling up batch size without the need for global pairwise similarities. By combining SigLIP with Locked-image…

In this episode we discuss Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading by Howard Chen, Ramakanth Pasunuru, Jason Weston, Asli Celikyilmaz. The paper introduces MEMWALKER, an approach to address the limitations of the self-attention mechanism in large language models (LLMs) when processing long sequences. MEMWALKER treats the LLM as an interactive…

In this episode we discuss HyperAttention: Long-context Attention in Near-Linear Time by Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, Amir Zandieh. The paper introduces “HyperAttention,” an approximate attention mechanism for handling long contexts in Large Language Models (LLMs). It proposes two parameters to measure problem difficulty and presents a linear time…

In this episode we discuss InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists by Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony Philippakis, Ahmed M. Alaa. The paper proposes a unified language interface for computer vision tasks that allows for task execution through natural language instructions. The approach involves training a text-to-image diffusion model using a…

In this episode we discuss Large Language Models Cannot Self-Correct Reasoning Yet by Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. The paper explores the effectiveness of self-correction in Large Language Models (LLMs) for improving the accuracy and appropriateness of generated content. It specifically focuses on the…

In this episode we discuss Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution by Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, Tim Rocktäschel. The paper presents PROMPTBREEDER, a method for evolving and adapting prompts for Large Language Models (LLMs) in order to enhance their reasoning abilities. It uses an LLM to mutate a population of task-prompts…

In this episode we discuss Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation by Eric Zelikman, Eliana Lorch, Lester Mackey, Adam Tauman Kalai. The paper presents a method called Self-Taught Optimizer (STOP) that utilizes a language model to enhance a scaffolding program for solving optimization problems. The language model suggests self-improvement strategies like beam search, genetic…

In this episode we discuss Tree of Thoughts: Deliberate Problem Solving with Large Language Models by Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan. The authors of this paper introduce a framework called “Tree of Thoughts” (ToT) to enhance language model inference. The ToT framework allows language models…