ArXiv Preprint – ControlLLM: Augment Language Models with Tools by Searching on Graphs

In this episode we discuss ControlLLM: Augment Language Models with Tools by Searching on Graphs
by Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang. The paper introduces a framework called ControlLLM that enhances large language models (LLMs) by allowing them to use multi-modal tools for complex tasks. ControlLLM addresses challenges such as ambiguous prompts, inaccurate tool selection, parameterization, and inefficient tool scheduling. It consists of three components: a task decomposer, a Thoughts-on-Graph paradigm, and an execution engine. The framework is evaluated on tasks involving image, audio, and video processing, and it outperforms existing methods in terms of accuracy, efficiency, and versatility.