arxiv preprint – Memory Mosaics

In this episode, we discuss Memory Mosaics by Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, Léon Bottou. Memory Mosaics are collective networks designed for prediction tasks, utilizing associative memories in a collaborative manner. These networks offer a simpler and more transparent alternative to transformers, maintaining comparable abilities in compositional learning and learning in context. The effectiveness of Memory Mosaics is established through medium-scale language modeling experiments, outperforming or matching the performance of transformers.