ICML 2023 – Learning-Rate-Free Learning by D-Adaptation

In this episode we discuss Learning-Rate-Free Learning by D-Adaptation
by Aaron Defazio, Konstantin Mishchenko. The paper introduces D-Adaptation, a learning-rate-free approach for setting the learning rate in convex minimization problems. It achieves the optimal rate of convergence without additional evaluations per step. The method is shown to match hand-tuned learning rates in diverse machine learning problems.