Saddle-to-saddle dynamics in diagonal linear networks

Abstract: When training neural networks with gradient methods and small weight initialisation, peculiar learning curves are observed: the training initially shows minimal progress, which is then followed by a sudden transition where a new “feature” is rapidly learned. This pattern is commonly known as incremental learning. In this talk, I will demonstrate that we can comprehensively understand this phenomenon within the context of a simplified network architecture. In this setting, we can establish that the gradient flow trajectory transitions from one saddle point of the training loss to another. The specific saddle points visited, as well as the timing of these transitions, can be determined using a recursive algorithm that is reminiscent of the Homotopy method used in computing the Lasso path.

Bio: Nicolas Flammarion is a tenure-track assistant professor in computer science at EPFL. Prior to that, he was a postdoctoral fellow at UC Berkeley, hosted by Michael I. Jordan. He received his PhD in 2017 from Ecole Normale Superieure in Paris, where he was advised by Alexandre d’Aspremont and Francis Bach. In 2018 he received the prize of the Fondation Mathematique Jacques Hadamard for the best PhD thesis in the field of optimization and in 2021, he was one of the recipients of the NeurIPS Outstanding Paper Award. His research focuses on learning problems at the intersection of machine learning, statistics, and optimization. He aims to develop algorithmic and theoretical tools that improve our understanding of machine learning and increase its robustness and usability.