Dear all,
Next week, we have the pleasure of having Dr. Yohai Bar-Sinai give a talk in the colloquium.
The seminar will be held on Monday, July 22nd at 14:00. Location: C221.
The title and abstract appear below.
Looking forward to seeing you, Sagie and Liat
*Title:* Grokking as a near-critical phenomenon
*Abstract:*
The empirical success of neural models far surpasses their theoretical understanding, and explaining their inner workings is important both for practical reasons and as a fundamental scientific question. In this talk we will focus on an intriguing phenomenon, grokking, in which a neural model learns to generalize long after it has completely fit the training data. This sharp transition between memorization and generalization has been observed in various synthetic and realistic scenarios. We study grokking in simplified settings of linear or almost linear models, both in regression and classification. Using tools from statistical mechanics, random matrix theory and optimization, we show analytically and numerically that some observations of grokking are linked to the existence of a phase transition, or singularity in the long-time dynamics of training. In particular, we show that the delayed generalization is a result of slow dynamical time scales, which generically appear in the vicinity of phase transitions, a phenomenon known in the physical literature as "critical slowing down". This allows us to derive concrete scaling predictions about how grokking time depends on dimensionality, dataset size, and regularization, predictions which apply also for more complicated scenarios due to the universal properties of criticality.
*Short Bio:* Yohai Bar-Sinai is a senior lecturer in the School of Physics in Tel-Aviv University. Before joining TAU in 2020, he conducted postdoctoral research in the Department of Applied Mathematics at Harvard, supported by the McDonnell Postdoctoral Fellowship, and then spent two years at Google Research (Israel). He holds a Bachelor's degree in Math and Physics from HUJI, a Master's degree from ENS and UPMC universities in Paris (FPGG scholarship) , and a PhD in theoretical physics from the Weizmann Institute. His interdisciplinary research focuses on using statistical physics, computational physics, and machine learning to investigate complex non-linear phenomena.
Reminder, this is happening today.
On Mon, 15 Jul 2024, 20:31 Sagie Benaim, sagie.benaim@mail.huji.ac.il wrote:
Dear all,
Next week, we have the pleasure of having Dr. Yohai Bar-Sinai give a talk in the colloquium.
The seminar will be held on Monday, July 22nd at 14:00. Location: C221.
The title and abstract appear below.
Looking forward to seeing you, Sagie and Liat
*Title:* Grokking as a near-critical phenomenon
*Abstract:*
The empirical success of neural models far surpasses their theoretical understanding, and explaining their inner workings is important both for practical reasons and as a fundamental scientific question. In this talk we will focus on an intriguing phenomenon, grokking, in which a neural model learns to generalize long after it has completely fit the training data. This sharp transition between memorization and generalization has been observed in various synthetic and realistic scenarios. We study grokking in simplified settings of linear or almost linear models, both in regression and classification. Using tools from statistical mechanics, random matrix theory and optimization, we show analytically and numerically that some observations of grokking are linked to the existence of a phase transition, or singularity in the long-time dynamics of training. In particular, we show that the delayed generalization is a result of slow dynamical time scales, which generically appear in the vicinity of phase transitions, a phenomenon known in the physical literature as "critical slowing down". This allows us to derive concrete scaling predictions about how grokking time depends on dimensionality, dataset size, and regularization, predictions which apply also for more complicated scenarios due to the universal properties of criticality.
*Short Bio:* Yohai Bar-Sinai is a senior lecturer in the School of Physics in Tel-Aviv University. Before joining TAU in 2020, he conducted postdoctoral research in the Department of Applied Mathematics at Harvard, supported by the McDonnell Postdoctoral Fellowship, and then spent two years at Google Research (Israel). He holds a Bachelor's degree in Math and Physics from HUJI, a Master's degree from ENS and UPMC universities in Paris (FPGG scholarship) , and a PhD in theoretical physics from the Weizmann Institute. His interdisciplinary research focuses on using statistical physics, computational physics, and machine learning to investigate complex non-linear phenomena.
-
Sagie Benaim