Dear all, 

Next week, we have the pleasure of having Dr. Yohai Bar-Sinai give a talk in the colloquium.

The seminar will be held on Monday, July 22nd at 14:00.

Location: C221.

The title and abstract appear below.

Looking forward to seeing you,

Sagie and Liat

Title:

Grokking as a near-critical phenomenon

Abstract:

The empirical success of neural models far surpasses their theoretical understanding, and
explaining their inner workings is important both for practical reasons and as a fundamental
scientific question. In this talk we will focus on an intriguing phenomenon, grokking, in which a
neural model learns to generalize long after it has completely fit the training data. This sharp
transition between memorization and generalization has been observed in various synthetic and
realistic scenarios. We study grokking in simplified settings of linear or almost linear models,
both in regression and classification. Using tools from statistical mechanics, random matrix
theory and optimization, we show analytically and numerically that some observations of
grokking are linked to the existence of a phase transition, or singularity in the long-time
dynamics of training. In particular, we show that the delayed generalization is a result of slow
dynamical time scales, which generically appear in the vicinity of phase transitions, a
phenomenon known in the physical literature as "critical slowing down". This allows us to derive
concrete scaling predictions about how grokking time depends on dimensionality, dataset size,
and regularization, predictions which apply also for more complicated scenarios due to the
universal properties of criticality.

Short Bio:

Yohai Bar-Sinai is a senior lecturer in the School of Physics in Tel-Aviv University. Before joining
TAU in 2020, he conducted postdoctoral research in the Department of Applied Mathematics at
Harvard, supported by the McDonnell Postdoctoral Fellowship, and then spent two years at
Google Research (Israel). He holds a Bachelor's degree in Math and Physics from HUJI, a
Master's degree from ENS and UPMC universities in Paris (FPGG scholarship) , and a PhD in
theoretical physics from the Weizmann Institute. His interdisciplinary research focuses on using
statistical physics, computational physics, and machine learning to investigate complex
non-linear phenomena.