Lecturers
Main Lecturers
- Federica Gerace (Università di Bologna, Italy)
Statistical Mechanics of Learning: From Random Data to Transferable Representations
- Boris Hanin (Princeton University, USA)
Optimization and Sampling in Deep Learning Through Non-equilirbrium thermodynamicAbstract
These lectures will survey new approaches to optimizer design and the study of diffusion models in deep learning. On the side of optimizer design, I will describe a recently constructed first order optimization scheme that provably decreases the sharpness of loss minima under minimal assumptions on the loss. This optimizer exhibits a separation of time-scales and drives a slow relaxation along the manifold of minima towards width regions in parameter space. For diffusion models, I will explain a new approach to mapping inference with score-based diffusion models to adiabatic transport through the construction of a new class of Schrodinger operators, which I call Score Hamiltonians. These Hamiltonians have as their ground states the density underlying the score that created them. I will explain how using standard spectral techniques gives new tight bounds on the quality of sampling with score-based diffusion models.
Matteo Negri (LPTM, CY Cergy Paris Université, France)
Modern Associative Memories for Modern AIAbstract
Associative memories were born out of neuroscience-inspired models of neural computation, and they turned out to be the progenitor of much of modern deep learning. After the foundational results of the 1980s and 90s, their theoretical development stalled — until Dense Associative Memories and their connections to attention mechanisms sparked a renaissance in the last ten years. This course traces that arc, from the classical foundations to the modern perspective.
We develop the theory from the energy-based formulation and Hebbian learning through Hopfield networks, their capacity limitations, and how Dense Associative Memories overcome them, the key step that reconnected the field to modern deep learning. From there we cover the typical-case analysis of networks trained on structured data, showing a prototypical “creativity phase transition”. Finally, we turn to the Gardner perspective on associative memories, which suggests to the interpretation of transformers as a pseudo-likelihood dynamics of an associative memory.
Topic Speakers
- Nicolas Brunel (Università Bocconi, Milano, Italy)
Storage and flexible retrieval of sequences in recurrent networksAbstract
Sequential neural activity is observed in multiple brain areas during execution of temporally structured movements. Networks with temporally asymmetric Hebbian rules can generate patterns of sequential activity with characteristics that are qualitatively consistent with experimental observations. However, in these models sequential activity is retrieved at a fixed speed. Here, we investigate the effects of a heterogeneity of plasticity rules on network dynamics. In a model in which neurons differ by the degree of temporal symmetry of their plasticity rule, retrieval speed can be controlled by varying external inputs to the network. Such networks can also switch flexibly between persistent and sequential activity, and can naturally generate separate ‘preparatory’ and ‘execution’ activity patterns with appropriate external inputs.
- Vittorio Erba (EPFL, Lausanne, Switzerland)
Exact asymptotics of quadratic neural networksAbstract
In the last decade, two sets of empirical observations have been put forth to try to understand and model how neural networks learn: spectral properties of the learned weights, and scaling laws for the generalisation error. In this talk I will introduce a solvable class of models, bilinear index models, in which both spectra and scaling laws can be analytically studied in a feature learning regime. This class of models includes one-hidden-layer extensive-width neural networks with quadratic activations and softmax attention layers as used in transformers, allowing to explore analytically spectra and scaling laws in non-trivial architectures. - Mauro Pastore (ICTP, Trieste, Italy)
Statistical mechanics of deep Bayesian neural networks near interpolation
Lecture
Abstract
In this contribution, I will present recent results on the supervised learning of a multi-layer perceptron. Importantly, (i) its width scales as the input dimension, making it more prone to feature learning than ultra wide networks, and more expressive than narrow ones; and (ii) the number of trainable parameters and data are comparable, which forces the model to adapt to the task. We will consider the matched teacher-student setting. A rich phenomenology emerges with various learning transitions. With enough data, optimal performance is attained through the model’s “specialisation” towards the target, but it can be hard to reach for training algorithms which get attracted by sub-optimal solutions predicted by the theory. Specialisation occurs inhomogeneously across layers, propagating from shallow towards deep ones, but also across neurons in each layer. Furthermore, deeper targets are harder to learn. The analysis provides insights on how the depth, non-linearity and finite (proportional) width influence neural networks in the feature learning regime.
References: J. Barbier, F. Camilli, M.-T. Nguyen, MP and R. Skerk. “Statistical physics of deep learning: Optimal learning of a multilayer perceptron near interpolation”. arXiv:2510.24616 (accepted on Phys. Rev. X)
- Rosilari Bellacosa (SynDiag, Torino, Italy)
To be announced - Francesco D’Amico
Lecture