Schedule
Day 1 · Mon 9 Nov · Likely
Orientation, and the science of deep learning
What it means for a system trained by gradient descent to pursue the goals we intended, and why nobody can yet show that one does. The session states the alignment problem in the language of optimisation and probability, separates specifying the right objective from getting training to produce it, and marks the places where a theorem rather than an experiment is what is missing.
How a frontier language model is built: pretraining on filtered web text, fine-tuning on human and verifiable feedback, then the evaluations, red-teaming and runtime monitoring that guard deployment. For each stage, what a lab can actually change and where misalignment has appeared anyway. Little mathematics; the aim is a precise picture of what current methods do and where they are known to break.
Three things classical theory says should not work and deep networks do anyway: approximating high-dimensional functions efficiently, generalising while vastly overparameterised, and training by gradient descent on non-convex landscapes that are hard in the worst case. The session sets out each mystery, the structure in real data that seems to make it tractable, and the complexity measures and hardness results involved.
A short talk on timelines opens: what the forecasts say, why people in the field act as if the deadline is close, and what that urgency asks of a mathematician choosing a problem. Then open questions on the alignment ecosystem, who works on what, and what alignment is a problem of.
Day 2 · Tue 10 Nov · Likely
Solomonoff induction, and computational mechanics
How an idealised agent should predict and act. A Bayesian mixture over all computable environments, weighted by Kolmogorov complexity, gives a formal Occam's razor with bounded total prediction error. Extending prediction to action gives AIXI, the Bayes-optimal reinforcement learner, with its optimality theorems and their limits. The mathematics is computability, Kolmogorov complexity and KL divergence.
What must any optimal predictor of a stochastic process remember about the past? Computational mechanics answers with causal states and the belief states of a hidden Markov model, whose geometry in the probability simplex is fixed by the process alone. The session builds these objects from scratch, using stochastic matrices, Bayes' rule and the simplex, and shows why generating data is easier than predicting it.
From theory to trained networks: transformers trained on data from a hidden Markov process represent the belief-state geometry linearly in their residual stream, where a probe can read it off. The session asks what this says about what a next-token predictor has learned, how far the picture extends beyond small toy processes, and what stands between it and interpretability at scale.
A conversation about where computational mechanics came from and where it is going: which structure any optimal predictor of a process must carry, and how far the evidence that trained networks carry it extends. Where it meets interpretability, world models and the physics of complex systems, and which questions a mathematician could take up.
Day 3 · Wed 11 Nov · Likely
Classical learning theory, and singular learning theory
The classical picture of learning: capacity, uniform convergence, the bias–variance trade-off, and the generalisation bounds that follow. Then the observations that broke it: overparameterised models that interpolate their training data and still generalise, double descent, and benign overfitting. The session asks what a theory of generalisation must look like once interpolation is the normal regime, using probability and high-dimensional statistics.
Neural networks are singular statistical models: the map from parameters to functions is degenerate, so the Fisher information is singular and classical asymptotics fail. Singular learning theory replaces them with Watanabe's free energy formula, in which the local learning coefficient, a birational invariant of the loss landscape, controls Bayesian learning and its phase transitions. The mathematics is algebraic geometry and Bayesian statistics.
How does a trained model respond to small changes in its data or its loss? Susceptibilities, borrowed from statistical mechanics, are the linear-response coefficients of the Bayesian posterior, and they localise which parts of a network react to which parts of the data. The session develops the formalism, its relation to the local learning coefficient, and how these quantities are estimated in practice.
What changes in a mathematician's working day when coding agents, proof assistants and shared research channels are cheap and plentiful, shown through live examples rather than described. Which parts of one's own work to hand to an agent, which to keep, and what the bottleneck becomes once formalisation and exploration are no longer the slow steps.
Day 4 · Thu 12 Nov · Likely
Scaling laws, statistical mechanics, and interpretability
Loss falls as a power law in model size, data and compute, with exponents stable across orders of magnitude that nobody has derived from first principles. The session presents the empirical scaling laws and one proposed explanation: networks learn discrete skills in order of frequency, so a Zipfian distribution of skills produces power-law scaling. What this predicts about emergence, and what would falsify it.
Learning as a thermal system: the Bayesian posterior as a Boltzmann distribution, the loss as an energy, and the free energy, partition function and phase transitions that follow. The session covers the large-width limits where networks become Gaussian processes or kernel machines, the mean-field scaling where they learn features, and what statistical field theory can say about grokking, double descent and scaling laws.
Can the tools of statistical physics give a complete, rather than piecemeal, account of what a trained network computes? The session asks why collective phenomena are the right language for how networks organise features, what renormalisation-style ideas might offer for bounding behaviour across scales, and what an ambitious interpretability programme would have to deliver before it could yield a safety guarantee.
Day 5 · Fri 13 Nov · Likely
Learning dynamics
How does a deep network's function change as gradient descent runs? An introduction to the dynamics of learning: what depth does to the trajectory, why learning proceeds in stages separated by plateaus, and why what a network learns first shapes what it can learn later. The session uses analytically solvable models to show the principles that experiments on large networks then confirm.
Two exactly solvable regimes of training. In deep linear networks, gradient flow moves saddle to saddle and learns features in order of strength; this is the implicit bias of the rich regime. At infinite width, a network at initialisation is a Gaussian process, training is kernel regression under the neural tangent kernel, and spectral bias gives a sharp theory of generalisation. Between them lies the lazy-to-rich transition.
The joint session runs on into its second hour, then working time at the board with the lecturers. Possible starting points: derive the gradient-flow equations for a two-layer linear network and find its saddles; compute the Gaussian-process and tangent kernels of a simple architecture; or test the spectral-bias prediction of kernel regression on a toy dataset. Bring the question you did not get to ask.
Four pictures of why networks learn: classical learning theory, training dynamics, singular learning theory and Bayesian learning. A moderated discussion on where they describe one phenomenon in different languages, where they make different predictions, and which gaps between them a mathematician could close.
Day 6 · Mon 16 Nov · Likely
Reinforcement learning, and world models
Why is the goal of an agent a sum of discounted rewards? Starting from a preference relation over trajectories and a minimal set of axioms, the von Neumann–Morgenstern theorem gives a utility function, and one further temporal axiom makes it a recursive discounted reward. The session derives the reward hypothesis this way, examines where its assumptions fail, and treats potential-based reward shaping as a first consequence.
The reinforcement learning actually applied to language models: a reward model fitted to human comparisons, or a verifier that checks answers, then policy-gradient updates against it. The session sets out the Markov decision process underneath, the Bellman equations and the policy gradient theorem, then what is known and unknown about these methods: reward hacking, overoptimisation, and why the trained model's goals need not be the reward's.
What is a world model, and what can an agent do with one that it could not do without? The session introduces the transducer, a stochastic automaton that generalises partially observed Markov decision processes and belief models, as a common architecture for world models in AI, control theory and neuroscience. It ends with abstractions as coarse-grainings of a world model, and the role of symmetries in choosing them.
Day 7 · Tue 17 Nov · Likely
Agent foundations, and mechanistic interpretability
How should a reasoner assign probabilities to mathematical statements it cannot yet prove? Logical induction gives a criterion: a market in sentences that no polynomial-time trader can exploit, which forces prices to converge to the truth and to obey probability theory in the limit. The session states the criterion, sketches why it can be satisfied, and asks what it says about agents reasoning about themselves.
How can an agent organise what it observes into reusable latent pieces? Condensation makes this precise: a family of latent variables satisfying a reconstruction condition and a Markov condition, with the theorem that two such families over the same observations determine one another, exactly and then with error bounds. The mathematics is conditional entropy and mutual information; the result connects to natural latents, sufficient statistics and computational mechanics.
What can we learn about how a model works from its internals? The session surveys the current approach: features in superposition, sparse dictionary learning to pull them apart, the geometry of activation space, and circuits that compose features into computations, with causal abstraction as the standard for when an explanation is right. Then an honest account of what the field can and cannot yet do.
Agent foundations has candidate formalisms but no agreed objects: what an agent is, what beliefs and goals are, and what a theory of agency would have to explain. A discussion of which of AIXI, logical induction, condensation and world models answers which question, and where the field is open to a mathematician arriving from outside.
Day 8 · Wed 18 Nov · Likely
Heuristic arguments, and open-source game theory
An introduction to heuristic arguments and mechanistic estimation: formal but defeasible arguments that estimate a quantity about a network, such as the probability of a rare behaviour, from its weights rather than by sampling. The session presents the agenda, how estimation would fit into a full solution to alignment, and the main unsolved technical and conceptual challenges. The mathematics is complexity theory and probability.
Either a second lecture that deepens the formalism, or a session on the open problems in heuristic arguments a mathematician could take up: what makes an estimate accurate, how an argument can compete with sampling, and what a presumption-of-independence argument can and cannot establish. Worked at the board, with researchers from the field there to answer questions.
When agents can read each other's source code, cooperation needs neither repetition nor trust. The session covers program equilibria: agents that cooperate conditional on a proof that the other will, grounded in Löb's theorem; commitment races and how to avoid them; and safe Pareto improvements. The mathematics is provability logic and game theory, and the questions bear directly on how AI systems will bargain.
Day 9 · Thu 19 Nov · Likely
Scalable oversight, debate, and data attribution
How can a human judge supervise a system that knows more than they do? Scalable oversight asks for protocols in which checking is easier than doing, so that weak supervision of a strong model stays reliable. The session frames the problem, surveys the main approaches, and sets out what a safety argument built on oversight has to show and where it remains open.
Debate aims to make honesty the winning strategy: two systems argue, and a judge who cannot verify the claim directly need only decide which argument is stronger. The session develops the complexity-theoretic picture, in which debate reaches PSPACE and cross-examination reaches NEXP, and the obstacles: obfuscated arguments, judge error, and whether prover-estimator debate can make the protocol work in practice.
Which training examples caused a model to behave as it does? Leave-one-out counterfactuals fail under overdetermination, which motivates Shapley values, classical and Bayesian influence functions, and unrolling the training dynamics; all are first-order perturbations of the map from data to parameters. The session develops these tools, how they scale to large models, and their connection to optimal transport, using the implicit function theorem and Hessians.
A conversation on moving from mathematical physics into AI safety: what carries over, what has to be relearned, how the research culture differs, and what an academic post allows that a lab position does not. Questions from the room are the point.
Day 10 · Fri 20 Nov · Likely
Verification, and the closing panel
Can a trainer plant hidden behaviour in a model that no one else can detect? Using cryptographic tools, the session shows how a backdoor keyed to a secret direction can be made statistically undetectable, with a random-ReLU classifier as the worked example, and what follows: worst-case interpretability and auditing face a cryptographic barrier. The mathematics is concentration bounds and computational hardness.
Hiding a message in plain sight: a sender, a receiver, a cover distribution, and an adversary who must not notice. The session shows how a language model's token sampling can carry a payload without changing its output distribution, why perfect security needs fresh randomness while computational security needs only a short key, and why this matters for monitoring chains of thought and detecting collusion. Mathematics: entropy and pseudorandom functions.
What the field needs from mathematicians, and what a mathematician needs from the field. A panel on where proofs would change what is possible, how to get involved from an academic position, and how to keep going after the fortnight.