I've been fascinated for a while by the parallels between how we engineer large datasets—LLMs included—and statistical thermodynamics, a bridge I've crossed before. So a video devoted to an empirical statistical regularity found in the study of scaling laws caught my attention: it plots model loss against compute, model size, and dataset size on a log-log scale, and each one collapses into a nearly straight line. It echoes the famous Chinchilla paper (2022), which set the ground rules for how model size and training data should scale together.
The video frames this as a hard floor that no amount of scale can cross. What actually caught my attention was a simpler question hiding underneath that framing: why does the curve show up at all?
Take a gas. A litre of air holds about 10²² molecules, each moving unpredictably—an unknowable state, impossible to track individually. And yet PV=nRT holds, with startling precision, for helium, nitrogen, and oxygen alike, even though those gases are chemically nothing like each other. At low density, none of that microscopic difference survives into the macroscopic law. Physicists call this universality: wildly different microscopic systems can share the exact same large-scale behavior, because most of what makes them different stops mattering once you zoom out far enough.
Neural network training looks like it's doing something similar. Two training runs never follow the same path—different architectures, different data orderings, a parameter space with almost nothing in common between one run and the next. And yet plot loss against compute across models that differ by orders of magnitude, and the same power law shows up, with an exponent that stays remarkably stable. The detail that should matter enormously—architecture, data mix, training recipe—barely moves the curve.
That's the real mystery, and it's a sharper one than "will scaling keep working." Physicists have run into this kind of question elsewhere too: why does something this microscopically complex—billions of parameters, trillions of token interactions, a training run that's different every single time—collapse onto one simple curve at all? There's no settled answer yet, just a handful of competing theories—some information-theoretic, some rooted in statistical mechanics—none of which has closed the case. One recent paper goes as far as mapping the training process directly onto ideal-gas thermodynamics, learning rate and weight decay standing in for temperature and pressure. Whether or not that particular mapping holds up, it's a sign the analogy is being taken seriously as physics, not just borrowed as a metaphor.
I've written before about treating LLMs as entropy-reducing machines, borrowing Boltzmann's bridge between statistical mechanics and information. This is the same bridge, crossed from the other direction: not what a trained model does to the entropy of what it reads, but what kind of statistical object a training run actually is. We perhaps found the equation before we found the theory behind it—which has happened before in physics, more than once. It might be exactly where we are with this one.
* * *
What strikes me most, watching all this unfold, is how the discipline keeps advancing on both fronts at once—in the very concrete applications that are now part of our Michelin teams' daily work, and in the conceptual, mathematical beauty running underneath it, as hard to fully grasp as it is fascinating.
