Imagine trying to understand a machine by watching it learn. Its parts are visible: every weight, every adjustment, every training example. Yet when billions of parameters change together, the whole process can look less like a machine than a storm. The surprising possibility is that the storm may have rules. A group of researchers argues that mathematics borrowed from physics could help reveal them—not by explaining every neuron, but by finding the patterns that govern learning at a larger scale.
Their paper, “There Will Be a Scientific Theory of Deep Learning,” proposes a name for this emerging effort: learning mechanics. The phrase is a research program, not a claim that neural networks are little engines obeying Newton’s laws. The connection is more useful than literal. In mechanics, scientists ask how a system changes over time as its parts interact. During training, a neural network’s parameters also change over time, nudged by gradients that depend on the model, its data, and its learning rule.
Physics rarely begins by solving the most complicated system imaginable. It finds simpler cases that preserve something essential. The orbit of a planet, the swing of a pendulum, or the behavior of a simple gas can each make a hard world easier to reason about. Learning theory has its own stripped-down settings: networks without nonlinear activations, or models whose behavior can be approximated by a linear one. These are not full portraits of modern AI. They are test chambers where researchers can follow the math closely enough to see what might matter in messier systems.
One clue from these simplified networks is that learning can happen in an order. When a network is trained on a task with several underlying patterns, it may pick up the strongest, most prominent patterns first, leaving weaker ones for later. That gives researchers a possible mechanism behind a familiar observation: models often learn easier or more obvious structure before harder details. A toy model cannot prove that every large language model learns this way. It can, however, turn a vague impression into a question that can be measured and tested.
Another physics-style move is to ask what happens as a system grows without bound. For neural networks, researchers study limits in which the number of units becomes extremely large. One such limit produces a striking contrast. In a “lazy” regime, a network’s internal features barely change as it trains, even while its output improves. In a “rich” regime, those features shift to fit the task. In one demonstration discussed in the paper, a network with 200 units showed these contrasting behaviors when researchers changed its output scaling: a setting of 0.1 produced feature learning, while a setting of 30 left the network’s internal features moving very little.
That contrast matters because it warns against treating “a neural network” as one uniform kind of learner. A seemingly small change in setup can steer training toward a different style of learning. The infinite-size analysis is not itself a realistic description of every practical model; its value is that it exposes a choice hidden inside the complexity. It also raises the next question: which of these clean mathematical regimes helps explain the systems people actually train?
Some answers come not from a perfect microscopic account, but from regularities in the aggregate. As models, datasets, and computing budgets grow, performance often follows predictable scaling patterns. During training, another recurring pattern appears in the shape of the loss landscape: its sharpness can rise and then hover near a threshold related to the learning rate. Such observations resemble the empirical laws that helped physics develop, where a dependable relationship can guide prediction before scientists fully understand why it holds.
The practical stakes become clear in the problem of tuning. Training a large model involves many settings—learning rate, batch size, initialization, width, and depth among them. Choosing them by trial and error can consume costly runs. The researchers point to mathematical parameterizations that let some settings be tuned on smaller models and transferred to larger ones. This is not a universal shortcut, and the underlying arguments are often asymptotic: they describe idealized limits. But the idea is powerful: understand how a system changes as its scale changes, then use that knowledge to make the next experiment less of a guess.
The broadest clue may be that different systems sometimes behave alike. Models with different architectures can reach comparable performance; representations learned across tasks can show similarities. In physics, researchers use the idea of universality to explain how systems with different microscopic details can share large-scale behavior. For learning mechanics, the hope is similar: perhaps many particulars of a network will matter less than a few common features. The analogy is suggestive, not settled. Even how to measure whether two models have learned “similar” representations remains an open problem.
That caution is central to the paper. It is a synthesis of evidence and open questions, not a report that a finished theory has arrived. The authors highlight the gaps: current solvable models often simplify away important nonlinear behavior; natural data is difficult to summarize; and researchers still lack a general account of why scaling laws take the values they do. They also ask whether finite networks can truly be understood as approximations to idealized infinite systems, or whether the finite details sometimes change the story.
If learning mechanics succeeds, its first payoff may be modest but meaningful: better predictions about which training choices are likely to work, which patterns should transfer between models, and where an explanation stops applying. It could also complement mechanistic interpretability, which searches inside models for specific features and circuits. The paper’s image is of two sciences with different scales: one tracing the components and mechanisms, the other describing the broad motion and shared rules.
For now, the adventure is less “AI decoded” than “a map is taking shape.” Physics offers mathematics for simplifying complexity without pretending it has vanished. Applied to neural networks, that strategy turns training from a sequence of engineering hunches into something that can be measured, modeled, and challenged. The next breakthrough may come not from staring harder at the whole storm, but from finding the few forces that organize it.
