[GS_C_MS] Thermodynamics of Transformers: A Lagrangian Field Theory of Attention Dynamics
ABSTRACT
We develop a physical framework for Transformer attention by reinterpreting it through the lens of thermodynamics and Lagrangian mechanics. Treating network depth as time and token representations as field variables allows us to derive a Lagrangian from Shannon entropy and the Fisher–Rao metric. The resulting Euler–Lagrange equations demonstrate that softmax attention aligns with the state that extremizes the Helmholtz free energy. This establishes a formal correspondence with the canonical ensemble. In this context, attention weights behave like dipole-type interaction energies between query and key vectors. Empirically, this framework identifies effective specific heat as a diagnostic for generalization. Controlled experiments on modular addition (moduli 19–113) reveal that sharp peaks in specific heat consistently precede the onset of grokking, signaling a fluctuation-driven representation crossover. Furthermore, rotary positional embeddings are derived as Goldstone modes from spontaneous symmetry breaking under the entropic potential. The results suggest that thermodynamic observables can provide quantitative metrics for generalization dynamics and indicate a symmetry-based foundation for neural architectures.
