Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization

Key points

• For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer.1
• Our theoretical analysis establishes that standard block-local first-order optimizers fail to scale updates with parameters under a shared learning rate.1
• To correct scale imbalance between paired Transformer matrices without changing predictions, we propose Symmetry-Aware Gradient Geometry Stabilization (SAGGS), which periodically equalizes the Frobenius norms of exact symmetry pairs, and its damped variant, Soft SAGGS.1
• In a 123.8M checkpoint-resume experiment, reciprocally rescaling matrix pairs barely changes validation loss at restart but increases final PPL by $3.78\pm0.42$; hard and soft rebalancing recover to within $0.02$ PPL of the control.1

Sources (1)

• [1]Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:08 AM

  We show that two Transformers representing the same function can follow different training trajectories.

  For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer.

Extractive summary: sentences quoted from the sources.