Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

Key points

• We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition.1
• We then generalize this insight into the $Test-Time Forecasting$ (TTF) framework in which a one-step forecaster leverages iterations of gradient descent as recurrent depth.1
• We demonstrate that lightweight TTF models trained on ODEs and PDEs generalize to out-of-distribution system dynamics during autoregressive rollout after being trained for one-step prediction.1
• Our results reinforce TTF as a theoretically motivated framework for designing recurrent depth models capable of forecasting unseen dynamical systems at test-time.1

Sources (1)

• [1]Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 04:08 AM

  Foundation models for dynamical systems can forecast unseen partial differential equations (PDEs), ordinary differential equations (ODEs), and time series during inference.

  We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition.

Extractive summary: sentences quoted from the sources.