WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting

Key points

• Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors.1
• This enables scenario analysis: forecasting how a target responds to events that set covariates to chosen future values.1
• WhatIf-Bench is semi-synthetic: time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled covariate-target relationships grounded in each system.1
• Evaluating TSFMs, frontier LLMs, and simple regression baselines, we find the LLMs reach the highest skill (improvement over a naive baseline), ahead of the best TSFMs. Every model forecasts well on Replay futures (48 to 85% skill) but drops to -7 to 50% on Novel futures, with the regression baselines falling the most.1

Sources (1)

• [1]WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:32 PM

  We introduce WhatIf-Bench, a benchmark measuring how well forecasters use covariates to forecast a target's response to future events.

  Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors.

Extractive summary: sentences quoted from the sources.