The Price of Current-Effect Inference in Nonstationary Multi-Armed Bandits

Key points

• These objectives conflict: reward maximization favors better interventions, whereas current-effect inference requires continued exploration of potentially suboptimal interventions as outcomes change over time.1
• We characterize this tradeoff for $K$-arm bandits with Hölder-smooth mean rewards of exponent $β\in(0,1]$ and constant $L$.1
• Requiring high-probability simultaneous coverage of all current effects by confidence intervals of width at most $w$, we establish matching minimax bounds, up to logarithmic factors, on expected dynamic regret.1
• Finally, we show that regret constraints can prevent honest adaptation to unknown smoothness and support our theory with extensive numerical experiments.1

Sources (1)

• [1]The Price of Current-Effect Inference in Nonstationary Multi-Armed Bandits
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 04:26 PM

  Adaptive experiments use bandit algorithms to favor interventions with higher expected rewards.

  These objectives conflict: reward maximization favors better interventions, whereas current-effect inference requires continued exploration of potentially suboptimal interventions as outcomes change over time.

Extractive summary: sentences quoted from the sources.