Transfer Learning in Bandits from Unknown Adaptive Policies: Problem-Dependent Lower and Upper Bounds

Key points

• In practice, historical observations come from a previous adaptive decision process whose policy is unknown.1
• Before the online game starts, the learner receives, for each arm $k$, $N'k$ observations drawn i.i.d.\ from an unknown distribution $ν'k$, collected by an arbitrary and possibly data-dependent strategy, so the sample sizes $N'k$ may be random.1
• We derive an asymptotic problem-dependent lower bound and propose KL-UCB-Transfer, which combines online and historical data and achieves logarithmic regret with an improved leading constant whenever $μ'a<μ1-La$, and recovers the classical KL-UCB guarantee otherwise.1
• Since bounding the source-target discrepancy is often unrealistic, we relax the assumption to the one-sided condition $μk-μ'k<Lk$ and show that logarithmic regret can survive under misspecification, characterizing three regimes ranging from improvement to a degraded logarithmic bound to a loss of guarantee.1

Sources (1)

• [1]Transfer Learning in Bandits from Unknown Adaptive Policies: Problem-Dependent Lower and Upper Bounds
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 09:30 AM

  Transfer learning in stochastic multi-armed bandits is typically studied assuming that prior data forms a fixed offline dataset or is generated by a known source.

  In practice, historical observations come from a previous adaptive decision process whose policy is unknown.

Extractive summary: sentences quoted from the sources.