RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

Key points

• Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds.1
• In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place.1
• We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions.1
• Building on this observation, we develop RINSE, a practical data filtering method for online RL.1

Sources (1)

• [1]RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:01 PM

  Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience.

  Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds.

Extractive summary: sentences quoted from the sources.