Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning

Key points

• A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment.1
• The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates.1
• Across classical control and deep reinforcement-learning benchmarks, Q($γH$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines.1
• By decoupling horizon adaptation from the underlying learning updates, Q($γH$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.1

Sources (1)

• [1]Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 12:20 PM

  We propose Q($γ_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal.

  A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment.

Extractive summary: sentences quoted from the sources.