On the Convergence of Semi-Gradient TD: A Mean Field Perspective
Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability.
ProofPaper ↗
Key points
- We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target.
- Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution.
- We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.
Sources (1)
- [1]On the Convergence of Semi-Gradient TD: A Mean Field PerspectivearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 11:26 AM
Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability.
We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning
- Oct 14, 2026MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces
- Oct 14, 2026Beyond Corrected Actions: Training LLM Agents on Textual Critiques
- Oct 13, 2026PairIT: Autoregressive Transformers for Low-Data Molecule Optimization
- Oct 13, 2026Learning When to Transfer: Reinforcement Learning for Carbon-Footprint Modeling in New Manufacturing Domains
- Oct 13, 2026LensErase: A Foundation Model for Erase Anything