Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study
In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates.
ProofPaper ↗
Key points
- Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling.
- Therefore, we ask: is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?
- To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: sufficient-statistic invariance and the martingale property.
- Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice.
Sources (1)
- [1]Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical StudyarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 10:53 PM
In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates.
Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch
- Oct 20, 2026Gradium Open-Sources Phonon, a 31M-Parameter On-Device TTS Model Under Apache 2.0
- Oct 20, 2026RANGER: Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards
- Oct 20, 2026MPS in PyTorch 2.15
- Oct 19, 2026Sparse RL Relaxation for LLM Reasoning
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models