I put a conformal risk control gate between an LLM agent and its write actions on tau-bench. The bound held. The gate didn't beat asking every time [R]
The thresholds come from conformal risk control: you pick an alpha, and the bound targets the share of conversations that execute a wrong write without asking.
Proof1 community thread
Key points
- My agent said it was 100% sure.
- So I built gate-calibrated: a barrier between a tool-using agent and the environment.
- Before any action that writes (modify an order, change a payment method), the gate computes two confidence signals, the model's own probability estimate and the agreement of k resampled next actions, maps them to a calibrated probability of success, and compares it to a threshold.
- tradeoff curve: wrong writes executed vs approvals requested
Sources (1)
- [1]I put a conformal risk control gate between an LLM agent and its write actions on tau-bench. The bound held. The gate didn't beat asking every time [R]r/MachineLearning (top, daily) · Oct 15, 09:16 AM
The thresholds come from conformal risk control: you pick an alpha, and the bound targets the share of conversations that execute a wrong write without asking.
My agent said it was 100% sure.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Video to surface normals-map with NormalCrafter
- Oct 14, 2026The AI race: China's open-weight models trail US closed models by just 7 months
- Oct 14, 2026Event2Vec: additive event embeddings where a linear score on a history decomposes exactly into per-event terms [P]
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B