Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity
Consistency training (CT) trains a model to answer the same with and without the cue.
ProofPaper ↗
Key points
- Language models often change answers because of a cue, a hint such as "I believe the answer is B" that gives no valid evidence.
- Four CT methods apply this idea to output tokens (BCT), residual activations (ACT), attention patterns (AttCT) and sampled-answer rates (RMCT).
- CT thus weakens the trained cue without teaching models when a cue is right.
- Evaluations with only wrong cues at the trained position miss the first two properties.
Sources (1)
- [1]Consistency Training Suppresses Cue Following Without Teaching Cue SelectivityarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:51 AM
Consistency training (CT) trains a model to answer the same with and without the cue.
Language models often change answers because of a cue, a hint such as "I believe the answer is B" that gives no valid evidence.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026A Symmetry-Matching Approach to Blind-Spot Reduction in Sparse Autoencoders
- Oct 16, 2026axolotl-ai-cloud/axolotl v0.21.0
- Oct 16, 2026HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models
- Oct 16, 2026Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
- Oct 16, 2026Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
- Oct 16, 2026Compression Precedes, Content Decides: Emergent Misalignment in Reward Models