How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination Detection
Under matched conditions, classifier-head fine-tuning (cls) paradigm consistently outperforms the LLM-as-Judge sequence-to-sequence (s2s) on binary hallucination detection.
ProofPaper ↗
Key points
- We trace this gap to the representations induced by the two objectives.
- Static analysis shows that cls learns a task-specific discriminative direction, pushing hidden states into a clear two-cluster geometry with stronger class separation and linearly decodable label information.
- Our analysis also reveals that the cls signal saturates before the final layer, after which upper layers form a plateau of statistically interchangeable and largely redundant representations.
- Further training-dynamics analysis reveals 1) a lower pretrained substrate, 2) a narrow upper-middle InfoWindow where task information emerges, and 3) a later-synchronizing upper plateau that mirrors this signal.
Sources (1)
- [1]How Fine-Tuning Objectives Shape Layer-Wise Information in LLM Hallucination DetectionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:30 AM
Under matched conditions, classifier-head fine-tuning (cls) paradigm consistently outperforms the LLM-as-Judge sequence-to-sequence (s2s) on binary hallucination detection.
We trace this gap to the representations induced by the two objectives.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Target-Aware Curation of Hallucination Stress Tests for Large Language Models
- Oct 19, 2026You Are the Trigger: Speaker-Activated Backdoors
- Oct 19, 2026From Black Box to Controller: Inference-Time Steering of LLM via Feature Amplification
- Oct 18, 2026MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers
- Oct 18, 2026[P] asmsense: 228k stripped x86-64 functions with source-aware labels, and a map of the data we still lack. [P]
- Oct 17, 2026The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check