Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models
Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1).
ProofPaper ↗
Key points
- Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack.
- Given that, we investigate whether LLM-based scorers form distinct internal representations for individual scoring rubrics and whether these representations can be independently controlled.
- Furthermore, scoring rubrics have partly separable directions in representation space for the longer tasks, though not for short responses (Part 1).
- First, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs, even among the seven models whose scores agree best with human raters ($r \geq 0.6$).
Sources (1)
- [1]Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:26 AM
Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1).
Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026PFedQFT: Personalized Federated Quantum-Inspired Fine-Tuning for Medical Large Language Models
- Oct 17, 2026A Cross-Domain Evaluation of Prompting Techniques for Large Language Models
- Oct 17, 2026Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework
- Oct 16, 2026"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws
- Oct 16, 2026Characterising and Analysing Retrieval-Augmented Generation Techniques
- Oct 16, 2026NL2SQL with Iterative Self-Refinement