Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks.
ProofPaper ↗
Key points
- Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining.
- Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms.
- These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training.
- Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.
Sources (1)
- [1]Evaluating Agentic Capabilities Before Post-Training: An End-to-End ApproacharXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 02:21 AM
We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks.
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
- Oct 14, 2026SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
- Oct 14, 2026Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
- Oct 13, 2026Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
- Oct 13, 2026Copy-Paste Language Models