Iterate on an LLM, Serve on an SLM
At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into.
ProofPaper ↗
Key points
- As frontier LLMs gain more capabilities, the cost to run them also goes up.
- We propose doing both in sequence.
- Then, once the behavior converges, distill the recorded production behavior (not synthetic imitations of it) into an SLM through supervised fine-tuning (SFT).
- We show a proper use case of this method through a scheduling agent for home-care agencies that fans out a single yes/no safety decision per caregiver-shift pair on GPT-5.4-nano.
Sources (1)
- [1]Iterate on an LLM, Serve on an SLMarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:51 AM
At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into.
As frontier LLMs gain more capabilities, the cost to run them also goes up.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026PsyMemBehav: Psychology-Informed and Memory-Grounded Two-Level Identity-to-Action Reasoning for Social-Media Behavior Simulation
- Oct 10, 2026Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents
- Oct 9, 2026Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
- Oct 9, 2026Microsoft's Decision-1 model enters the fast-growing AI decision model race
- Oct 9, 2026Qwen/Qwen-Image-2.1-Turbo
- Oct 9, 2026The Thinking Leakage in Reinforcement Learning for LLMs