ResearchResearch paperEfficiency & Inference · Large Language Models1 source · Oct 11, 2026

Context-Aware Precision Scheduling for Small Language Models

Small language models (SLMs) used as on-device agents must balance accuracy and latency as their context grows.

We calibrate a timing model from measurements at different precisions and context lengths, and combine it with tensor sensitivities to choose configurations under a latency or accuracy budget.
We argue that fewer bits do not always mean faster execution, so our predictor estimates latency from calibrated execution costs rather than weight bytes alone.
We evaluate executed schedules on a laptop GPU, including a longer-context schedule with three weight formats and a tool-use benchmark measuring full response time and correctness.
Under an accuracy budget, measured error from a quantized KV cache determines the budget left for weights.

[1]Context-Aware Precision Scheduling for Small Language Models
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 02:16 PM
Small language models (SLMs) used as on-device agents must balance accuracy and latency as their context grows.
We calibrate a timing model from measurements at different precisions and context lengths, and combine it with tensor sensitivities to choose configurations under a latency or accuracy budget.

Extractive summary: sentences quoted from the sources.

Before this