Local Inference of LLMs on Long Contexts: A Data Systems Perspective
In this paper, we set out to study the limitations that prevent serving long-context models for single users, besides the capacity issue.
ProofPaper ↗
Key points
- Large language models can solve complex problems across a variety of subjects, such as mathematics and coding.
- The common understanding, however, is that local LLMs need compression (e.g., quantization) to run on a single GPU.
- Rather than characterizing a single model on one GPU, we analyze how different models behave across several GPU generations.
- Our analysis confirms that, as expected, self-attention dominates the runtime for long-context inference.
Sources (1)
- [1]Local Inference of LLMs on Long Contexts: A Data Systems PerspectivearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 01:56 PM
In this paper, we set out to study the limitations that prevent serving long-context models for single users, besides the capacity issue.
Large language models can solve complex problems across a variety of subjects, such as mathematics and coding.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Oct 19, 2026LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models
- Oct 18, 2026DiaLong: Benchmarking Reminiscence in Long-Context Dialogues