GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]
I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory.
Proof1 community thread
Key points
- Setup: pretrained GPT-2 small (124M), NVIDIA A40, fp32 with TF32 off, batch size 1, greedy decoding.
- Timings are bracketed by torch.cuda.synchronize(), after warmup, using the median of five runs per context length.
- Latency: at context length 1,000, the FLOP estimate is about 1,000x smaller with caching, but the per-token time goes from 29.69 ms to 4.95 ms (~6x).
- CPU/kernel-launch overhead is a hypothesis, not a conclusion supported by a profiler yet.
Sources (1)
- [1]GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]r/MachineLearning (top, daily) · Oct 18, 08:23 AM
I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory.
Setup: pretrained GPT-2 small (124M), NVIDIA A40, fp32 with TF32 off, batch size 1, greedy decoding.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 18, 2026vllm-project/vllm v0.32.0
- Oct 18, 2026How to Make Zero-Training Tabular Predictions with NVIDIA Structured Data Models
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)
- Oct 16, 2026nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-MXFP8
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning