Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs

Key points

• An agent that drives an LLM sends the model the same conversation again and again, and each turn adds only a little text at the end.1
• Inference engines reuse the KV cache of the shared prefix, so only the new tokens pass through the model.1
• The incremental path shortens the tokenization of a session turn by roughly 40x relative to the Hugging Face reference tokenizer, and on 16 cores the chunk-parallel path encodes a four-million-character request 8.2x faster than a serial encode and 1.6-2.1x faster than LoPT.1
• Inside SGLang, Delta Tokenizer reduces the median TTFT through the chat interface by 59% at 100K tokens and brings the P99 TTFT of agent sessions down 9.5x under load and 23x when long requests tie up the front end.1

Sources (1)

• [1]Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:44 AM

  We present Delta Tokenizer, a CPU tokenization front end that tokenizes only the delta between a request and the previous request of the same session.

  An agent that drives an LLM sends the model the same conversation again and again, and each turn adds only a little text at the end.

Extractive summary: sentences quoted from the sources.