Online Tokenization with Optimal Delay in Linear Time
Tokenization is an essential preprocessing step in almost all large language models (LLMs).
ProofPaper ↗
Key points
- One of the most popular frameworks is BPE tokenization, which consists of special token segmentation, pretokenization, and byte-pair encoding (BPE).
- We introduce the notion of delay for online tokenization which characterizes the extent to which the algorithm output lags behind the (possibly computationally intractable) optimum.
- We implement a practical variant for tokenizers from OpenAI's tiktoken and HuggingFace's tokenizers.
- We obtain a $7.9\times$-$34.8\times$ latency advantage (measured as time to first token) over tiktoken (as well as the recent gigatoken tokenizer) and our implementation maintains approximately constant memory usage independent of input size.
Sources (1)
- [1]Online Tokenization with Optimal Delay in Linear TimearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:08 PM
Tokenization is an essential preprocessing step in almost all large language models (LLMs).
One of the most popular frameworks is BPE tokenization, which consists of special token segmentation, pretokenization, and byte-pair encoding (BPE).
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026Agentic-MatriBO: Capability-Compiled Agentic Bayesian Optimization for Materials and Chemical Design
- Oct 12, 2026Density-Matrix Reinforcement Learning
- Oct 12, 2026Learning to Design SR Solvers: Harness Optimization for Symbolic Regression
- Oct 12, 2026From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction