ttok 1.0
I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
Proof1 independent outlet
Key points
- OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
- All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures.
- GPT-6 introduces no input-count change on this corpus.
- Tags: projects, ai, openai, generative-ai, llms, tokenization
Sources (1)
- [1]ttok 1.0Simon Willison's Weblog · Oct 9, 12:34 AM
I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026[AINews] not much happened today
- Oct 8, 2026Understanding Jev, the new model everyone is talking about
- Oct 8, 2026Pollo AI turns creative ideas into campaigns with OpenAI
- Oct 8, 2026Reasoning-Informed Visual Editing
- Oct 7, 2026[AINews] Claude Haiku 5.5 — better than GPT-6 Luna at the same pricing
- Oct 7, 2026Google rolls out improved SynthID AI content detector, now available globally
