AI labs report benchmark scores 6.4% higher than independent testers, on average
I've been building whatmodel.sh, an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.
Proof1 community thread
Key points
- Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks.
- AI labs reported scores 6.4% higher on average for the same models on the same tests.
- Everything is open: the dataset is CC BY-SA 4.0 and the code is MIT.
- Sources and their licenses are recorded, with links on each model's page.
Sources (1)
- [1]AI labs report benchmark scores 6.4% higher than independent testers, on averager/LocalLLaMA (top, daily) · Oct 19, 09:47 AM
I've been building whatmodel.sh, an open source AI leaderboard, and wanted to see how a labs' own benchmark scores compare with independent testers.
Using 1,629 published benchmark results, I matched 611 comparisons of 180 models on 103 benchmarks.
Extractive summary: sentences quoted from the sources.
Hype check
Questioned1 reported issue since
- Issue · Oct 19, 2026Fem Malfoy's Remembrall Incident
- Evaluation · Oct 19, 2026Krea 2 Turbo @ 4 steps (w/ my 4-step LoRA) vs MOSS SFT at 4 steps (ww AK's 4-step LoRA) - same prompt set you've seen me publish in the past (headline: MOSS is really good! and great job by AK on his LoRA)
- Evaluation · Oct 19, 2026The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral
- Evaluation · Oct 19, 20266M param character renderer: one reference image in, a minute of video out, tested against Blender truth
- Evaluation · Oct 19, 2026AI generates research ideas faster than I can test them. Anyone else? [R]
- Evaluation · Oct 19, 2026How to continue a GitHub Copilot app session in VS Code
Before this
- Oct 18, 2026Google Open-Sources Harness2, a Recursive Agent Harnessing Framework
- Oct 17, 2026I made a custom node that turns illustrations into screentone manga panels
- Oct 17, 2026deepseek-ai/DeepSeek-V4.1-Flash-1017
- Oct 16, 2026Comfy-Org/MOSS-Image
- Oct 15, 2026open source framework that makes your AI agents generate UI elements
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
