Forty Questions: Identifying a Language Model from One-Word Answers
A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet".
ProofPaper ↗
Key points
- A model's profile records how often it gives each answer to each question.
- On a panel of 94 models, 40 answers identify the exact model 92.5% of the time, against 1.1% by chance.
- Profiles built through OpenRouter identified all ten models served by DeepInfra under blinded labels.
- Grok 4.7 is billed 1,247 input tokens for a one-line question that costs GPT-4o only 12 tokens, and five models state the current date without being billed for it.
Sources (1)
- [1]Forty Questions: Identifying a Language Model from One-Word AnswersarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:33 PM
A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet".
A model's profile records how often it gives each answer to each question.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 19, 2026You Are the Trigger: Speaker-Activated Backdoors
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills
- Oct 19, 2026Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode
- Oct 18, 2026TinyFish Makes Search Free: What It Really Costs an AI Agent to Read the Web