Speedrunning Relational Foundation Model Pretraining
We introduce modded-nanoRT, the first such leaderboard for Relational Transformers.
ProofPaper ↗
Key points
- Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception.
- Speedrun leaderboards remove that barrier by decentralizing the search rather than the training run.
- We open the leaderboard with typed trilinear attention, a relational operator that conditions attention scores and values on the foreign key traversed, together with Muon and SIGReg.
- We will release the benchmark, its frozen artifacts, and the record runs.
Sources (1)
- [1]Speedrunning Relational Foundation Model PretrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 07:29 PM
We introduce modded-nanoRT, the first such leaderboard for Relational Transformers.
Progress on foundation model pretraining is gated by who can afford a trial, and relational pretraining is no exception.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Toward Agentic AI for the 6G Physical Layer: From Algorithm Discovery to Verified Baseband Silicon
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever
- Oct 18, 2026Google Open-Sources Harness2, a Recursive Agent Harnessing Framework
- Oct 17, 2026model : add Kolibri-1 (Aleph-Alpha) support by riteme · Pull Request #30673 · ggml-org/llama.cpp