RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer.
ProofPaper ↗
Key points
- Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains.
- We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever.
- On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors.
- Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.
Sources (1)
- [1]RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient PredictionsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:27 AM
Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer.
Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
- Oct 13, 2026Copy-Paste Language Models
- Oct 12, 2026AReA: Adaptive Receptive-Field Attention for Efficient High-Resolution Diffusion Transformers
- Oct 11, 2026autotrust/JEV-35B-NVFP4
- Oct 11, 2026History Replays Itself: Unleashing Long-Context Capability in Linear Attention
- Oct 11, 2026ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs