Recurrent In-Context Learning for Tabular Foundation Models
Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the number of independently parameterized layers.
ProofPaper ↗
Key points
- We ask whether the ICL stage of such a model needs a distinct set of parameters for every layer, or only depth.
- We introduce Tab-Loop, which replaces the 12 untied ICL blocks of TabICLv2 with 4 tied blocks applied $k$ times and changes nothing else: the encoders, decoder, synthetic prior and training recipe are shared with an untied baseline trained for the same compute.
- At matched pretraining compute (18.2 EFLOP) and matched inference compute (12 block applications), Tab-Loop matches the baseline with 62% fewer parameters (10.4M vs.\ 27.6M): held-out log loss is 0.4583 vs.\ 0.4588, mean accuracy on 200 TALENT datasets is 84.10% vs.\ 83.92%, and Elo on 38 TabArena datasets is 1409 vs.\ 1396.
- At this scale, the ICL stage of a tabular foundation model needs depth more than it needs distinct parameters.
Sources (1)
- [1]Recurrent In-Context Learning for Tabular Foundation ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 03:26 PM
Tabular foundation models adapt to new datasets through in-context learning (ICL), and, like most transformers, they tie computational depth to the number of independently parameterized layers.
We ask whether the ICL stage of such a model needs a distinct set of parameters for every layer, or only depth.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Transformer-based In-Context Score Operator Learning for Diffusion Models
- Oct 9, 2026In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation
- Oct 9, 2026PDE-PFN: Prior-Data Fitted Neural PDE Solver
- Oct 9, 2026Tabular Foundation Models Are Effectively Shallow
- Oct 9, 2026Qwen/Qwen-Image-2.1-Turbo
- Oct 8, 2026An all-transistor physical-learning transformer can be trained to perform in-context learning