When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Key points

• Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers.1
• We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation.1
• A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries.1
• Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model.1

Sources (1)

• [1]When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:57 AM

  Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers.

  Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers.

Extractive summary: sentences quoted from the sources.