Characterizing the Separability Gap in Prior-Data Fitted Networks

Key points

• Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent.1
• We show that common PFN priors are non-separable, even in simple settings.1
• We characterize its difference from the predictive based on the context posterior alone and show that this difference vanishes under posterior concentration.1
• Our findings highlight the need for explicit rules on unlabeled data use in PFN evaluation and establish the dependence between feature and label distributions as a consequential prior design choice.1

Sources (1)

• [1]Characterizing the Separability Gap in Prior-Data Fitted Networks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 03:07 PM

  Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions.

  Standard PFN theory assumes a separable task prior, under which the feature distribution and the mechanism generating labels from features are independent.

Extractive summary: sentences quoted from the sources.