Native Geometry vs. Linear Recoverability: How Alignment Changes Platonic Scaling

Key points

• Do larger models learn increasingly similar representations of the same world?1
• The Platonic Representation Hypothesis (PRH) predicts that they should, but similar can mean native cosine geometry---nearest-neighbour structure induced by cosine similarity before any learned alignment---or linear recoverability: correspondence revealed by a supervised linear map.1
• The learned maps remain far from rotations or uniform scaling, and Sparse Autoencoder (SAE) and Block-Sparse Featuriser (BSF) features give no advantage over the same alignment applied to dense embeddings.1
• Model scale thus improves linear recoverability more clearly than native cosine-neighbourhood correspondence, without establishing convergence toward a shared native metric geometry.1

Sources (1)

• [1]Native Geometry vs. Linear Recoverability: How Alignment Changes Platonic Scaling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 11:38 PM

  Using matched Legacy Survey and Hyper Suprime-Cam (HSC) embeddings across five model families, we find that Ridge alignment strengthens the relationship between model size and cross-dataset neighbourhood correspondence relative to native cosine geometry.

  Do larger models learn increasingly similar representations of the same world?

Extractive summary: sentences quoted from the sources.