A Symmetry-Matching Approach to Blind-Spot Reduction in Sparse Autoencoders

Key points

• Building on a symmetry-matching view from algebraic error-detection theory, we add a differentiable $V{Gini}$ regularisation term to SAE training that penalises uneven perturbation response within a specified family.1
• Across GPT-2, Gemma 2, and Qwen 2.5, a frozen multi-seed evaluation shows that $V$-regularisation raises both relative and absolute lower-tail responses while reducing response-profile imbalance, with no material general-text reconstruction degradation.1
• Family-geometry controls show that the effect is not explained by template duplication or a single dominant perturbation family, and held-out probes plus null-calibrated separation tests evaluate how far the representation-level change carries to downstream discriminability.1
• A zero-shot audit on out-of-distribution clinical radiology minimal pairs finds the relative weak-tail lift in all evaluated model-by-family cells, not perfectly rank-aligned with the trained statistic, and a held-out family audit finds it on family kinds never used in training.1

Sources (1)

• [1]A Symmetry-Matching Approach to Blind-Spot Reduction in Sparse Autoencoders
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 11:10 PM

  Language models can treat semantically distinct inputs as interchangeable at the representation level, creating blind spots that standard sparse autoencoder (SAE) objectives do not target.

  Building on a symmetry-matching view from algebraic error-detection theory, we add a differentiable $V_{Gini}$ regularisation term to SAE training that penalises uneven perturbation response within a specified family.

Extractive summary: sentences quoted from the sources.