The Identifiability and Observability of Deep Normalized Attention

Key points

• For known positive nonconstant real-analytic normalizers, the function generically determines the effective scores and combined value map up to the signs induced by even normalizers.1
• We then classify exceptional fibers under explicit normalizer conditions, identifying when collapse makes later scores unobservable, and establish sharp Taylor orders for local identification.1
• Near simultaneous query/key collapse, we compute the complete native Jacobian decay spectrum on separating finite input banks.1
• For common first nonconstant normalizer degree $k$, layer $i$ has contact order $2k3^{i-1}-1$, with exact multiplicities and kernel dimension.1

Sources (1)

• [1]The Identifiability and Observability of Deep Normalized Attention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 07:59 AM

  We study which parameters of deep, unmasked, single-head attention are determined by its input--output function.

  For known positive nonconstant real-analytic normalizers, the function generically determines the effective scores and combined value map up to the signs induced by even normalizers.

Extractive summary: sentences quoted from the sources.