FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention

Key points

• Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position signals bound to the training length.1
• Existing attention-free alternatives all optimize the single function of generation.1
• At 30M parameters, iso-parameter and iso-token budget, FON reaches perplexity 131.03 on WikiText-103, beating Transformer (197.73) and Mamba2 (152.53); the advantage persists under all depth-aligned baselines and at the 70M tier (-20.8%).1
• We report all costs and boundaries: impaired long-range retrieval, synthetic-task disadvantages, weaker out-of-domain code infilling than a same-recipe Transformer, and a 6.5-12.3x wall-clock gap.1

Sources (1)

• [1]FON: A Draft-Revise Dual-Organ Language Model without Sequence-Level Quadratic Attention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 07:28 AM

  We propose FON (Fusion-Organ Network), which differentiates one weight stack into a drafting organ (multi-tap learnable delays + content-adaptive recursion; causal, O(S) training, O(1) decode state, no sequence-level quadratic attention) and a revision organ that re-processes the written prefix through multi-scale ripple kernels injected via weak gates, never generating.

  Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position signals bound to the training length.

Extractive summary: sentences quoted from the sources.