Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes

Key points

• We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria.1
• The framework distinguishes boundary representations, summaries of skipped computation, executed operations, and downstream loss.1
• Experiments in two frozen pretrained language models show that representation fidelity and task loss can favor different fitting objectives.1
• In a prespecified comparison on a 1.7-billion-parameter model, next-token-loss training lowers in-distribution negative log-likelihood by 0.1901 nats per scored token compared with reconstructing the skipped blocks' output, using the same optimizer and update count.1

Sources (1)

• [1]Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 12:21 AM

  Computational coarse-graining aims to replace a sequence of updates with a cheaper module while preserving task performance.

  We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria.

Extractive summary: sentences quoted from the sources.