LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization

Key points

• We introduce LinMorph, which recasts Video DiT linearization as a learnable softmax-to-linear weight transformation problem.1
• Instead of copying predefined counterparts, LinMorph learns a cross-parameter morphism that lets each target Gated DeltaNet (GDN) parameter draw jointly from pretrained query, key, value, and output projections, including for parameters with no direct softmax counterpart.1
• The resulting weights are then materialized as independent GDN modules for progressive distillation and joint recovery.Since the morphism operates in weight space rather than token space, it can be learned on short, low-resolution clips and directly reused at larger video scales.1
• Across pretrained Video DiTs, LinMorph enables markedly faster recovery and better final generation quality than random initialization and heuristic weight reuse.1

Sources (1)

• [1]LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:06 AM

  Post-hoc linearization is an attractive way to accelerate pretrained Video Diffusion Transformers (Video DiTs), yet existing methods largely treat the quality loss induced by attention replacement as a recovery problem.

  We introduce LinMorph, which recasts Video DiT linearization as a learnable softmax-to-linear weight transformation problem.

Extractive summary: sentences quoted from the sources.