LightVAE: Towards Compact and Efficient Video Autoencoders

Key points

• Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive.1
• Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D.1
• LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction.1
• Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction.1

Sources (2)

• [1]LightVAE: Towards Compact and Efficient Video Autoencoders
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:46 AM

  To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth.

  Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive.

• [2]LightVAE: Towards Compact and Efficient Video Autoencoders
  Hugging Face Daily Papers · Oct 10, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.