JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

Key points

• While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets.1
• With the key observation that noise corruption removes high-frequency artifacts and preserves low-frequency light transport, we introduce an asynchronous joint-modeling paradigm that facilitates fine-grained illumination control.1
• At inference time, we introduce an asynchronous hijacked generation pipeline that facilitates flexible illumination control.1
• For training data, we develop a multi-illumination high-quality synthetic video dataset with paired irradiance maps using a powerful game engine.1

Sources (1)

• [1]JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:08 PM

  Understanding and modeling light transport in human-centric scenes is a longstanding challenge.

  While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets.

Extractive summary: sentences quoted from the sources.