FreeCHROMA: Generative Chroma Keying from Unaligned Background References

Key points

• In film production, chroma keying composites an approved foreground performance into a new environment, but conventional workflows require coordinated background plates and dedicated harmonization.1
• Recent generative methods often require pre-aligned backgrounds, focus on background-conditioned foreground relighting, or represent the environment only through an edited first frame.1
• To train the model, we construct pseudo-paired data from readily available real monocular videos through foreground-background separation and augmentation with existing foundation models.1
• FreeCHROMA relaxes the constraints of existing tools to enable end-to-end generative chroma keying from unaligned inputs, with exploratory experiments demonstrating its potential in robotics.1

Sources (1)

• [1]FreeCHROMA: Generative Chroma Keying from Unaligned Background References
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 01:21 AM

  We present FreeCHROMA, which adapts a pretrained video diffusion model in a parameter-efficient manner for generative video compositing from one to three unaligned background references.

  In film production, chroma keying composites an approved foreground performance into a new environment, but conventional workflows require coordinated background plates and dedicated harmonization.

Extractive summary: sentences quoted from the sources.