Controllable Diverse Sampling through Pullback Directions

Key points

• Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment.1
• We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-embedding space to the image space to identify directions of maximal output sensitivity.1
• By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal.1
• Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on prompt-conditioned image generation and inpainting.1

Sources (1)

• [1]Controllable Diverse Sampling through Pullback Directions
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 08:17 PM

  Conditional diffusion models can generate high-quality images that align closely with a given prompt.

  Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment.

Extractive summary: sentences quoted from the sources.