Policy-Mask: Budget-Aware Adaptation of Diffusion Features to Discriminative Tasks

Key points

• We introduce Policy-Mask, a budget-conditioned policy trained on the downstream objective that jointly selects all three under a prescribed channel budget.1
• At matched budgets, Policy-Mask improves over fixed extraction and feature-filter baselines across segmentation, classification, and semantic correspondence, reaching 38.9 mIoU on ADE20K, 75.4% top-1 on CIFAR-100 at 128 dimensions, and 76.3 PCK@0.1 on SPair-71k.1
• Component-wise ablations show that block selection materially affects downstream accuracy, and the highest recorded channel-budget score occurs at 256 retained channels, identifying a practical operating point for compact diffusion representations.1
• We further adapt Policy-Mask to diffusion transformers, including PixArt-$Σ$ and SD3.5 MMDiT, by replacing U-Net channels with image-token hidden dimensions.1

Sources (1)

• [1]Policy-Mask: Budget-Aware Adaptation of Diffusion Features to Discriminative Tasks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:13 PM

  Selecting intermediate features from a frozen diffusion model for discriminative tasks is non-trivial because the useful signal is distributed across the denoising trajectory, network depth, and channels.

  We introduce Policy-Mask, a budget-conditioned policy trained on the downstream objective that jointly selects all three under a prescribed channel budget.

Extractive summary: sentences quoted from the sources.