DiffusionOPSD
Less compute. Broader exploration.
The same self-distillation principle.
DiffusionOPSD turns image-level reward guidance into explicit targets for clean-output predictions at sampled queries. In v2, we keep that principle and make the training recipe more efficient: one reward-ascent step for target construction, broader prompt coverage, and rollouts that stop at the query. We also correct the HPSv3 reward pipeline.
The controlled experiment preserves held-out performance while reducing training GPU time by approximately 30.5%. A separate measurement of one complete update with the old and new recipes reduces update time by 39.7%.
Spend less on each target. Explore more prompts. Stop the rollout where training needs it.
v1 takes two normalized reward-gradient ascent steps to construct its positive target. v2 takes one, reducing reward-gradient computation while retaining bounded positive and negative targets and finite fitting. Both the behavior-policy EMA and the checkpoint EMA are retained.
We move from 48 × 24 to 1,152 × 1 × 1: 1,152 prompt groups, one trajectory per group (K = 1), and one optimizer update per outer iteration. Both recipes collect 1,152 samples and use a finite fitting budget of one optimizer update per outer iteration. The new allocation trades repeated samples of the same prompt for broader prompt coverage.
DiffusionOPSD does not depend on image diversity within each prompt in the same way as FlowGRPO or DiffusionNFT. We can therefore shift the sampling budget to diversity across prompts, with one trajectory per prompt. In our experiments, this allocation also produces more stable training reward curves.
DiffusionOPSD fits detached targets for clean-output predictions at sampled low-noise queries. With K = 1, the group-normalized fitting weight is fixed at 0.5, so finite fitting no longer requires the rollout endpoint. The v2 query-prefix rollout collects the required query state and clean-output anchor, then skips the remaining denoising, endpoint decoding, and endpoint reward evaluation.
The previous differentiable preprocessing path omitted CLIP mean/std normalization. This affects reward values, image gradients, and the resulting reward-gradient targets. v2 uses the official differentiable preprocessing with normalization; evaluation uses the official reward API and its unscaled logits[:, 0].
A controlled 50-update experiment.
SD3.5-M · CLIPScore
One complete update, measured on 8 × A100.
Explore every reward-specific run on SD3.5-M and Z-Image-Turbo, plus the SD3.5-M joint run. v2 is designed for more stable training; the raw traces below let you inspect the behavior reward by reward.
Loading recorded training traces…
The held-out results match Table 1 of the paper. Each reward-specific cell evaluates its own trained checkpoint; joint rows evaluate one checkpoint across all ten metrics.
The joint-training overview uses elapsed W&B run time, including initialization, evaluation and checkpoint overhead. These historical runs are not a controlled speed benchmark. The 30.5% reduction measures training GPU time in the controlled experiment, excluding initialization, checkpoint writing and evaluation. The separate 39.7% measurement compares one complete update on the same host with 8 × A100, SD3.5-M and CLIPScore; it is not a full-run speed estimate.
v1 records rollout endpoint reward; v2 records local reward at the clean-output anchor, so their levels are not directly comparable quality scores. Epochs are zero-based. Raw traces are shown by default; optional smoothing uses EMA 0.6. Hover, tap, or use ← / → to inspect.
All v2 evaluations use 1,000 images from 200 DrawBench prompts. Historical training seeds are not uniformly matched to v2 seed 42. Hover a table cell for its source or the difference from v1 at displayed precision. Higher scores are better; ‡ marks published reference results and † two-stage OPD.
The HPSv3 v1 training curve uses the corrected old-recipe rerun. Official HPSv3 implementation ↗
The update keeps the central idea intact: construct bounded positive and negative targets from reward gradients, fit them as detached supervision through finite fitting, and refresh the behavior policy. v2 makes that loop leaner.
Explore DiffusionOPSD ↗