The generic framework for diffusion models admits sampling from an estimate of the data distribution after training on samples from this distribution. In most practical scenarios though, we want to condition this sampling on some kind of class variable or text prompt (e.g. generate a random image vs. generate an image of a dog). This requires adapting the diffusion model to admit conditional sampling. In practice, guidance heuristics are often added to improve fidelity and condition adherence.
Direct Conditioning
The simplest way to learn conditional scores is to make the learned score function conditional on . Define the conditional score as
The conditional score identity states that
Assume that the forward noising process is independent of once is known, so that . Starting from the integral on the right and applying Bayes’ rule gives
The ideal conditional score-matching objective is
Since the second term does not depend on , its gradient is zero and we can drop it for gradient-based optimization. We use below to denote equality of gradients with respect to .
The first annotated step uses the conditional score identity proved above to replace with . The second uses measurability: since is determined by , it can be moved inside the conditional expectation. The third uses the law of iterated expectations to remove the nested conditional expectation. In the last line we dropped the second term since it is also independent of .
Guidance
Direct conditioning is sufficient to sample from a learned conditional distribution . In practice, guidance is often used during inference to trade sample diversity for greater fidelity and stronger adherence to the condition.
Classifier Guidance
Dhariwal and Nichol introduced classifier guidance, which combines a diffusion model with a separately trained, noise-aware classifier. The classifier learns from noisy samples at different times. It is not used to train the diffusion model; the two models can be trained independently and are combined only during sampling.
Using Bayes’ rule,
The condition prior does not depend on , so . Therefore, the conditional score satisfies
The classifier supplies an estimate of the second term, giving the guided score
where controls the guidance strength. Under exact models, this is the score of the tilted distribution
Thus, recovers the conditional score. Increasing places more emphasis on the condition, usually increasing fidelity while reducing diversity.
Classifier-Free Guidance
Ho and Salimans introduced classifier-free guidance, which avoids training a separate classifier. During training, the conditioning information is randomly dropped so that one network learns both the conditional and unconditional scores. During sampling, the two predictions are combined as
With this convention, gives ordinary conditional sampling, while strengthens the conditioning signal. Classifier-free guidance is the dominant method used to enforce conditioning in state-of-the-art diffusion models for image generation, video generation etc.