Diffusion Models Explained: From Noise to Images

Diffusion Models Explained: From Noise to Images

Diffusion models have revolutionized image generation, powering systems like Stable Diffusion, DALL·E, and Midjourney. Despite producing stunning results, the core idea is surprisingly elegant: teach a neural network to gradually remove noise from an image, one small step at a time.

The Forward Process: Adding Noise

The forward (or diffusion) process takes a clean image and progressively adds Gaussian noise over $T$ timesteps until the image becomes pure random noise. At each step $t$, a small amount of noise is added according to a variance schedule $\beta_t$:

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}, x_{t-1}, \beta_t I)$$

This process is fixed and requires no learning. After enough steps (typically T=1000), the image is indistinguishable from pure Gaussian noise. A key mathematical property allows us to jump directly from the clean image $x_0$ to any noisy version $x_t$ without computing intermediate steps.

The Reverse Process: Removing Noise

The magic happens in the reverse process. A neural network (usually a U-Net architecture) learns to predict the noise that was added at each step, effectively learning to denoise:

$$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$$

Starting from pure noise $x_T$, the model iteratively removes noise to produce a clean image $x_0$. Each denoising step makes a small refinement, and the cumulative effect produces remarkably detailed and coherent images.

Training Objective

Training is straightforward: sample a random timestep $t$, noise the image to that level, and train the network to predict the added noise. The simplified loss function is:

$$L = \mathbb{E}{t, x_0, \epsilon}\left[|\epsilon - \epsilon\theta(x_t, t)|^2\right]$$

This mean-squared-error objective is stable and easy to optimize—a significant advantage over the adversarial training required by GANs.

Conditioning and Guidance

To generate images from text prompts, diffusion models incorporate cross-attention layers that condition the denoising process on text embeddings (typically from CLIP or T5). Classifier-free guidance amplifies the influence of the conditioning signal by interpolating between conditional and unconditional predictions, trading diversity for adherence to the prompt.

The Latent Diffusion Breakthrough

Running diffusion in pixel space is computationally expensive. Latent Diffusion Models (LDMs), the architecture behind Stable Diffusion, first encode images into a compact latent space using a pre-trained VAE encoder. Diffusion operates in this lower-dimensional space (e.g., 64×64 instead of 512×512), dramatically reducing compute requirements while maintaining quality.

Current Frontiers

Recent advances include flow matching (straighter sampling trajectories), consistency models (single-step generation), and DiT architectures (replacing U-Nets with Transformers). Video diffusion models extend these concepts to temporal sequences, opening new creative possibilities. The field continues to push the boundaries of what generative models can achieve.