But how do AI images and videos actually work? | Guest video by Welch Labs

Original Media · 37m · English · YouTube · 3Blue1Brown

TLDR

This guest video by Stephen Welsh (WelshLabs) explains how diffusion models generate images and videos by reversing a noise process connected to Brownian motion. It covers CLIP's shared text-image embedding space, the DDPM training objective of predicting total added noise, why random noise must be added during generation to avoid blurry averages, DDIM's deterministic sampling for faster generation, and classifier-free guidance which amplifies the difference between conditioned and unconditioned model outputs to achieve strong prompt adherence — as used in models like DALL-E 2, Stable Diffusion, and WAN 2.1. The video builds intuition through 2D visualizations of vector fields and spiral datasets. The host (3Blue1Brown) introduces this as a guest video during paternity leave. The pieces fit together remarkably: text encoders steer complex diffusion processes in high-dimensional spaces using relatively simple geometric principles. All you need to create lifelike video is language.

For you

PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.

Sign in to unlock

Surprising Insights

•

The model learns to predict the *total* noise added across all diffusion steps, not just the next step — a seemingly harder task that actually works better because it reduces training variance.

•

Adding random noise *during generation* improves sharpness and diversity; removing it causes all outputs to collapse to the dataset mean, producing blurry averages.

•

The same trained diffusion model can be used with both stochastic (DDPM) and deterministic (DDIM) samplers, producing the same final distribution — the theory guarantees distributional equivalence, not sample equivalence.

•

Classifier-free guidance requires no separate classifier; it just drops conditioning on a fraction of training examples, letting the same model produce both conditioned and unconditioned predictions at inference time.

•

WAN's standard negative prompt is passed to the text encoder in Chinese, not English, yet effectively steers the English-prompted generation away from artifacts like extra fingers.

Key takeaways

•

Diffusion models learn to reverse a forward process where Gaussian noise is progressively added to data until it becomes pure noise; this forward process is mathematically equivalent to Brownian motion in high-dimensional space.

•

The DDPM breakthrough was training the model to predict the total noise added to the original image (the vector pointing from noisy sample back to clean data) rather than predicting a single denoising step — this reduces variance and lets the model learn the score function (direction toward higher data density) more efficiently.

•

During generation, adding random noise at each step is not a bug but a theoretical necessity: the model learns the mean of a Gaussian distribution, so sampling requires adding zero-mean noise to actually draw from that distribution; without it, all trajectories collapse to the dataset average, producing blurry images.

•

DDIM showed that the same trained model can be used with an ordinary differential equation (no random steps) by rescaling step sizes, enabling deterministic, high-quality generation in far fewer steps — this principle underlies modern fast samplers and flow matching used by WAN.

•

CLIP creates a shared embedding space where text and image vectors align via contrastive learning; the geometry of this space supports vector arithmetic (e.g., 'man with hat' minus 'man' ≈ 'hat'), providing the semantic steering signal for diffusion models.

•

Conditioning alone (passing text embeddings to the diffusion model) is insufficient for strong prompt adherence; classifier-free guidance computes the difference between conditioned and unconditioned predictions, scales it by a guidance factor, and uses that amplified direction — this literally grows requested objects (like a tree) in size and detail as guidance increases.

•

WAN 2.1 extends guidance by using a negative prompt (in Chinese) listing unwanted features (extra fingers, walking backwards) and subtracting its embedding from the positive prompt's embedding, steering the diffusion away from specific failure modes.

•

The core components — CLIP's semantic space, diffusion's score-based generative process, DDIM/flow matching's efficient sampling, and classifier-free guidance's steering — were developed independently but compose cleanly because they operate on compatible geometric principles in high-dimensional vector spaces.

Notable quotes

“At the core of how these models operate is a deep connection to physics.”

▸ Notable quotes · 0:10

“This generation of image and video models works using a process known as diffusion, which is remarkably equivalent to the Brownian motion we see as particles diffuse, but with time run backwards, and in high-dimensional space.”

▸ Notable quotes · 0:14

“The clever idea from here is to make use of the similarity not just between the corresponding images and captions, but between all image-caption pairs in the batch when training our models.”

▸ Notable quotes · 4:52

“The learned geometry of CLIP's embedding space allows us to operate mathematically on the pure ideas or concepts in our images and text, translating the differences in the content of our images, like if there's a hat or not, into a literal distance between vectors in our embedding space.”

▸ Notable quotes · 6:16

“How is it that adding random noise while generating images leads to better quality, sharper images?”

▸ Notable quotes · 10:37

“The fact that we can take a trained text encoder from CLIP or elsewhere and use its output to actually steer the diffusion process, which itself is highly complex, seems almost too good to be true.”

▸ Notable quotes · 34:46

Chapters

•

0:00 — Introduction: Diffusion and Physics: Introduces diffusion models as reversing Brownian motion in high-dimensional space; demonstrates WAN 2.1 generating video from pure noise via iterative transformer steps.

•

2:26 — CLIP: Shared Text-Image Embedding Space: Explains CLIP's contrastive training on 400M image-caption pairs, cosine similarity in 512D space, and how vector arithmetic in the embedding space captures semantic concepts like 'hat'.

•

8:13 — DDPM: Training to Predict Total Noise: Describes the 2020 DDPM paper: forward diffusion as random walks, training objective of predicting total added noise (not single-step), and why this reduces variance and learns the score function.

•

10:37 — Why Generation Requires Adding Noise: Uses 2D spiral visualization to show that without random noise steps, all trajectories collapse to the data mean (blurry images); adding noise samples from the learned Gaussian distribution.

•

22:02 — DDIM: Deterministic Sampling with Fewer Steps: Introduces the Fokker-Planck connection: an ODE exists with the same final distribution as the stochastic process, enabling deterministic DDIM sampling with rescaled steps and far fewer iterations.

•

25:07 — Conditioning and Classifier-Free Guidance: Shows how CLIP text embeddings condition diffusion; demonstrates that conditioning alone fails, but classifier-free guidance (amplifying conditioned minus unconditioned vectors) achieves strong prompt adherence.

•

33:42 — WAN 2.1 and Negative Prompts: Describes WAN's flow matching (DDIM generalization) and negative prompt guidance — subtracting a Chinese-encoded negative prompt embedding to steer away from artifacts like extra fingers.

•

35:18 — Conclusion and Host Outro: Reflects on how independently developed pieces compose cleanly; host (3Blue1Brown) introduces guest creator Stephen Welsh and explains paternity leave guest video series.

Speakers

•

Stephen Welsh — Guest presenter

•

Grant Sanderson — Host (3Blue1Brown)

⋮