Transformers, the tech behind LLMs | Deep Learning Chapter 5

Original Media · 27m · English · YouTube · 3Blue1Brown

TLDR

This video provides a visually-driven, high-level explanation of how transformer models like GPT work, covering tokenization, word embeddings, the transformer architecture (attention and feed-forward blocks), the unembedding process, softmax, and temperature sampling, using GPT-3's 175 billion parameters as a running example.

For you

PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.

Sign in to unlock

Surprising Insights

•

The same basic transformer architecture handles text, images, and audio by simply changing how inputs are tokenized (words, image patches, sound chunks).

•

Word embeddings capture analogies via vector arithmetic (e.g., Italy - Germany + Hitler ≈ Mussolini; Germany - Japan + sushi ≈ bratwurst) without explicit supervision.

•

The model only uses the last token's representation to predict the next token during inference, despite all tokens having rich contextual embeddings — this is a training efficiency choice.

•

Temperature is implemented by dividing logits by T before softmax; T=0 makes the model purely greedy, while the API caps T at 2 to avoid generating nonsense.

Key takeaways

•

Transformers are the core neural network architecture behind modern AI (ChatGPT, DALL-E, etc.), originally invented for translation but now used for next-token prediction.

•

Input text is broken into tokens, each mapped to a high-dimensional vector (embedding) where semantic relationships correspond to geometric directions (e.g., king - man + woman ≈ queen).

•

The transformer alternates between attention blocks (where tokens exchange context) and feed-forward blocks (where each token is processed independently), progressively enriching token representations.

•

Only the final token's vector is used to predict the next token via an unembedding matrix and softmax, though during training all positions predict their next token simultaneously for efficiency.

•

GPT-3's 175 billion parameters are organized into ~28,000 matrices across eight categories; the embedding and unembedding matrices each contribute ~617 million parameters.

•

Softmax converts raw logits into a probability distribution; temperature controls randomness: low temperature yields deterministic outputs, high temperature increases creativity but risks nonsense.

•

Deep learning models rely on matrix multiplications (weighted sums) with tunable weights; backpropagation enables scaling to billions of parameters without overfitting or intractability.

Notable quotes

“A transformer is a specific kind of neural network, a machine learning model, and it's the core invention underlying the current boom in AI.”

▸ Notable quotes · 0:23

“The story just doesn't actually really make that much sense. But if I swap it out for API calls to GPT-3 instead, which is the same basic model, just much bigger, suddenly almost magically we do get a sensible story.”

▸ Notable quotes · 2:22

“You should draw a very sharp distinction in your mind between the weights of the model, which I'll always color in blue or red, and the data being processed, which I'll always color in gray. The weights are the actual brains, they are the things learned during training, and they determine how it behaves.”

▸ Notable quotes · 12:01

“Think about your own understanding of a given word. The meaning of that word is clearly informed by the surroundings, and sometimes this includes context from a long distance away, so in putting together a model that has the ability to predict what word comes next, the goal is to somehow empower it to incorporate context efficiently.”

▸ Notable quotes · 19:05

“Softmax is the standard way to turn an arbitrary list of numbers into a valid distribution in such a way that the largest values end up closest to 1, and the smaller values end up very close to 0.”

▸ Notable quotes · 23:00

Chapters

•

0:00 — Introduction: What is a Transformer?: Defines GPT (Generative Pretrained Transformer), explains transformers as the core architecture behind modern AI (ChatGPT, DALL-E, Midjourney), and introduces the next-token prediction paradigm.

•

3:03 — High-Level Data Flow in a Transformer: Overviews the pipeline: tokenization → embedding → alternating attention and feed-forward blocks → final unembedding + softmax → next-token probability distribution → autoregressive generation.

•

7:29 — Deep Learning Background: Weights, Matrices, and Backpropagation: Explains machine learning as tuning parameters (weights) via backpropagation; deep learning scales via matrix multiplications; inputs/outputs as tensors; weights only interact through weighted sums.

•

12:27 — Tokenization and Word Embeddings: Details tokenization, the embedding matrix (W_E), high-dimensional vector spaces, semantic directions (gender, country, plurality), dot products as similarity measures, and GPT-3's embedding parameter count (~617M).

•

18:57 — Context, Position, and the Transformer's Goal: Describes how embeddings soak up context through the network, the fixed context window (2048 for GPT-3), and the goal of enriching token representations with long-range dependencies.

•

20:29 — Output Generation: Unembedding and Softmax: Covers the unembedding matrix (W_U), using only the last token for prediction (training efficiency), softmax normalization, logits terminology, and parameter count update (~1.2B total so far).

•

23:57 — Temperature Sampling and Conclusion: Explains temperature in softmax, demonstrates its effect on story generation (deterministic vs. creative vs. nonsense), previews next chapter on attention mechanism, and mentions Patreon early access.

Speakers

•

Narrator — Educator/Host

⋮