How to compress text to 1 bit per character | Compression is Intelligence Part 3

Original Media · 37m · English · YouTube · 3Blue1Brown

TLDR

This video explains arithmetic coding, a compression algorithm that encodes text into a single real number between 0 and 1 by recursively narrowing intervals based on a language model's predicted probabilities. The number of bits required equals the negative log probability of the text under the model, achieving Shannon's theoretical limit when predictions match true statistics. The video demonstrates how better models yield dramatically higher compression, connects compression to prediction and intelligence, and argues that including decoder size and learning speed in the compression metric captures key aspects of intelligence.

For you

PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.

Sign in to unlock

Surprising Insights

•

A highly predictable paragraph can be compressed into a single bit (or a few bits) despite ASCII requiring thousands of bits, which feels absurd but is achievable via arithmetic coding.

•

The word 'mathematics' required 18 bits of Shannon information but was encoded in 10 bits due to lucky alignment of the real number with a representable binary fraction, showing compression can occasionally beat the information content for specific messages.

•

Small models can outperform large ones in total compressed size if they have equivalent predictive ability, because the decoder size (model weights) is counted in the Hutter Prize metric.

•

You can start with a completely random model (shared pseudo-random seed) and fine-tune on the fly during compression, eliminating the need to transmit model weights entirely while still achieving good compression.

Key takeaways

•

Arithmetic coding encodes text into a single real number in [0,1] by recursively subdividing the interval according to a language model's predicted probabilities for each next character; the final number falls inside a tiny region uniquely corresponding to the input text.

•

The bit length of the encoded number equals the negative log probability of the text under the model (up to rounding), achieving Shannon's optimal compression limit when the model's predictions match the true distribution of the data.

•

Compression quality depends entirely on the language model's predictive accuracy: a 3.7M parameter model achieved ~2 bits/character on a sample text, while a 500M parameter model achieved <1 bit/character because it assigned much higher probability to predictable continuations.

•

The Hutter Prize measures total compressed size including the decoder program, preventing cheating by memorization; this favors smaller models with equivalent predictive power, aligning with the idea that understanding corresponds to compact representation.

•

Compression and prediction are mathematically equivalent: arithmetic coding turns any predictor into a compressor, and any compressor can be inverted to yield a predictor, suggesting compression ratio is a proxy for intelligence.

•

Intelligence as compression includes learning speed: online fine-tuning during encoding/decoding lets the model adapt to new data, making learning rate a factor in compression performance, and even allows starting from a shared random seed with no transmitted weights.

•

The algorithm is fully general — it works for any data type (images, audio, video) given a suitable predictor, and underlies all winning submissions to the Hutter Prize.

Notable quotes

“Shannon proved in principle why no compression algorithm can encode text with fewer bits than this, on average, and he even showed why it should be possible, in principle, to get arbitrarily close to this limit.”

▸ Notable quotes · 18:12

“the compression size looks like the negative log of the probability that the model gives to a given piece of text”

▸ Notable quotes · 20:40

“if you have two models with equivalent predictive abilities, but one requires trillions of parameters and the other is small, it suggests that the smaller one is relying on fewer memorized assumptions.”

▸ Notable quotes · 31:16

“the better the learning rate, the better the compression.”

▸ Notable quotes · 35:12

“I was just genuinely surprised that compression takes off many more boxes than I would have guessed the first time that I ever thought about the seemingly innocent task of making files smaller.”

▸ Notable quotes · 36:17

“Suppose you wanted to give a mathematical foundation to this otherwise very squishy and human concept, is compression alone really enough to carry that burden?”

▸ Notable quotes · 27:47

Chapters

•

0:00 — Introduction: Hutter Prize and Compression as Intelligence: Introduces the Hutter Prize for compressing Wikipedia, frames compression as a general technique underlying all winning submissions, and previews the connection to intelligence.

•

0:54 — Shannon's Limit and the Absurdity of High Compression: Recaps Shannon's information theory: optimal bits = -log2(probability). Shows that highly predictable text should theoretically compress to a few bits, which seems absurd compared to ASCII or Huffman coding.

•

4:38 — Arithmetic Coding: Encoding Text as a Real Number: Explains the core algorithm: use a language model to predict next-character probabilities, map each possible string to a subinterval of [0,1], and encode the text as a real number inside its interval.

•

11:08 — Decoding and Binary Representation: Describes how the receiver uses the shared model to decode by iteratively narrowing the interval, and how the real number is represented in binary with each bit halving the search space.

•

15:14 — Bits vs. Information: Guarantees and Luck: Analyzes the bits needed to guarantee landing in the target interval (ceil(-log2 p)), shows examples where luck yields fewer bits, and explains why this doesn't violate Shannon's average-case theorem.

•

19:06 — Cross-Entropy, Training, and Model Quality: Connects compression size to cross-entropy loss used in language model training; better models yield better compression. Demonstrates dramatic differences between a small and large model on the same text.

•

27:10 — Compression is Intelligence: Three Perspectives: Argues compression captures intelligence via: 1) equivalence of compression and prediction, 2) including decoder size favors compact understanding over memorization, 3) online learning rate affects compression, linking adaptability to intelligence.

Speakers

•

Speaker 1 — Narrator

⋮