TLDR
NVIDIA's paper "You don't need long context" challenges the assumption that large context windows are essential for long-horizon and long-input tasks. The authors show that two compression strategies — end-of-conversation compaction (EoCC) and single-action compaction (SAC) — maintain accuracy within a few percentage points while reducing peak context from 200K to under 10K tokens. A selective-reading approach using grep-like tools outperforms full-context ingestion on retrieval benchmarks. However, cost savings are modeled assuming tiered pricing (not current flat-rate APIs), benchmarks favor retrievable tasks, and open models like Qwen 3 Coder Next collapse under tight context budgets.
For you
PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.
Sign in to unlockSurprising Insights
Feeding a model an entire 200K-token codebase at once performs worse than letting it grep for relevant files, because the model still has to locate the needle in the haystack.
A model that sees only ~9K tokens per step (≈10 pages of Chinese text) can solve tasks that previously required 200K context, with no accuracy loss on several benchmarks.
Under current flat per-token pricing, using compression strategies can actually increase nominal costs because you pay for extra compression calls without a volume discount.
Key takeaways
Extending context from 16K to 256K on H100 GPUs increases per-million-output-token cost by 3.6× to 7× across models like GLM 5.2, Kimi K2.6, and Qwen 3 Coder Next, because attention computation and KV-cache memory scale superlinearly.
EoCC (end-of-conversation compaction) keeps context between 16K–48K and matches full-history accuracy on six long-horizon benchmarks (SWE-bench, Terminal-Bench, FRAMES, etc.) with modeled cost savings of 2.2–8.8× for Opus and 2.4–5.0× for GPT, but only if providers adopt context-length-tiered pricing.
Selective reading — storing long inputs as files and letting the model grep/sed/awk relevant passages — beats full-context baselines on seven retrieval-heavy benchmarks: RULER recall jumps from ~63% to ~96%, BABILong from 67–71% to 92–94%, because the model skirts the needle-in-haystack search problem entirely.
SAC (single-action compaction) compresses after every tool call or reply, limiting median peak context to 7K–9K tokens (max 15K–37K) while holding SWE-bench Verified at 70.2% vs 70.0% baseline for Opus and improving Terminal-Bench from 47.2% to 51.8%.
The paper proposes a Semantic State Space Model (SSSM) where models are trained to cycle through: read natural-language state, reason next action, execute, observe, update state — replacing opaque neural hidden vectors with auditable, human-readable state.
Qwen 3 Coder Next collapses from 53.0% to 3.6% on SWE-bench at 16K context because its verbose output trajectories leave no room for progress between compressions, showing short-context viability depends heavily on model verbosity and training.
All tested models (Claude Opus 4.5, GPT 5.6 Sol) were themselves trained on long context; the paper only proves runtime context can stay small, not that long-context pretraining is unnecessary.
Notable quotes
“你不需要长上下文”
▸ Notable quotes · 0:07“长上下文模式下只能生成七分之一的token”
▸ Notable quotes · 0:24“好的压缩不是简单摘要……它是为未来工作挑选事实的简报,不是执行历史的叙述”
▸ Notable quotes · 3:18“把长输入当文件按需读取,效果就比一次性全塞进去好”
▸ Notable quotes · 6:20“每次调用模型只让它看最多9K token……居然能完成原来需要200K上下文才能完成的任务”
▸ Notable quotes · 8:02“长上下文依然是有价值的能力,只是未必需要成为每一步推理的默认运行方式了”
▸ Notable quotes · 12:12Chapters
0:00 — Paper motivation and cost analysis: Introduces NVIDIA's paper "You don't need long context" and shows H100 cost measurements: extending context from 16K to 256K raises per-million-output-token cost 3.6×–7× across major models.
1:50 — Three research questions and strategies: Frames three questions — long-horizon tasks, long-input tasks, single-action compression — and defines baseline FHA plus two compression strategies EoCC and SAC, plus a selective-reading approach for long inputs.
3:15 — Compression template design: Details the five-component compression format: original task, current intent/next plan, decomposed steps, completion conditions, verified facts and dead ends — a brief for future work, not a narrative of past actions.
3:51 — Long-horizon task results (EoCC): On six benchmarks with Opus and GPT, EoCC at 16K–48K context matches full-history accuracy (±3–5%) with modeled 2–8× cost savings, contingent on tiered pricing.
5:42 — Long-input task results (selective reading): Selective reading via grep/sed/awk beats full-context on seven retrieval benchmarks: RULER 63%→96%, BABILong 67–71%→92–94%; gains come from retrieval, not compression.
7:23 — Single-action compression (SAC) results: SAC holds Opus accuracy on SWE-bench (70.2% vs 70.0%) and improves Terminal-Bench (51.8% vs 47.2%) while cutting peak context from 130–200K to 15–37K (median 7–9K).
8:11 — Semantic State Space Model proposal: Proposes SSSM: models trained to cycle through readable natural-language state — read, reason, act, observe, update — replacing opaque hidden vectors with auditable state.
9:00 — Caveats and industry implications: Lists six caveats: modeled not real costs, retrieval-biased benchmarks, no whole-understanding tasks, uncounted tool orchestration costs, narrow model sample (Qwen collapses), and runtime ≠ training. Suggests Agent memory shift to state + external retrieval, new evaluation metrics, and relevance for local inference.
Speakers
大飞 — Host