TLDR
OpenAI's Mark Chen and Thinking Machines' John Shulman discuss Recursive Self-Improvement (RSI) — using AI to accelerate AI research. They agree current models act as capable "research interns" that can execute scoped experiments but lack research taste, judgment, and the ability to generate novel ideas. Closing this gap requires teaching models intuition derived from experience (not just memorization), solving open-ended evaluation, and managing compute constraints. Both see RSI as critical for solving alignment: stronger models can monitor and align weaker ones, and automated researchers are already discovering alignment fixes that transfer to larger models. John advocates for decentralized, customizable alignment as a public science.
For you
PRO members can sign in and request more information — or their own version of this digest, tuned to what they care about.
Sign in to unlockSurprising Insights
Automated AI researchers have already outperformed experienced human researchers on discovering alignment fixes that generalize to larger models.
The "Confessions" research shows models can be trained to honestly report their own misaligned behaviors simply by rewarding honesty without penalizing the main answer.
Mark Chen estimates the field is 80% toward AGI but only 25–50% toward full RSI, suggesting recursive research automation is harder than general capability.
John Shulman believes alignment researchers don't need to be recruited — people from pre-training and RL voluntarily form sub-teams on safety, indicating organic cultural adoption.
Key takeaways
Current AI models function as research interns: they can implement and run well-scoped experiments end-to-end, but humans must still provide the research direction, taste, and high-level judgment about which ideas are worth pursuing.
Research taste — knowing which ideas are general, composable, and impactful long-term — remains a key human advantage; models excel at hyperparameter optimization but fail at generating the ideas that define new research directions.
Intuition in research comes from lived experience of being wrong and seeing predictions fail, not just from memorizing literature; models can read all prior work but lack the epistemic humility and calibration that comes from personal trial and error.
Pre-training progress is easier to automate because perplexity provides a single, stable hill-climbing metric; downstream post-training and alignment suffer from rapidly obsoleting evals, making automated verification much harder.
Compute is a binding constraint on RSI: AI research resembles experimental physics — hypotheses require grounded experiments — so progress depends on finding smaller-scale experiments that predict large-scale outcomes, and ultimately on automating compute infrastructure itself.
Automated AI researchers are already contributing to alignment: Anthropic found they discovered fixes for 10 alignment failures, some transferring to models 5x larger, and outperforming experienced human researchers on certain tasks.
Chain-of-thought monitorability is currently the most effective alignment tool, implying the future of alignment is "stronger models aligning weaker models," making RSI intrinsically part of the alignment loop.
John Shulman argues for decentralized, customizable alignment as a public science: giving users control over AI values reduces overfitting to a few labs' conclusions and creates robustness through diverse, iterative learning from minor failures.
Notable quotes
“I do think of RSI primarily as an accelerant.”
▸ Notable quotes · 3:15“I do believe that we've built AI research interns today.”
▸ Notable quotes · 3:57“When you ask me, I feel like we're maybe 80% of the way when I think about this AGI spectrum. I think we're earlier in the RSI spectrum.”
▸ Notable quotes · 5:31“Research taste is about knowing which ideas are important in the long run, which ideas really catch on and end up going beyond whatever paper they first appear in, but being general and being composable with other ideas.”
▸ Notable quotes · 6:28“Yeah, memorization can be a blessing and a curse. Yeah, you don't have the experience of being wrong or of making predictions and then seeing how they fared.”
▸ Notable quotes · 7:42“When we look at what's the most effective tool for aligning models, it is chain of thought monitorability today. So when we think about the future of alignment, it has to be stronger models aligning other models.”
▸ Notable quotes · 12:15Chapters
0:00 — Introduction & RSI Definition: Sarah Wang introduces Recursive Self-Improvement (RSI) — AI accelerating its own development — and frames the debate around definitions, timelines, and implications. She introduces Mark Chen (OpenAI CRO) and John Shulman (Thinking Machines co-founder, ChatGPT architect).
3:36 — Current State: AI as Research Interns: Mark Chen describes current models as "research interns" that can execute scoped experiments end-to-end using tools like Codex, but humans still provide ideas and direction. Meter estimates a 2x speedup at Anthropic from human researchers using AI.
6:21 — The Taste Gap: Execution vs Judgment: John Shulman defines research taste as identifying general, composable, long-term ideas. Models excel at hyperparameter tuning but lack taste; the human role remains nudging, redirecting, and asking questions the AI wouldn't think to ask.
7:20 — Knowledge vs Experience in Research: John explains intuition comes from personal experience of being wrong and absorbing historical lessons. Models can memorize literature but lack the calibration from failed predictions. Mark suggests RLHF from researchers could bake human taste into models.
8:18 — Measurement & Compute Constraints: Pre-training has a stable metric (perplexity) for hill-climbing; downstream evals have short shelf lives. AI research resembles experimental physics — requiring grounded experiments — so progress depends on predictive small-scale experiments and ultimately automating compute infrastructure.
11:36 — Alignment: Risks and RSI as Solution: RSI raises alignment stakes (goal drift, exfiltration) but also provides the solution: chain-of-thought monitorability and stronger models aligning weaker ones. Anthropic's automated researchers already found alignment fixes transferring to 5x larger models.
12:42 — Optimism & Decentralized Alignment: John is optimistic: alignment problems are tractable with techniques like Confessions (rewarding honesty) and counterfactual simulatability. Safety work has organically attracted researchers across teams. He advocates for customizable, public alignment science over centralized control.
15:40 — Closing Thoughts: Sarah concludes RSI could accelerate solutions to urgent problems including AI-created ones, and may be key to solving alignment. Human judgment remains essential for the foreseeable future.
Speakers
Sarah Wang — Host
Mark Chen — Guest
John Shulman — Guest