World Models | John Langford (Microsoft AI Labs)

John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.

We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.

Topics:

• Implicit vs. explicit world models, and the case against JEPA-style objectives
• Compact belief states: why compression beats a growing KV cache
• Does research still matter in the age of scale? The Kimi K3 argument
• Agent-driven research: 2,000 pre-training experiments in 90 days
• The invention of CAPTCHA
• Optimizers from SGD and Vowpal Wabbit to Muon

• ---

Chapters

• 00:00 Why world models: the sample-complexity gap
• 09:48 The case against JEPA; a transformer-style implicit world model
• 15:52 Compact belief states: epicycles vs. heliocentrism
• 23:41 Does research still matter? The Kimi K3 argument
• 27:35 Open vs. closed models
• 35:57 Recursive self-improvement and agent-driven research
• 42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility
• 54:54 The invention of CAPTCHA
• 1:00:53 Optimizers: from Vowpal Wabbit to Muon

• ---

  Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0