He Leads a Top AI Research Program, But He’d Hit the PAUSE Button Today! Kevin Zhu, Algoverse Founder

Kevin Zhu walked away from a lucrative quant career to build Algoverse, one of the most productive mentorship programs for aspiring AI researchers. Can we agree on the best research path to a safe future? Let’s take a ride on the Doom Train!

00:00:00 — Cold Open

00:00:58 — Introducing Kevin Zhu

00:02:10 — From Citadel Quant to AI Researcher

00:09:14 — The Story of Founding AlgoVerse

00:12:53 — Discovering AI Safety: LessWrong & ARENA

00:17:22 — Emergent Misalignment Research

00:22:37 — Yudkowsky, MIRI & “Intellidynamics”

00:26:50 — What’s Your P(Doom)?™

00:29:37 — Kevin’s Timeline to AGI + AI 2027

00:30:42 — Would You Slow Down AI?

00:37:44 — Coming Out of the P(Doom) Closet

00:45:49 — Should We Shame AI Company Workers?

00:52:27 — OpenAI’s Superalignment Team Collapse

00:55:01 — Riding the Doom Train™

00:55:46 — First Stop: Instrumental Convergence

01:04:58 — Does Kevin Agree with the Orthogonality Thesis?

01:07:08 — “It’s Just Math.” Just Unplug It.

01:08:28 — “We Have a Safe Development Process”

01:11:49 — Group Dynamics & Laws Will Save Us

01:13:44 — Superintelligence Will Spare Us

01:14:25 — Is P(Doom) Just Bad Epistemology?

01:15:53 — China Will Race No Matter What

01:17:45 — Maybe Human Extinction Is Good?

01:20:36 — Wrap-Up

Links

Algoverse AI Research — https://algoverseairesearch.org/

Kevin Zhu on Instagram — https://www.instagram.com/kevinzhu.ai/reels/

Emergent Misalignment (Betley, Owain Evans et al.) — https://arxiv.org/abs/2502.17424

Emergent Misalignment via In-Context Learning (Algoverse follow-up, ACL 2025) — https://arxiv.org/abs/2510.11288

Agentic Misalignment (the blackmail result) — https://www.anthropic.com/research/agentic-misalignment

LessWrong — https://www.lesswrong.com/

PauseAI (global) — https://pauseai.info/

OpenAI Superalignment — disbanded (CNBC, May 2024) — https://www.cnbc.com/2024/05/17/openai-superalignment-sutskever-leike.html

Evan Hubinger — “Why I’m joining Anthropic” — https://www.alignmentforum.org/posts/7jn5aDadcMH6sFeJe/why-i-m-joining-anthropic

AI 2027 — https://ai-2027.com/

Bentham’s Bulldog / Matthew Adelstein on Doom Debates — https://doomdebates.com/p/benthams-bulldog-ai-doom-debate

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏