An AI disaster hits. The president calls Xi. Then what? | Jeremie Harris (crosspost from NonZero)

This is an (unpaid) cross-post of an episode from the podcast NonZero, which Rob Wiblin thought you might be interested in. The originals are on the NonZero website: Part 1 and Part 2. If you like it and want more similar stuff in the future, consider subscribing to NonZero on YouTube, Spotify, Apple, or wherever you get podcasts.

What will it take for Washington to slow down AI? Not better arguments, says Jeremie Harris, CEO of Gladstone AI and coauthor of a 2024 State Department–commissioned action plan on AI risk. He argues it will take an “AI Chernobyl”: a disaster so severe the president fears for their own life, and Washington’s official stance on AGI risk flips.

Jeremie has spent six years betting that AI’s scaling curve won’t bend. His biggest miss ran the other way: “Frankly, I thought we ought to be dead by now.” Capabilities keep improving almost automatically, but safety doesn’t, because “no one knows how to make a curve of control or a curve of alignment.”

He was making that bet before it was respectable. Jeremie recounts his years trying to get people in government to take AI risk more seriously, including his role in the battle to get “AGI” into a Biden-era State Department report.

He also takes aim at Open Philanthropy, now Coefficient Giving (80,000 Hours’ largest funder), which was then the dominant funder in AI safety. Jeremie claims it discouraged outsiders from briefing Washington, for fear officials would hear “powerful weapon” and launch a Manhattan Project for superintelligence. He criticises its long timelines and a grantee culture of telling funders what they wanted to hear, which he says left the field slow to engage government. Still, he credits Coefficient Giving with getting the problem “much more right than almost anyone.”

His assessment of the AI companies is worse. Jeremie thinks OpenAI has gone through “evaporative cooling,” shedding its most safety-minded staff round after round. He also doubts Sam Altman ever fully believed his own warnings.

We’re sharing this because Jeremie’s takes on warning shots, government response, and past mistakes in AI governance were new to us. Not everyone will remember the history the way he does, but it’s provocative in a useful way.

Chapters:

• Rob’s intro (00:00:00)
• Jeremie’s podcast and prescient AI report (00:01:31)
• Is scale still (mostly) all you need? (00:07:32)
• Jeremie: We’re way past “stochastic parrots” (00:13:45)
• Inside the early AI safety ecosystem (00:22:35)
• What Yudkowsky got less wrong than most (00:34:24)
• Why capability is outpacing alignment (00:42:18)
• Were Altman’s fears ever sincere? (00:49:49)
• Would pausing AI crash the economy? (00:56:04)
• Overtime preview: Debating China (01:01:45)
• Bob’s beef with “kumbaya” (01:10:22)
• Will it take an AI Chernobyl to convince the skeptics? (01:12:19)
• Is China-hawkism the safest option? (01:22:55)
• The case for “organic transparency” with China (01:50:43)
• How alignment progress could backfire (01:58:02)
• Has China infiltrated US tech? (02:03:18)
• Breaking down Dario’s democracy bloc idea (02:12:38)