“The likely outcome of an AI pause is that we unpause too early and everyone dies” by MichaelDickens

Cross-posted from my website.

As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay. But my old mental model underrated the likely possibility that we get a global pause on AI, solve a problem that looks superficially like the alignment problem, resume scaling, and then proceed with building a misaligned superintelligence that kills everyone.

A lot of people have become more concerned about misalignment recently. This seems driven by the fact that current AI models are visibly misaligned. But ASI misalignment is a whole different ball game. The primary danger comes from AI that's smarter than people, and smart enough to conceal any evidence of misalignment.

Whatever group of people makes the decision to unpause, I'm worried that they won't understand the difference between visible and actual misalignment, and they will unpause too early.

source: MetaKnowing on reddit. This meme is almost a year old but it's only gotten more relevant since then.

Case in point: AI companies keep calling their new models "our most aligned model ever!" when what they actually mean is [...]

---

Outline:

(02:04) If we unpause when the legible problems are solved, we die

(03:28) A pause alone doesn't get us to a science of alignment

(04:50) What would change things?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
September 28th, 2026

Source:
https://www.lesswrong.com/posts/PgbAnqdqdJcrS2Gsy/the-likely-outcome-of-an-ai-pause-is-that-we-unpause-too

---

Narrated by TYPE III AUDIO.

---