“Planning for Extreme AI Risks” by Josh Clymer

Subtitle: Are we ready for this?.

This post should not be taken as a polished recommendation to AI companies and instead should be treated as an informal summary of a worldview. The content is inspired by conversations with a large number of people, so I cannot take credit for any of these ideas.

For a summary of this post, see the thread on X.

Many people write opinions about how to handle advanced AI, which can be considered “plans.”

There's the “stop AI now plan.”

On the other side of the aisle, there's the “build AI faster plan.”

Some plans try to strike a balance with an idyllic governance regime.

And others have a “race sometimes, pause sometimes, it will be a dumpster-fire” vibe.

So why am I proposing another plan?

Existing plans provide a nice patchwork of the options available, but I’m not satisfied with any.

[...]

---

Outline:

(03:50) The tl;dr

(06:36) 1. Assumptions

(09:07) 2. Outcomes

(10:07) 2.1. Outcome #1: Human researcher obsolescence

(13:24) 2.2. Outcome #2: A long coordinated pause

(14:33) 2.4. Outcome #3: Self-destruction

(15:46) 3. Goals

(19:16) 4. Prioritization heuristics

(21:58) 5. Heuristic #1: Scale aggressively until meaningful AI software RandD acceleration

(25:41) 6. Heuristic #2: Before achieving meaningful AI software RandD acceleration, spend most safety resources on preparation

(27:46) 7. Heuristic #3: During preparation, devote most safety resources to (1) raising awareness of risks, (2) getting ready to elicit safety research from AI, and (3) preparing extreme security.

(30:19) Category #1: Nonproliferation

(34:52) Category #2: Safety distribution

(38:06) Category #3: Governance and communication.

(39:34) Category #4: AI defense

(40:31) 8. Conclusion

(42:13) Appendix

(42:15) Appendix A: What should Magma do after meaningful AI software RandD speedups

---

First published:
January 29th, 2025

Source:
https://redwoodresearch.substack.com/p/planning-for-extreme-ai-risks

---

Narrated by TYPE III AUDIO.

---