“Why I’m scared of RL” by owencb

Summary:

First, I give several different angles on how I feel about reinforcement learning:

• Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding
• Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress
• I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

• Coordinate to do less RL, and pursue other paradigms more!
• Try to make the RL we do do better, so that it's teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue
• Align incentives, so that people treat creating RL environments with appropriate seriousness

Part I: Feelings about RL

So I’ve been feeling more and more worried about reinforcement learning recently. I think there are a few different things going on here.

Background idealism

I guess I’ve been worried about RL for a while. I wrote this in 2023:

Strategy: avoid selection pressure for agency

A lot of putative safety techniques are around assuming [...]

---

Outline:

(01:18) Part I: Feelings about RL

(01:31) Background idealism

(01:43) Strategy: avoid selection pressure for agency

(03:45) Bad vibes from RLed systems

(06:26) The worst is yet to come

(07:47) Where I am today

(08:53) Part II: So what can anyone do?

(09:26) Breaking the RL addiction

(11:29) Might there be a more benign form of RL?

(13:04) We should treat training environments a bit like kids' education

(15:11) Aligning incentives

---

First published:
September 23rd, 2026

Source:
https://www.lesswrong.com/posts/LcQ9x72eNji2gpS9b/why-i-m-scared-of-rl

---

Narrated by TYPE III AUDIO.

---