“Risk from fitness-seeking AIs: mechanisms and mitigations” by Alex Mallen

Subtitle: Fitness-seeking is increasingly what misalignment looks like in practice—how should we respond?

Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call “fitness-seeking“—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern.

In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis.

Fitness-seekers are, in many ways, notably safer than what I’ll call “classic schemers”. A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world's resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied1 with modest-to-trivial [...]

---

Outline:

(03:50) Overview

(11:32) The basic reasons fitness-seekers might be safer than classic schemers

(16:01) Four mechanisms for risk and their mitigations

(16:46) Potemkin work

(22:13) Instability

(27:49) Manipulation

(31:25) Outcome enforcement

(36:52) Cross-cutting mitigations

(37:18) Deals

(40:05) Control

(44:45) Alignment

(44:48) Preventing fitness-seeking from arising

(48:51) Making any fitness-seeking motivations safer

(51:32) How does online training change the picture?

(55:09) Overall recommendations

(59:12) Conclusion

The original text contained 18 footnotes which were omitted from this narration.

---

First published:
May 1st, 2026

Source:
https://blog.redwoodresearch.org/p/risk-from-fitness-seeking-ais-mechanisms

---

Narrated by TYPE III AUDIO.