“Against strong decision-theoretic realism” by Emery Cooper

N.B. Some of this post argues by analogy between decision theory and values. I expect at least these parts of the post to be unconvincing to anyone who expects sufficiently smart agents to converge on the same values, as some moral realists do. I will not argue against moral realism here (see e.g. this sequence for one such argument).

Note that I take a convergence-based definition of realism for this post. One could hold that there is a truth about the correct decision theory, but that agents won't necessarily converge on it. I don't argue against views like that here. I am interested more in the question of convergence than of truth, because the former bears on whether ASI's decision theory is path dependent. In an upcoming post, I will argue further for path dependence, and for the time sensitivity of interventions to influence AI's decision theory.

This is the third post in our sequence Intro to acausal interactions.

Introduction

In this post, I argue against the following claim, which I call strong decision-theoretic realism: that sufficiently smart agents will all converge on the “correct” decision theory (DT). In doing so, I also argue against a related claim [...]

---

Outline:

(01:08) Introduction

(02:55) Decision theories are self-preserving

(05:14) Decision-theoretic reflection is not like learning new empirical information

(07:03) Decision-theoretic disagreements tend to hit bedrock

(11:47) Anti-path-dependence intuitions might not be enough

(13:47) Failures of selection pressure

(15:39) Does decision-theoretic antirealism imply that decision theory doesn't matter (as much)?

(16:52) Acknowledgements

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
September 29th, 2026

Source:
https://www.lesswrong.com/posts/5ro9kSxhP8mHXGmn4/against-strong-decision-theoretic-realism-1

---

Narrated by TYPE III AUDIO.