“Categorical taboos are much better than threshold taboos: neuralese edition” by Linch

I think what's going on in the “Does Astra use neuralese?” debate is that there's an important sense in which models *already* do self-communication in neuralese: between each layer in the forward pass the attention stream is already very hard to interpret, and clearly not in natural language.

Yet CoT monitorability is still a big deal and it'd be bad if all self-communication from models are no longer in natural language. So there have been two different proposed definitions of what is "true" neuralese:

1. (My preferred) categorical definition: Since natural language currently gates recurrence in the standard transformer+CoT loop, having recurrence in neuralese is the natural category for whether something counts as "true" neuralese.
2. The threshold definition. Total number X of serial steps before something appears in natural language. True neuralese counts as going above X. I think most technical experts who studied this issue, including many people at companies, prefer definition #2.

There are complicated technical arguments on both sides, but I think technical experts overall prefer #2 because they think it's more causally relevant (there's nothing inherently more difficult about monitoring a 128-layer model looped 8x than monitoring a 1024-layer model), have less weird edge cases [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
September 10th, 2026

Source:
https://www.lesswrong.com/posts/xPkmfsZ3qx4nrAco7/categorical-taboos-are-much-better-than-threshold-taboos

---

Narrated by TYPE III AUDIO.

---