“How to prevent collusion when using untrusted models to monitor each other” by Buck Shlegeris

Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques, because by default you are unlikely to notice the problem until the hard-to-reverse outcome has already occurred.

One of the main classes of techniques that you could apply in the hope of preventing these catastrophic actions is what we call untrusted monitoring (introduced in our paper here), which is roughly “before you execute a model's action, ask another instance of that model whether the [...]

---

Outline:

(06:31) A basic design for untrusted monitoring

(07:19) Untrusted monitoring during deployment

(12:22) Training your untrusted monitor

(14:28) How helpful would this be if the untrusted monitor did its best?

(15:56) Analyzing this under the adversarial assumption: can the model implement a conditional policy that only lies on real attacks?

(20:48) Rare failures aren't an option for untrusted monitors

(22:01) How hard is it for a schemer to distinguish real examples of attacks from synthetic ones?

(22:23) Synthetic input contexts look obviously fake

(25:25) The monitor finds it naturally easy to distinguish synthetic attacks from real attacks

(29:23) Using a coordinated strategy that makes it easy to distinguish

(34:28) Preventing collusion under the assumption that the models can choose a coordination strategy

(38:06) A note on inference costs

(39:22) Conclusion

(41:26) Appendix: More on how helpful untrusted monitoring would be if the untrusted model did its best

(44:29) Appendix: A rule of thumb for estimating safety with execute/replace/audit protocols

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
September 25th, 2024

Source:
https://redwoodresearch.substack.com/p/how-to-prevent-collusion-when-using

---

Narrated by TYPE III AUDIO.

---