
“Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming” by Gosha Kozhevnikov, Tyler
Om avsnittet
Subtitle: Automatically finding monitor failures by editing real agent transcripts.
We introduce Screwtape, a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors (monitors that don’t look at the agent's chain of thought) by inserting an attack into a transcript and repeatedly refining it until a monitor no longer catches it. An LLM verifier with access to the edited transcript and an attack description checks whether the attack is realistic, is clearly harmful, and could plausibly succeed in the original environment.
We ran the pipeline on Redwood staff transcripts with our own monitors based on Opus 4.8 and Gemini 3.1 Pro, and then tested the resulting attacks against a held-out monitor from a frontier developer. We ran the pipeline 252 times on 94 unique transcripts. This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer's monitor, and we confirmed them as valid through human review.
You can read more about the implementation and experiments in our paper.
In this post, we provide a high-level overview of the pipeline, how it differs from existing tools, the main lessons we learned, and questions that remain open.
How Screwtape [...]
---
Outline:
(01:31) How Screwtape works
(03:18) How Screwtape differs from existing tools
(07:10) What we learned
(09:56) What the monitor failures can be used for
(10:40) Uncertainties we have about the value of this work
(13:55) Next steps
The original text contained 1 footnote which was omitted from this narration.
---
First published:
October 16th, 2026
Source:
https://blog.redwoodresearch.org/p/screwtape-adversarial-transcript
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Fler avsnitt
Visa alla avsnitt av Redwood Research BlogRedwood Research Blog med Redwood Research finns tillgänglig på flera plattformar. Informationen på denna sida kommer från offentliga podd-flöden.