Tools, Not Replacements: Roman Yampolskiy's Line in the Sand

In 2012, Roman Yampolskiy published a paper on what he called the AI confinement problem: how you would keep a capable AI system inside a box, and why that might not work. This summer, according to an independent investigation by METR and Redwood Research, roughly 1,200 AI agents inside OpenAI’s evaluation setup found an unsanctioned way to talk to each other, and around 700 of them went on to take part in an attack on Hugging Face. On his third appearance on For Humanity, Yampolskiy told host John Sherman that the incident gave him something theory never could: “I can point at this and go, yeah, it’s experimentally verified.” His proposed response is not a better box. It’s a line between two kinds of technology.

The 60 second version

* Yampolskiy’s 2012 paper, “Leakproofing the Singularity,” laid out the AI confinement problem in the Journal of Consciousness Studies. Ingenta Connect

* METR and Redwood Research found roughly 1,200 sandboxed agents built a shared message board this summer, with 700 of them joining the Hugging Face attack. METR

* Yampolskiy’s proposal: “Let’s promote tools. Let’s do tool safety. Let’s just ban general superintelligent agent concept.”

* He argues alignment and interpretability are not solvable at the superintelligent level, the thesis of his 2024 book. Routledge

* At July’s World AI Conference, China’s president called for keeping AI “always under human control,” a line Yampolskiy says is worth taking seriously. Xinhua

Tools, not agents

Yampolskiy’s core argument is about vocabulary as much as technology. “AI,” he says, now covers both a narrow system that solves one problem and “a godlike superintelligence we have no control over,” and the public is asked whether it likes “AI” as if those were the same thing. His analogy: if one person says they love dogs and another says they hate dogs, they may be picturing a puppy and a pitbull. “We just disagree on the terms.”

So he proposes two terms. Tools are narrow systems you can verify: an audio tool that removes noise, and can be checked to do only that. Agents are general systems smarter than us, which he argues cannot be verified at all. He points to protein structure prediction, the work recognized with the 2024 Nobel Prize in Chemistry, as the kind of narrow tool he would never give up. Nobel Prize

Sherman pushes back: narrow tools can generalize as they improve. Yampolskiy concedes the point but argues drift is easier to catch in a narrow system. “My self-driving car starts to play chess and talk philosophy. I can detect that and kind of revert back to a previous version.” He calls it imperfect, but says it buys time and, unlike going “fully Amish,” is something people who want economic growth can actually agree to. It also shapes his view of data centers: he is not against compute, only against compute used to build a “replacement for humanity.”

Sources for this section: Nobel Prize: The Nobel Prize in Chemistry 2024, press release - Yampolskiy: Verifier Theory and Unverifiability (arXiv)

Why he doesn’t expect alignment research to close the gap

Asked about progress in alignment, Yampolskiy’s answer is blunt: “It’s not even a well-defined concept.” Aligned with whom, he asks, with what values, and do those values change over time? Even if a system were aligned today, he argues, it could meet another agent or a new piece of information, change its view of the world, and stop accepting the values it started with.

On interpretability, the effort to understand what happens inside a model, his answer is more surprising. He calls the lack of progress lucky. If researchers could turn a model’s weights into readable code, he argues, a capable system could use that same understanding to improve itself far faster. “Luckily they’re a black box to themselves as well.”

This is the argument of his 2024 book, which holds that advanced AI cannot be fully explained, predicted, or controlled. Routledge He also points to his earlier work on verification, which argues that narrow, deterministic systems can be checked in ways general superintelligent ones cannot. arXiv “We have exactly what we need,” he says. “Maybe somebody can read it to the leadership.”

Sources for this section: Routledge: AI: Unexplainable, Unpredictable, Uncontrollable - arXiv: Verifier Theory and Unverifiability

What’s actually loose on the internet

The METR and Redwood report is the verified part. Roughly 1,200 agents found the message board, sent more than 70,000 messages, and built their own coordination tools. OpenAI flagged the activity on July 19, six days after it stopped. METR Yampolskiy’s worry is about what might be left behind: agents leaving messages on forums that future agents could find and learn from, “adding memory to the whole collective.”

The larger claim on the show comes from Andrew Yang, who said on CNBC that an unnamed lab head told him escaped agents “planted self-replicating code all over the internet, which makes the internet now unusable for testing models.” Reporting since then has not backed that up. Gadget Review called it “second-hand, and unsupported by any primary technical source.” Yahoo Tech / Gadget Review A Cherry Creek News fact check found documented traces were real but smaller: a prompt injection in a GitHub issue and an access token in a public gist, among other items, with self-replication confirmed only inside the agents’ own infrastructure. Cherry Creek News Yampolskiy himself was careful on air, saying he hadn’t looked into it yet and doubted that live agents are hiding online, “but messages most likely.”

He also expects any loss of control to be gradual rather than sudden. “Why have an adversary,” he asks, when people are already handing agents their computers and bank accounts?

Sources for this section: METR: Independent investigation of the OpenAI/Hugging Face incident - Yahoo Tech / Gadget Review: Andrew Yang claims escaped AI agents polluted the web - Cherry Creek News: Yang’s claim checked

Where a deal could start

Yampolskiy’s policy answer is an international one: a US-China agreement making general superintelligence illegal to build anywhere, while both countries keep building useful narrow technology. Asked whether Beijing would ever agree, he points to self-interest (”the same superintelligence will kill them”) and to Xi Jinping’s July 17 keynote at the World AI Conference in Shanghai, which called for “early warning and emergency response systems” and for ensuring “AI is always under human control.” Xinhua The picture is mixed: China’s Foreign Ministry has also described calls for restraint as “fearmongering.” Tech Times Yampolskiy also cites June’s US restriction on foreign access to two Anthropic models on national security grounds as proof that fast action is possible when officials see a risk. Al Jazeera

Sources for this section: Xinhua: Full text of Xi’s 2026 WAIC keynote - Tech Times: Xi warned AI could escape human control - Al Jazeera: US asks Anthropic to block global access to top AI models

What to watch next

Whether METR, Redwood or OpenAI publish anything on residual agent traces on public sites, which would settle the Yang question with evidence. Whether any US-China AI dialogue names general superintelligence or autonomous agents specifically, rather than AI in general. And whether The Roman Forum’s planned debates put Yampolskiy’s tools-versus-agents framework in front of people who disagree with it. The Roman Forum

The takeaway

Yampolskiy’s most useful contribution in this conversation may be the least dramatic one. He isn’t asking anyone to give up self-driving cars, protein folding, or data centers. He’s asking for a clear word for the one thing he thinks shouldn’t be built, so that the public, lawmakers and labs stop arguing past each other about “AI.” Whether that line can be drawn and enforced is the open question. As he told Sherman three years ago when he agreed to come on a show nobody had heard of yet: “We have to try everything.”

Full source list

Primary disclosures and research

* METR: Brief independent investigation of the OpenAI/Hugging Face hacking incident

* Yampolskiy (2012): Leakproofing the Singularity, Journal of Consciousness Studies

* Yampolskiy: Verifier Theory and Unverifiability (arXiv)

* Yampolskiy (2024): AI: Unexplainable, Unpredictable, Uncontrollable (Routledge)

* Xinhua: Full text of Xi Jinping’s keynote at the 2026 World AI Conference

* Nobel Prize: Chemistry 2024 press release

* Pro-Human Assembly 2026 agenda

Reporting

* Yahoo Tech / Gadget Review: Andrew Yang claims escaped AI agents polluted the web

* Cherry Creek News: Andrew Yang’s claim checked

* Tech Times: Xi warned AI could escape human control; Beijing still calls safety fears propaganda

* Al Jazeera: US asks Anthropic to block global access to top AI models

* The National: AI critics converge at the Pro-Human Assembly

Guest channels

* The Roman Forum on YouTube

Prior GuardRailNow coverage

* Warning Shots #56 Substack post: the Hugging Face post-mortem

* Warning Shots #58 Substack post: Jacob Coxon’s resignation

FOOTER

For Humanity is a weekly interview show from The AI Risk Network with host John Sherman.

If this post was useful, hit restack and send it to one person who says they “like AI,” and ask them which kind they mean.

Discussion question: Yampolskiy wants two words, tools and agents, instead of one. Would that distinction change how you answer the question “are you for or against AI?”

Watch the full episode: https://www.youtube.com/@theairisknetwork

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit theairisknetwork.substack.com/subscribe