“Not Pinning Your OpenRouter Provider Might Invalidate Your Research” by Matthew Khoriaty

Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted.

• When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available provider.
• OpenRouter providers have variable quality. Ensuring that your provider is high quality is really difficult. 
• There is precedent for an AI safety paper accepted to NeurIPS having its core results entirely overturned by these issues.
• A review of influential AI Safety research codebases that use OpenRouter for their reported results found that 31/32 (97%) of them use OpenRouter unsafely.[2]
• Researchers who wish to do research using OpenRouter or similar providers should take precautions to minimize the risks to their research,[3] though the current selection of providers is insufficient for ideal scientific reliability.
• Replicators should test whether results hold up when these bugs are fixed.
• Core AI Safety codebases should make fixes so that downstream users are able to implement best practices.

This is a tangent I took for a few days during the Pivotal AI Safety Research Fellowship. I’m doing an AI Control project mentored by Adam Kaufman and James Lucassen of Redwood Research and partnered with Aniruddh [...]

---

Outline:

(02:04) OpenRouter Kills an AI Safety NeurIPS Paper

(03:29) Background

(04:15) Prior work/writing on the subject.

(05:38) No Guarantees from Fixing a Provider

(06:14) No Guarantees From Large N

(07:09) No Guarantees from "The Benchmark Didn't Move"

(08:19) No Guarantees from "Choose the Most Expensive Provider"

(08:54) No Guarantees from "Set a floor"

(09:12) When is Default OpenRouter Fine?

(09:50) Developing Best Practices

(11:45) Spotting Fingerprints

(13:12) Highlighting Some Impacted Work

(13:57) en-US-AvaMultilingualNeural__ Infographic titled "Provider Routing Inspector" with statistics and bar chart on OpenRouter reliability.

(14:10) Should The Community Do Something About This?

(14:15) Fixing Repos

(14:34) Replicating Research

(15:01) A New Org?

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
July 23rd, 2026

Source:
https://www.lesswrong.com/posts/KsyoSAyBRXtwzSugg/not-pinning-your-openrouter-provider-might-invalidate-your

---

Narrated by TYPE III AUDIO.

---