LessWrong (30+ Karma)

“On METR’s AI Coding RCT” by Zvi

21 min • 18 juli 2025

METR ran a proper RCT experiment seeing how much access to Cursor (using Sonnet 3.7) would accelerate coders working on their own open source repos.

Everyone surveyed expected a substantial speedup. The developers thought they were being substantially sped up.

Instead, it turned out that using Cursor slowed them down.

That surprised everyone, raising the question of why.

Currently our best guess is this comes down to a combination of two factors:

  1. Deeply understood open source repos are close to a worst-case scenario for AI tools, because they require bespoke outputs in various ways and the coder has lots of detailed local knowledge of the codebase that the AI lacks.
  2. The coders in question mostly did not have experience with similar AI tools. The lack of a learning curve during the experiment challenges this, but the tools very clearly have a [...]

---

Outline:

(01:27) Epic Fail

(02:42) The Core Result

(07:10) Okay So That Happened

(12:21) Beginner Mindset

(19:43) Overall Takeaways

---

First published:
July 18th, 2025

Source:
https://www.lesswrong.com/posts/m2QeMwD7mGKH6vDe2/on-metr-s-ai-coding-rct

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Table listing
Four-panel meme using Gru character presentation format about AI's impact on coding speed
Table titled
Graph showing
Bar graph comparing developer forecasts versus actual implementation times for AI-allowed and AI-disallowed tasks.
Table showing
Graph showing AI's impact on developer productivity, forecasts versus actual results

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Senaste avsnitt

Podcastbild

00:00 -00:00
00:00 -00:00