Skip to content
MUNEEB SHAFIQAll writing
Email

5 min readllm-evaluation · developer-productivity · measurement

Two studies, opposite answers, and why both are right

One controlled trial found AI made developers 55.8% faster. Another found experienced developers 19% slower, while they believed they'd been sped up by 20%. The studies don't contradict each other, and the reason they don't is the useful part.

Muneeb Shafiq · Associate AI Engineer, Symufolk · Every claim below was checked against a primary source.

If you want to know whether AI coding tools make developers faster, the literature will hand you two numbers that point in opposite directions.

+55.8%. Peng, Kalliamvakou, Cihon and Demirer ran a controlled experiment where recruited developers were asked to build an HTTP server in JavaScript from scratch. The group with GitHub Copilot finished 55.8% faster than the control group.

−19%. METR ran a randomized controlled trial with 16 experienced open-source developers working 246 real issues on repositories they had contributed to for years, projects averaging 22,000+ stars and over a million lines of code. With AI tools allowed, they took 19% longer.

Both are real experiments with control groups. Neither is a vendor benchmark. And people quote whichever one supports the position they already hold.

They're measuring different jobs

Look at what each task actually required.

The Copilot study is a greenfield task: one self-contained program, written from zero, by developers recruited for the experiment. No existing codebase to understand. No conventions to match. No reviewer. No correctness bar beyond "it works." That is the situation where a model that produces plausible, conventional code is closest to being purely additive, because there is almost nothing to be wrong about.

METR's task is the opposite on every one of those axes. Mature repository, high standards, real review, contributors who already hold the architecture in their heads. Here the model's output has to be checked against context the developer already has and the model doesn't, and checking is the expensive part. When you already know the answer, reading a plausible suggestion and deciding whether it's right can cost more than writing it yourself.

So the studies aren't in conflict. They're two points on the same axis, and the axis is roughly: how much context does the task require that the model doesn't have?

At the greenfield end, AI is a large accelerant. At the mature-codebase-you-know-well end, it can be a net tax. Most real work sits somewhere between, which is exactly why "does AI make developers faster" has no general answer and every team asking it should be measuring their own case.

The finding that should actually worry you

Bury the headline numbers for a moment, because METR's most uncomfortable result isn't the slowdown.

Those developers predicted a 24% speedup before the trial. Afterwards, having done the work, they estimated they'd been sped up by 20%.

They were measured at 19% slower.

That's a ~39-point gap between experienced engineers' felt experience and their measured output, on their own codebases, after the fact. Not a prediction error. A perception error that survived contact with the outcome.

I find this the single most useful result in the AI productivity literature, because it invalidates the evidence most teams are actually running on. "The team says it's helping" is the standard basis for these decisions. This is a controlled measurement of exactly that signal being wrong, in the optimistic direction, by a wide margin.

It also rhymes with something from a completely different study. When Bai et al. asked frontier models to estimate their own token consumption before running a task, letting them explore the repository first, the correlation with actual usage ran between 0.04 and 0.39, and the models systematically underestimated.

Humans misjudge their own throughput. Models misjudge their own cost. In both cases the introspective estimate is confidently wrong in the cheap-and-fast direction, and in both cases the fix is the same and unglamorous: instrument it.

What to take from this

Cite both, or cite neither. Anyone quoting 55.8% without METR, or 19% without the greenfield context, is selling something. The honest version of the claim has a scope condition attached: for this kind of task, on this kind of codebase.

Check the dates. The Copilot study is from February 2023 and describes original-Copilot autocomplete, not agents, three-plus years of tooling ago. METR's ran in early 2025 with Cursor and frontier Claude models. Neither describes what shipped this year, and both should be read as directional rather than current. (METR has since published a revision to its experiment design, which is what a serious research group does and is worth more trust than a result that never gets revisited.)

Note who ran them. The Copilot study's authors include GitHub and Microsoft employees. That doesn't make 55.8% wrong, it's a published, controlled experiment, but it belongs in the sentence when you're weighing evidence.

Measure your own case. Both studies are strong and neither one is about your codebase. The transferable finding is not a percentage. It's that the gap between how fast this feels and how fast it is turns out to be large, measurable, and pointed the wrong way.

Sources