The AI Productivity Paradox: Why AI Coding Studies Disagree

Three different instruments share one label.

After METR's 2025 trial, 16 experienced developers estimated AI had cut their time by about 20%. Their reported implementation time was 19% longer. Same people, opposite signs. [1]

That gap is the AI productivity paradox: AI feels or looks faster while measured gains in delivered work stay unclear. In the sources I compared, numbers that clash count different things. METR timed real issues [1], and its 2026 update calls its later estimates unreliable. [3] The Conference Board recorded perceived change. [4] Vendor telemetry counts activity. [5] Compare who was measured, what they were given, what was counted, what counted as done and when.

Put that beside two other things published about AI and work. On 29 October 2026 The Conference Board summarized a survey of 3,000 US consumers and reported predominantly positive perceptions of how generative AI has affected their thinking and work outcomes. [4] Separately, one Hacker News thread squeezed the METR contrast into a title: "The gauge broke: devs felt 20% faster with AI, measured 19% slower." [8] That is the thread's wording. METR reported a 19% longer duration, which is not the same thing as a 19% drop in speed. Lined up together, these look like a fight between studies. Laid out slot by slot, they answer different questions, and the disagreement that survives that comparison is worth reading closely.

The next time someone forwards you one of these numbers, you should be able to say what it counted before you repeat it. By the end you will have five checks for that, each tied to a pair of sources it separates.

What METR's 19%-Longer Result Actually Timed

Did METR find that AI makes developers slower? For its own population, tools and window, yes: issue implementation time ran 19% longer with AI allowed. That is a longer duration on review-ready issues, not a general 19% loss of speed, and METR's own page for the study now carries a notice: "These results are out of date." [1] The details below are what keep it from stretching further.

The study was a randomized controlled trial with a narrow, deliberately chosen population: 16 experienced open-source developers working on 246 real issues in mature repositories they already knew. For each issue, AI tools were allowed or disallowed at random. When allowed, developers primarily used a commercial AI code editor running the models available in early 2025. The work ran from February to June 2025. [1]

The unit was the total time developers self-reported spending on each issue, including work done before and after pull request review. They also screen-recorded their work, and METR analyzed a subset of those recordings separately to see how the time was spent. [2] With AI allowed, that time was 19% longer. METR's 2026 update gives a confidence interval running from 2% to 39% longer. [3]

The finish line matters as much as the duration. METR's comparison table defines a finished issue like this:

"Human user is satisfied code will pass review - including style, testing, and documentation requirements." [1]

That is a demanding bar, and it is not the same as shipping to production. Not every pull request in the study had merged by the end. If you want to carry the question into your own team, Kemi Okoro's piece on what counts as finished work picks it up from there.

The 2026 Follow-Up Did Not Flip the Answer

METR's 2026 update is not a clean reversal of the 2025 result, and its authors say so themselves. In February 2026 METR described a larger follow-up that began in August 2025: 57 developers, 143 repositories, more than 800 tasks. The authors write that "the data from our new experiment gives us an unreliable signal" and that "our data is only very weak evidence for the size of this increase." [3]

Their reasons are specific. Developers keenest on AI, and the tasks they most wanted AI for, were less likely to end up in the experiment. Pay dropped from $150 to $50 an hour. Some participants ran several agents at once, which made their time records hard to trust. [3]

The authors do think developers are probably more sped up now than in early 2025, and they present that as their judgment, not as a measurement. [3]

This is where the unit starts to bite. Time per task is a tidy measure when one person works one task at a time. Once work overlaps, the same measure gets harder to read. I am leaving the follow-up's point estimates out of this piece on purpose. Its authors say those numbers cannot carry the weight, so I will not ask them to.

The AI Productivity Paradox Starts With What People Feel

Perceived-benefit surveys and timed-task results can point different ways because they count different things, and in the sources here they also ask different people. Go back to the 2025 trial for a moment. The same 16 developers produced two numbers: an estimate, after the study, that AI had cut their time by about 20%, and reported implementation time that ran 19% longer. [1] The after-the-fact estimate and the per-issue time reports came from the same people and pointed in opposite directions. That alone tells you they are different instruments.

The Conference Board survey measures something else again. Its public summary says: "We asked 3,000 US consumers how often they use generative AI tools and how they perceive this has affected their own thinking and work outcomes compared to last year." [4] The unit is a respondent's perceived change over a year in thinking and work outcomes. There is no timed task and no definition of finished work, and the full methods sit behind a sign-in.

One case asks developers about their own issues; the other asks a broad consumer sample about a year of work. My read is that perception data is real evidence about experience and adoption. It is not a stopwatch. Nothing in these sources gives a basis for ranking surveys above or below timed studies in general.

More Pull Requests Is a Count, Not an Outcome

Can more code or more pull requests mean less productivity? It can, if review time, churn or incidents per unit of work rise faster than the work itself. The public pages I inspected do not show which way any particular organization went.

Vendor telemetry adds a third kind of unit. Faros AI, an engineering analytics vendor, describes its Q2 2026 report as "Two years of telemetry from 22K developers across 4K teams", comparing low and high AI adoption within the same organizations. The units it advertises include pull request (PR) size, median review time, bugs per PR, incidents per PR, and code churn. [5] Those carry different denominators even inside one report. Bugs per PR and median review time are not the same kind of number, and the full report sits behind a work-email form, so I am not quoting its figures.

Here is a made-up example, not data. A team merges twice as many pull requests after adopting an assistant, and each one is half the size. The total change is identical, just sliced thinner. If review time per PR stays flat, reviewers now spend twice as long in total. A dashboard counting PRs shows a doubling. A lead asking whether more useful work shipped has learned nothing yet. The example shows a logical possibility, not a causal finding.

DORA frames AI's role in its 2025 report "as an amplifier, magnifying an organization’s existing strengths and weaknesses." Its generative AI summary also keeps individual self-reported productivity apart from team delivery throughput and stability, which can move differently. [6] Faros, for its part, says: "Our data directly contradicts DORA’s 2025 findings." [5] That is a vendor's claim. Without comparable methods from both sides, I would not treat it as a settled disagreement in either direction.

What Each Source Actually Counted

Here are the six sources side by side, one row each, drawn from references 1 to 7. Where a method is not public, the cell says so rather than guessing.

No two rows share all five slots. The closest pair is the two randomized trials that both time tasks, and that is exactly where the next problem shows up.

Where the Disagreement Is Real

The strongest objection to everything above goes like this: maybe the studies disagree because AI really does help some people on some tasks and hurt others, and no amount of translation will make that go away. That may well be right, and nothing here rules it out. The best case for it is the pair of randomized trials. Google researchers gave 96 of their engineers random access to three internal AI features and timed one assigned enterprise task, from workspace creation to last IDE interaction. They put the reduction in time at "about 21%, although our confidence interval is large." [7] The paper is a preprint. METR, also randomizing and also timing, found the opposite direction. [1]

Now look at what differs between those two rows. Maintainers in their own repositories against engineers on an assigned task. Real backlog issues against one standard task. Review-ready code including post-review work against a task that was done once its tests passed. Internal 2024 features against a commercial early-2025 code editor. Any of those could drive the gap, and so could differences between people inside each study. The public evidence cannot say which. Translation does not dissolve this conflict. It turns "studies disagree" into a question someone could answer: under which conditions does assistance help? Please do not average 21% and 19% into a tidy figure on the way. They are not on the same scale.

To be plain about the limits of this argument: it does not claim AI always helps or always hurts, and it is not a forecast about current models. Adoption telemetry does not identify causes. Self-reports are not worthless. Differences in effect between people and tasks are a plausible hypothesis that this article cannot confirm.

Five Checks Before You Quote a Productivity Number

Before you forward the next AI productivity number, see whether it can answer these five questions. They are my synthesis from the sources above, not a validated instrument.

1. Who was measured? Experienced maintainers in their own repositories and 3,000 US consumers cannot settle a question for each other.
2. What were they given? Random access to AI tools in a trial is a different treatment from high adoption observed in telemetry, as the METR and Faros AI rows show.
3. What was counted? Reported implementation time and perceived time savings moved in opposite directions within the same 16 METR developers.
4. What counted as done? Code ready to pass review and code that passes a fixed set of tests are different finish lines, one of the places METR and the Google trial part ways.
5. When, and with which tools? An early-2025 commercial code editor, internal 2024 features and late-2025 parallel agents are different treatments wearing the same label.

If a number cannot fill all five, it says little about the AI productivity paradox either way. Treat it as a reason to dig, not a conclusion to pass along.

Try it on the last AI productivity number someone posted in your team channel. Work out which slot was missing, tell the person who posted it, and send them this piece.

References