Our AI Coding Numbers Were Perfect. We Were Shipping Slower.
A field story about the thing every AI dashboard is measuring — and the thing it's quietly hiding.
Names and the company are invented. Every number in here is real, and sourced at the end. If your AI adoption charts are green and your release dates keep slipping, this is written for you.
The email
It arrived at 6:47 a.m. from our CEO, subject line just a question mark, with a screenshot attached.
The screenshot was our own AI-adoption dashboard. Ninety-four percent of engineers active on the coding assistant. Suggestions accepted: 38%, up nine points. AI-generated code: 61% of everything new we wrote — a number our board had applauded three weeks earlier.
Under it, one line: “Then why did Meridian slip again?”
Meridian was our flagship release. It had now slipped twice. I’d run engineering at this company for four years, and for the first time I couldn’t answer a schedule question. That was the part that scared me. Not the slip — the not knowing why.
Six months earlier we’d done exactly what everyone in our industry was doing. Bought licenses for the whole team. Ran the enablement sessions. Made the tools mandatory, and put AI usage into the performance-review template. I signed off on all of it. The logic was airtight on a slide: if engineers write code five times faster, we ship five times faster.
Every dashboard said the mandate was working. Every calendar said we were slower. Both could not be true, and yet there they were.
What the dashboards couldn’t see
Here’s the thing about that adoption dashboard: read it closely and you’ll notice it measures one activity. Writing code. Acceptance rate, lines generated, percent-of-code-AI-authored — all of it is a speedometer bolted to a single stage of a long assembly line.
So I stopped reading dashboards and went to watch the actual line.
I started with Priya, the most senior engineer on our platform team — the person whose name is in the history of every file that matters. I found her doing code review. Her whole calendar was code review. Morning, afternoon, evening.
“Show me what you built this month,” I said.
She laughed, not warmly. “I haven’t built anything in six weeks. I review. The pull requests never stop, and they’re huge now — four hundred lines where it used to be eighty. And they’re...” she hunted for the word, “...almost right. Which is worse than wrong. Wrong I catch in five minutes. Almost-right takes me an hour to find the two lines that’ll take production down at 2 a.m.”
“Can’t you push the authors to tighten them up first?”
She looked at me. “The authors are hitting their AI-usage targets, Maya.”
Down the hall was Arjun — two years out of school, one of our “most productive” engineers by every number we tracked. He walked me through a service he’d shipped. Genuinely fast, genuinely impressive work. Then I pointed at a retry block and asked why the backoff was set up that way.
He looked at the code like it was a stranger’s suitcase. “The assistant wrote that part. It passed the tests.”
I asked what would happen if the downstream service returned a partial response. The pause that followed told me more than any dashboard had in six months. He was shipping code he couldn’t explain. We’d made him fast and, without meaning to, kept him from getting good — and not one of my metrics could tell the difference between the two.
Security was the same story with scarier words. Findings per release: up. A cross-site-scripting hole had slipped two review gates deep. And nobody could tell me which code was AI-written and which was human — so nobody could tell me where to look harder.
That night I wrote down what I’d seen, and it did not read like a technology problem:
Pull requests bigger and more frequent; review queues growing without end
My best engineers doing almost nothing but review
Junior engineers producing volume they couldn’t explain
Bugs and security findings climbing
Everyone individually faster — everything collectively slower
I stared at it for a long time. It didn’t look like a software problem at all. It looked like a traffic jam. Every car on the road had gotten faster, and the commute had gotten longer.
The question a factory guy asked me
That weekend my uncle was over for lunch. He’d spent thirty years running manufacturing plants — textiles, then car parts — and he’s retired now and dangerous at a dinner table. He asked how work was going, and because I was too tired to be polite, I told him the whole thing.
He listened all the way through, which is not like him. Then he asked one question:
“You bought a very fast machine for one station on your line. Was that station the slow one?”
“It’s not a line, it’s software,” I said.
“Humor me. When something shipped late — before all this — was it late because people couldn’t type the code fast enough?”
I started to say yes and couldn’t. When had a project ever been late because of typing speed? Projects were late because requirements shifted. Because reviews sat for days. Because things broke at integration. Because we built the wrong thing and only found out in production.
“No,” I said slowly. “Writing the code was almost never the slow part.”
He nodded like a man watching someone finally read the instructions. “Then here is what you did. Every line has one slowest step that sets the pace for the whole thing — doesn’t matter how fast the other steps run, the finish line is decided by the slowest one. Speed up any other step and you don’t ship faster. You just pile up a mountain of half-finished work in front of the slow step. We did this in 1989 with a stamping robot. Local numbers looked spectacular. Actual output of the plant? Nothing. All that robot gave us was a bigger pile of parts waiting at the one oven that could only go so fast.”
He let that sit. Then the part that actually stung:
“Your dashboards. Are they measuring the whole line? Or are they measuring the machine you just bought?”
I drove home doing the translation in my head and hating how cleanly it fit. Our fast new machine: the coding assistant. Our mountain of half-finished parts: unreviewed pull requests. The one oven everything has to pass through, no matter how fast the rest runs: human judgment — Priya, reading four-hundred-line almost-right diffs at nine at night.
And our usage mandate? It was an instruction to run the new machine flat-out, all the time, no matter what the slow step could absorb. We hadn’t just failed to help the bottleneck. We’d ordered the entire company to bury it deeper.
It wasn’t just us — the data was everywhere
I couldn’t sleep, so I went looking for evidence that I was wrong. I found the opposite, and it’s all public. I’d just never assembled it, because each piece gets reported as a separate problem.
A controlled study by METR put experienced developers on codebases they knew well and measured them with and without AI. They were 19% slower with the AI — while estimating afterward that it had made them 20% faster. A thirty-nine-point gap between what they felt and what happened. That’s the mirage, measured in a lab.
Google’s DORA program — the largest ongoing study of software delivery there is — found pull requests 154% larger and code review taking 91% longer in AI-heavy teams, and delivery stability dropping roughly 7% for every 25% bump in AI adoption. Their headline finding in 2025: AI is an amplifier. It makes a well-run system faster and a badly-run one more chaotic. It doesn’t fix your line. It multiplies whatever your line already is.
GitClear, looking at 211 million changed lines of code, found refactoring down about 40%, duplicated blocks up eightfold, and rework climbing — code copied and pasted instead of cleaned up. That’s rust on the pile.
A study of open-source projects after Copilot arrived found the productivity gains came mostly from newer contributors, while the core maintainers wrote 19% less of their own code and did 6.5% more review. That’s Priya’s calendar, showing up across thousands of projects.
An Anthropic experiment found engineers who leaned on AI to get work done scored 17 points lower on understanding and debugging the very code they’d just produced. That’s Arjun’s pause, quantified.
And a widely-cited MIT report found that around 95% of company AI initiatives showed no measurable bottom-line impact — not because the models were weak, but because organizations bolted them onto workflows they never changed.
Five symptoms. One cause. Somewhere around 1:40 a.m. I wrote the sentence I’d bring to the CEO in the morning:
We didn’t buy a productivity tool. We bought a machine that manufactures unfinished work — and we aimed it straight at our scarcest resource.
The fix, in five unglamorous moves
Our CEO read my one-pager twice. “You’re telling me the problem isn’t the AI.”
“The problem is where the AI is. All of its horsepower is aimed at the one step that was never our bottleneck. Here’s the plan. It’s not clever. It just points the effort at the right place.”
One: find the actual bottleneck — measure the line, not the machine. We built a new dashboard in a week. Five numbers, none of them “usage”: how long the review queue is, how long work waits for review, how our senior engineers spend their hours, time from open to merged, and how often a change breaks something. Each split by AI-assisted or not. The picture was instant and ugly. Work was arriving at review three times faster than it was leaving. Our bottleneck finally had a face, and the face was exhausted.
Two: stop wasting the bottleneck on work a machine could do. A good factory puts inspection before the slow, expensive step, so the oven never wastes heat on a part that’s already scrap. We did the same to review. Static analysis, security scans, complexity checks, tests, and an AI first-pass review now all run before a human sees the PR. Anything a machine can catch, a machine catches. And we routed by risk — routine changes go through a fast lane, and only the genuinely risky stuff reaches Priya. Her job title finally matched her calendar again.
Three: pace the fast machine to the slow step. This was the fight. We capped how big a pull request could be. We capped how many could be open at once. Generation now runs at the speed review can actually absorb — you don’t let the fast machine bury the slow one. And we pulled AI-usage out of the performance review entirely.
The CEO pushed back hard on that last one. “The board loves that number.”
“The board loves a number that goes up while we get worse,“ I said. “It’s the machine’s speedometer. Let me give them the one for the whole line.”
Four: now — and only now — spend money to widen the bottleneck. More AI-assisted review to widen the funnel. A real program to turn our juniors into strong reviewers — which had a bonus I hadn’t fully seen coming: reviewing is how you build the understanding the assistant had been quietly eroding. Arjun reviews now. He asks the assistant why more than he asks it what. Last week he rejected a generated diff with one line that made me stupidly proud: “This duplicates our retry policy. Extend it instead.”
Five: assume today’s fix becomes tomorrow’s problem. Bottlenecks move. Every cap and gate we installed has an expiry attached to it — “revisit when review wait drops below a day” — so no rule outlives the problem it was built for. That mandate taught me the lesson the expensive way: yesterday’s sensible policy, still running against today’s reality, is exactly how smart companies do dumb things.
What was hiding behind the bottleneck
Here’s what nobody warns you about: fix the bottleneck and the fog clears, and you see the next one. Three things were waiting behind review.
We were quietly bloating. With code nearly free to produce, our codebase was gaining weight — over-cautious, over-engineered, the same logic written five ways, modules that looked tidy in the file tree and were secretly welded together underneath. So we put the codebase on a budget: automated checks in the pipeline that won’t let new code worsen the complexity or duplication trend. And the move that mattered most — we gave the assistant our own code to work from. It now searches our repo, our design docs, our existing components before it writes, and its standing order changed from “write me a retry helper” to “find our retry helper and extend it.” Cheap code, it turns out, is like cheap calories. The bill shows up later, in the joints.
Security moved to the front. Instead of only scanning code after it existed, we started constraining it as it was written: our security rules and known failure patterns live in a file the assistant reads on every request, and it reviews its own output before showing us — which the research says is the single most effective trick available for getting safer code out of these models. Every flaw that slips through becomes a new rule. The rulebook ratchets tighter over time instead of resetting every sprint.
And the big one. Two months in, the CEO asked the question the whole exercise had been walking toward: “If review isn’t the bottleneck anymore — what is?”
I’d been waiting for it. “Deciding what to build. We’ve gotten frighteningly good at shipping. Nobody’s checking whether the thing was worth shipping.” So we moved that decision to the front of the line: nothing enters the build without a clear, falsifiable bet attached — what outcome we expect, how we’ll measure it, and when we’ll kill it if it’s wrong. And here, finally, the AI earned its keep at the right station: a working prototype in front of real users in two days, before we commit the whole line to building it. We killed four planned features in the first month. The old me would have shipped all four, on time, with a beautiful dashboard.
Ninety days later
Meridian shipped. Not heroically — boringly, which is better. Review wait is down two-thirds. Change-failure rate is back below where it was before the mandate. And adoption, for what it’s worth, is higher than it was under the mandate. Turns out engineers don’t resist good tools. They resist being graded on the machine’s speedometer while drowning in its output.
The board got a new favorite number: time from a validated idea to production. It’s the only metric I’ve ever put in front of them that you can’t fake by making the pile bigger.
There’s an index card on my monitor now. It’s as close as I have to a philosophy:
The goal is validated value shipped — not code produced.
Find your slowest step before you buy speed. It is almost never typing.
Time saved anywhere except the bottleneck is an illusion. A dashboard full of illusions is just an illusion with a budget.
Never let your scarcest human judgment do a machine’s job — and never let a machine do your judgment’s job.
Unfinished work is a liability, not an asset. Code nobody understands is unfinished even after it ships.
When the bottleneck moves, your best policy becomes your next problem.
My uncle, when I told him all of it, went quiet for a second. Then: “We learned this with steel, forty years ago. Every generation buys a shiny new machine and rediscovers the oven.” He topped up my chai. “You only lost six months. Some people lose the whole plant.”
Three phone calls: why your version won’t look like mine
Word gets around. In the months after Meridian shipped, two friends called with the same question — our AI numbers are green and nothing’s faster, what did you do? — and I learned something that nearly sent me back to the start: my answer didn’t transfer. Because we weren’t actually in the same business.
The first call was Ravi, who runs delivery at a services firm — his engineers build software for other companies, on contract. I walked him through the bottleneck hunt and he stopped me cold. “My bottleneck isn’t my review queue. It’s my client’s sign-off queue. And Maya—” he dropped his voice like a confession, “—when the AI saves my team a hundred hours on a bill-by-the-hour contract, I don’t book a win. I book a hundred fewer billable hours.”
Sit with that. In his business, the whole thing inverts. An hour saved isn’t fake progress — it’s real lost revenue. His slow step isn’t a queue at all. It’s how he prices the work. And you can’t tool your way out of a pricing model; you can only change it. So his plan looked nothing like mine: move contracts away from bill-by-the-hour toward fixed-price and outcomes, so speed becomes profit instead of self-harm; measure the client’s wait, because their sign-off is his oven; and rebuild his junior ranks around reviewing and understanding — because AI was eating exactly the work his juniors used to learn on, and this year’s juniors are next year’s seniors. One more thing I told him, as someone who buys services: all that code his teams generate fast, and don’t fully understand, walks out the door to his clients at handover. The good firms will start guaranteeing they can stand behind it — before their clients learn to demand it.
The second call was Elena, who runs technology at a bank. Her problem was stranger: coding was clearly 40% faster and nothing else had moved at all. We drew her pipeline on a whiteboard — intake, business case, annual funding, an architecture board, a build split across her own teams and three vendors, a security review, a change-approval board, a quarterly release window. Her engineers’ new speed was parked several steps upstream of a bottleneck made of committees. And between her teams and her vendors sat something my map didn’t even have — a seam, a contract boundary where work piled up invisibly, because neither side could see the other’s backlog. Her fixes: turn compliance and security checks into automated code so approvals run at machine speed instead of meeting speed; invest in test environments, which turned out to be her literal oven; and rewrite vendor contracts around flow — wait-time commitments, code you can stand behind — so the discipline reaches across the company line. Her hardest discovery was her oldest blind spot: nobody had ever gone back to check whether a funded project actually delivered what it promised. Her real bottleneck — deciding what to fund — had been binding for years. She’d just never measured it.
Three companies. Three different answers. One method underneath. The steps are always the same: figure out what you actually get paid for, find your slowest step, protect it, pace everything to it, widen it, and keep looking because it moves. But the first question — “what do we actually get paid for?” — has a different answer depending on whether you own the code, rent out the coders, or inherit everyone’s homework. Start anywhere else, and you’ll spend a fortune tuning a machine in someone else’s factory.
Before you go
If any scene in this felt a little too familiar — the green dashboard sitting on top of the red dates, the senior engineer who’s quietly become a full-time reviewer, the AI mandate everyone complies with and no one believes — that discomfort is a signal. It usually means your bottleneck has moved and nobody’s gone looking for it yet.
Finding it is what I do. Whether you’re just starting down the AI-assisted development road and want to skip the six months this story cost, or you’re deep into a rollout where the numbers stubbornly refuse to add up, I’d like to hear where the work is actually piling up in your pipeline. Product company, services firm, or enterprise — the diagnosis starts the same way, even though the answer won’t.
Reach me at pw@prima-partners.com. Tell me what your dashboard says and what your gut says — especially when the two disagree. That gap is usually exactly where we’ll find the thing that’s slowing you down.
If this was useful, subscribe, and send it to whoever owns the AI mandate at your company. There’s a good chance they’re staring at a green dashboard right now, wondering the same thing our CEO was at 6:47 in the morning.
The company and people here are composites, but the research is real.
Sources:
METR’s randomized study of experienced developers (−19% actual vs. +20% perceived speed).
Google’s DORA reports, 2024 and 2025 (pull requests +154%, review time +91%, stability −7.2% per +25% adoption, “AI as amplifier”).
GitClear’s 2025 code-quality analysis of 211M changed lines (refactoring ~−40%, 8× duplication).
Xu et al., 2025, on open-source contribution patterns (core maintainers −19% own code, +6.5% review).
Anthropic’s study on AI assistance and skill formation (−17-point comprehension gap). MIT’s 2025 report on enterprise AI adoption (~95% without measurable impact).
Veracode’s security research (~45% of unguided AI-generated code fails security checks).
Full links available on request.