In August 2024, Andy Jassy posted that Amazon’s coding assistant had cut the average time to upgrade an application to Java 17 from about 50 developer days to a few hours. He put the saving at 4,500 developer years. Developers shipped 79% of the generated code reviews without changing a line.
In March 2026, the Financial Times reported that Amazon had called its engineers into a mandatory session after a run of outages, including a six hour failure of its main retail site. The briefing pointed to a “trend of incidents” with a “high blast radius” and to “Gen-AI assisted changes”. Dave Treadwell, a senior vice president, wrote that site availability “has not been good recently”.
Same company. Two ledgers. They never appear on the same page.
The thesis
AI productivity gains and AI delivery risk do not land in the same part of the codebase. The gains concentrate in new, simple code where nothing depends on the change. The exposure concentrates in old, complex code where everything does. The standard AI business case nets the two off as one blended number per engineer, as if they were the same system. They are not, and in the column where most of the money sits, the net can be negative.
Where the gain lands, and where the risk lands
Yegor Denisov-Blanch’s team at Stanford has productivity data on more than 100,000 developers across more than 600 companies. In the 136 teams where they split the work by maturity and complexity, the pattern is blunt. New code, low complexity: 30 to 40% gain. New code, high complexity: 10 to 15%. Existing code, low complexity: 15 to 20%. Existing code, high complexity: 0 to 10%, with some teams slower. They also found that once rework is subtracted, a 30 to 40% jump in apparent output nets to roughly 15 to 20%.
Now look at where the money is. GAO reported in July 2025 that the US federal government spends more than $100 billion a year on IT and that about 80% of it goes to operating and maintaining existing systems. Eleven of the most critical are between 23 and 60 years old. Commercial ratios vary. The direction does not: an established company spends most of its engineering budget changing systems that already make money.
So the gain is largest where the spend is smallest, and smallest where the spend is largest. That alone breaks the blended number.
The risk runs the other way. A new service with no users fails quietly. A bad change to a twenty year old billing path fails in front of customers, and it fails through dependencies nobody wrote down. Old code carries implicit contracts: the field that must never be null because a downstream report divides by it, the batch job that assumes it runs after midnight. No model sees those contracts, because they are not in the repository. They are in the heads of the people who were there.
I have written code for 30 years. In old systems the costly failure is rarely bad logic. It is a correct change to a system whose real behaviour was never written down. That is the failure an assistant is worst placed to avoid, and it sits in exactly the code where it saves the least time.
The blended number is not an average of two outcomes. It is a gain from one system used to approve work in another.
Three readings
Amazon, both columns. The Java upgrade is the clean version of the gain: mechanical change, verified by tests, on code where correct is easy to define. The March incidents are the other column. Amazon disputes that AI caused them, and in the earlier case of a 13 hour outage of AWS Cost Explorer in one China region in December 2025, where its Kiro agent chose to “delete and recreate the environment”, it called the cause user error, specifically misconfigured access controls, not AI. Take that at face value and the point survives. The agent ran with the operator’s permissions on a live system, and no second person had to approve it. The control that failed was a control for old systems with live dependents. The fix Amazon reportedly reached for was senior sign off on AI assisted changes. That is a cost, and it sits in the legacy column only.
METR, the expert on the mature repository. In July 2025 METR ran a randomised trial with 16 experienced developers on 246 real issues in their own projects, repositories averaging more than 22,000 stars and over a million lines of code. Before starting, they expected AI to make them 24% faster. Afterwards, they believed it had made them 20% faster. It made them 19% slower. The people best placed to judge a mature codebase misjudged the direction of the effect. A business case built from self reported time savings is built on that misjudgement.
The pilot. An AI business case usually carries one productivity figure, hours saved per engineer per week, taken from a pilot. Pilots get run where a team can show progress in six weeks, which means new internal tools and greenfield services. The figure is then multiplied across the whole engineering headcount, most of whom spend their week on the old systems. When I read one of these cases for a capital partner, the first thing I ask is which repositories the pilot ran on. If the answer is new ones, the pilot measured the 30 to 40% column and the rollout spends in the 0 to 10% column.
Where this breaks
The strongest objection is the first case. Amazon’s Java upgrade was old code, at scale, and the gain was enormous. That is true, and it forces a sharper version of the thesis. The line does not run between new and old. It runs between change whose correctness a machine can check and change whose correctness lives in someone’s head. A framework upgrade with a strong test suite is old code in the first category. A pricing rule in a legacy billing engine is in the second. Age is the proxy most business cases can measure today. Verifiability is the real variable.
Second, the data dates fast. METR itself now says its 2025 results are out of date, and tools improved sharply over late 2025. Grant it. A better model closes the context gap only on contracts that are written down somewhere it can read. The implicit ones stay implicit.
Third, some companies have very little legacy. A three year old SaaS business is mostly the first column, and for it a blended number is close to honest. The thesis bites hardest on established companies and on anything private equity has bought for its cash flow, which is to say the companies writing the largest AI business cases.
What I would do on Monday
Split the productivity line into two columns: new and existing systems. Use 30 to 40% and 0 to 10% as the opening assumptions until your own data replaces them.
Ask where the pilot ran. If it ran on a greenfield service, its number applies only to greenfield work, and the business case should say so in writing.
Mark every AI assisted change in version control with a commit trailer, so every incident review can count them. Amazon had to ask the question in a meeting. Your post incident template should answer it automatically.
Put a priced risk line in the existing systems column: change fail rate on those repositories before and after rollout, multiplied by your cost per incident hour.
Point the agents at verifiable legacy work on purpose: upgrades, migrations, dependency bumps behind a strong test suite. Keep behavioural change on revenue paths under named senior review, and budget that review time as a cost of the programme.
Two ledgers
Jassy’s post and Treadwell’s email were written about the same engineering organisation, nineteen months apart. One counted hours saved. The other counted hours down. Neither mentioned the other, because they were written for different audiences about different systems.
Your business case is written for one audience. It should carry both.
I read AI business cases from the operator’s side and the investor’s side. If yours carries one blended productivity number, open my calendar.