AI writes the code faster; someone still has to read it

Every software company I know has been asked the same question by its board in the past two years: if AI can write code, how many fewer engineers do we need? It is a fair question, and the honest answer at Sternhollow has turned out to be less exciting than the question. The tools help. They help some work much more than other work. And the time they save at the keyboard is partly spent again by the people who read what the tools produced. A board that hears only the first part of that answer will make a staffing decision the evidence does not support.

I should say where I started. I was sceptical of the early claims, many of which measured how often engineers accepted a suggestion or how many lines an assistant produced. Neither measures anything a customer would notice. Lines of code are a cost, not an output, and an accepted suggestion can still be wrong. So when we decided to adopt assistants, I asked for a trial designed to show what reached our customers rather than what happened in an editor window.

What six months of measurement showed

From January to June 2025, 240 of our engineers used AI coding assistants while comparable teams continued without them. We chose the groups by team rather than by individual, because engineers on the same team review each other's work and a mixed team would have blurred the effect. We measured four things: the time from an engineer's first commit on a change to its merge into the main branch, the review effort each change required, the share of changes reverted within thirty days, and defects that reached customers.

The results were uneven in an instructive way. In our editor and tools code, which is mostly user-interface work, data handling and tests written in a managed language, median time to merge fell by about 11 per cent. In the services behind Sternhollow Online the gain was smaller, around 6 per cent. In the engine's performance-critical C++, where a change is judged by what it does to a frame budget measured in fractions of a millisecond, we could not detect any improvement at all. Engineers there told us the suggestions were plausible and often wrong in ways that took longer to find than writing the code would have.

Two other findings mattered more. Engineers using the assistants wrote noticeably more tests, which I had not expected and which I count as a real benefit. And the share of changes reverted within thirty days rose slightly, from 2.0 per cent in the comparison teams to 2.4 per cent among the assisted ones. The difference is small and the sample limited. It was enough to make me look harder at review.

The cost moves to the reviewer

When we examined review effort, the pattern was clear. Changes written with assistance were larger on average, and they drew about a fifth more review comments. The extra reading fell mostly on our senior engineers, who in a typical week spent between four and six more hours reviewing code than they had the year before. Some of that is healthy, because reviewers were catching problems before they shipped. But it means that time saved by a mid-level engineer writing a change was partly transferred to a principal engineer reading it, and principal engineers are the scarcest people we have.

This is the point most adoption plans miss. Software organisations are rarely limited by typing speed. They are limited by the capacity to understand changes well enough to accept them, and that capacity sits with a minority of experienced people. A tool that increases the volume of plausible code without increasing that capacity moves the bottleneck rather than removing it. If the reviewers are overloaded, they either slow everything down, which erases the gain, or they read less carefully, which shows up months later as defects. In an engine that studios ship inside their games, a careless merge does not stay in our building.

There is also a provenance question that matters more to a vendor than to most companies. Code an assistant suggests may closely resemble code published under licences we could not accept in our runtime. We configured our tools to filter suggestions that match public code where the tool supports it, and anything destined for the shipped runtime passes through the same licence and provenance checks as every other contribution. Those checks cost time, and that time belongs in any honest calculation of the gain.

What I tell a board that asks for the saving

We rolled the assistants out to all 620 of our software engineers in September 2025, at a cost of about C$900,000 a year. I believe the spending is justified. Across the whole organisation I estimate the gain at between 5 and 8 per cent of engineering capacity, concentrated in tools and services work and in testing. That is worth having. It is not a reason to reduce headcount, and our board has agreed that no workforce plan will assume a saving until we can see it in delivery rather than in a trial.

We made three changes to protect the gain. We now plan review capacity explicitly, team by team, as part of quarterly planning, and we have trained forty more engineers as approved reviewers for parts of the codebase that only a handful of people could previously approve. We cap the size of changes that can merge without a second reviewer, which counters the tendency of assisted changes to grow. And we report reverted changes and escaped defects by team alongside delivery speed, so that a team that becomes faster and sloppier is visible as both.

When directors ask me what AI has done to engineering productivity, I now give them four numbers rather than one: time to merge, review hours per change, the revert rate and defects that reached customers. If the first improves while the others hold, the gain is real. If the first improves while the others worsen, we have borrowed from our future and called it productivity.

I expect the tools to improve, and I expect our numbers to change with them. I would rather report a modest gain we can defend than a dramatic one that our customers later discover was paid for with their time. The question for a board is not how many engineers AI replaces. It is whether the company has enough people who can read what is being written.