Our build machines are production systems

For most of my career, the computers that compiled our software were treated as a convenience. They sat under desks, then in a corner of a data centre, then in the cloud. Engineers configured them by hand when a build broke, and the people who looked after them were rarely the people who worried about security. Our production servers, the ones customers could see, received the firewalls, the access reviews and the on-call rotas. The build machines received whatever attention was left.

I now think that order of attention is backwards for any company whose software other companies ship. If an attacker gains control of a production server, the damage is serious but bounded: we can see the server, isolate it and restore it. If an attacker gains control of the build, every customer who installs the next release installs the attacker's work, signed with our name. The compromise travels outward through channels designed to be trusted.

Thousands of studios inherit our build

At Sternhollow the reach of a release is unusually wide. About 3,100 studios build games with our engine, and the runtime we compile is embedded in every game they ship. A malicious change in that runtime would not stop at our customers. It would reach the players of their games, on consoles, phones and personal computers, in numbers no studio could quickly count. Our editor, which runs on developers' own workstations with broad access to their source code and credentials, is a second channel of almost equal sensitivity.

The industry has had enough warnings. Attacks on build and update systems over the past decade, and the backdoor discovered in 2024 in a widely used compression library after a contributor spent years earning its maintainer's trust, showed that capable attackers aim at the places where software is assembled. In June 2024 our own red team found a long-lived access token, issued to a contractor who had left a year earlier, that could still publish packages to our internal mirror. There was no evidence it had been used. It should not have existed. We revoked it within the hour and then asked an outside firm to examine every credential that could touch our build and release systems. The firm found eleven more that should have been withdrawn. None had been misused, but each was a door we had stopped noticing, and the review changed how we issue access: credentials for build systems now expire after ninety days by default, belong to one named person or one automated job, and are reviewed each quarter by someone outside the team that uses them.

Signatures before scanners

When boards ask about software security, they are usually shown counts from vulnerability scanners: how many issues were found, how many fixed, how many remain. I report those numbers to our Audit and Risk Committee, but I have asked the committee to give them little weight. Scanner counts rise when we add a scanner and fall when we tune it. They say nothing about whether the software a customer receives is the software we intended to ship.

The controls I ask directors to examine are different. First, isolation: every build at Sternhollow now runs on a short-lived machine created for that job and destroyed afterwards, so that nothing an attacker plants can persist into the next build. Second, pinning: every dependency is fetched from our own mirror and checked against a cryptographic hash recorded when it was approved, so a changed package fails the build instead of entering it. Third, review: changes to build configuration need two approvers, one of them outside the team making the change, because build scripts are code with more power than most of our product code. Fourth, signing: release keys live in hardware security modules from which no individual can export them, and signing a release requires two people.

The fifth control is the one I am proudest of and the one that took longest. Our runtime binaries are now reproducible. Two independent build environments, administered by different teams, compile the same source, and we sign the release only if the outputs match byte for byte. If an attacker compromised one environment, the mismatch would stop the release. Reaching that point took fourteen months, because compilers and tools embed timestamps, file paths and other incidental details that had to be removed one at a time. Our editor is not yet reproducible, and I tell the committee so each quarter.

Customers receive a software bill of materials with every release, listing each component, its version and its licence. That practice began after a licence problem in 2022 rather than a security one, but it serves both purposes: a studio that hears of a compromised library can check within minutes whether our release contains it.

What the controls cost us

None of this was free. The programme cost about C$3.4 million over two years, most of it in the time of eleven engineers. For one quarter in 2024 we slowed editor releases from weekly to fortnightly while builds moved onto short-lived machines, and studios noticed. Some engineers resented the second-approver rule for build changes, which can turn a five-minute fix into a half-day wait. Pinning dependencies by hash means that upgrading one now requires a deliberate decision rather than happening by default, and that slows us when an upstream project issues a security fix we want quickly. We built a fast path for those cases, with its own approval, and it has been used nineteen times.

There is also a cost that is easy to miss: these controls make some problems more visible. When an approval chain stops a release, a deadline slips and somebody has to explain why to a customer. A company that wants its controls to work must accept that they will occasionally be inconvenient in public.

For directors, the practical questions are short. Can anyone change what our build produces without a second person seeing it? Would we know if the software a customer received differed from the source we reviewed? Who can sign a release, and could one person do it alone? How long would it take to tell every customer whether a newly compromised component is in our product? A company that can answer those questions has made its build a production system. A company that answers with a scanner count has not yet started.