Chief Technology Officer / Canada
Decisions taken inside projects we could not see
- Company revenue, 2025
- C$402M
- Engine subscriptions, revenue share on shipped titles and hosted online services
- Engineers and specialists led
- 840
- Across Vancouver, Montréal and Victoria, through seven direct reports
- Sternhollow Online availability, 2025
- 99.96%
- Up from 99.71 per cent in the first quarter of 2023
Decisions & consequences
01Telling every customer that our runtime carried a licence we could not honour
In August 2022 our new licence scanner found copyleft code compiled into a runtime that 412 shipped games carried. It was a mesh-simplification library, brought in from a research prototype in 2020, that was statically linked into the runtime module studios ship with their games. The library was offered either under a strong copyleft licence or, for a fee, under a commercial one. Nobody had bought the commercial licence, and because studios ship closed-source games, the copyleft terms were ones neither they nor we could meet.
The exposure was mostly our customers'. Console and mobile platform holders expect studios to account for the licences of what they submit, and several publishers' contracts with us contained representations about third-party code. Our general counsel and our chief revenue officer proposed replacing the library quietly in the next scheduled release, four months away, because its authors had never enforced their licence. I disagreed. A studio that later found the library in its own scan would learn two things at once: that we had made a mistake, and that we had known.
I recommended notifying every customer within two weeks, asking the library's authors for a retroactive commercial licence covering every version already shipped, and replacing the library in all supported engine versions. Our chief executive accepted the recommendation once our general counsel agreed that a retroactive licence would itself need disclosure. The authors were reasonable. We paid C$520,000 for a licence covering past distribution, and fourteen engineers spent eight weeks writing a replacement that we shipped as patches to every supported version, including the oldest long-term branch.
Two studios had to resubmit builds to a console platform holder, and one delayed a content update by three weeks. One publisher asked for a licence warranty in its renewal, which we gave. I know of no customer we lost over it. Runtime dependencies have since fallen from 143 to 88, each with a named owner, and every release ships with its bill of materials.
I would change two things. Our first letter said too little because we were still investigating, and a second letter nine days later had to correct it; one complete letter would have cost a week and saved trust. And I would have required licence scanning before any prototype could be promoted into the runtime, which is now our rule.
Freezing online feature work for six weeks after a launch-weekend outage
Matchmaking on Sternhollow Online failed for six hours and forty minutes on the Saturday of a customer's biggest launch, in March 2023. The customer, Fallowdeep Interactive, had released a cooperative game that drew 1.1 million players in two days. A Friday change to the matchmaking service's scaling rules limited how quickly new capacity could start. When queues lengthened, game clients built with our development kit retried at once and repeatedly, because its retry logic had no backoff, and the retries kept the service down long after the original fault was corrected.
The stakes went beyond one customer. Availability for the quarter fell to 99.71 per cent against a 99.9 per cent commitment, and three other publishers planned launches on the service within six months. I proposed stopping all new online features for six weeks so that our 140 online engineers could work on resilience: a corrected development kit with backoff and jitter, load shedding, staged configuration changes and launch rehearsals run against each customer's own client code. I also proposed publishing the full incident review to every online customer, with Fallowdeep's consent, and paying service credits well beyond the contract minimum.
Our chief revenue officer opposed the freeze. A cross-platform lobby feature had been promised to a large publisher in a negotiation worth about C$7 million, and six weeks without feature work meant missing its date. Our chief executive initially sided with him. I argued that a second launch failure would cost more than one delayed deal, and the executive team accepted the freeze.
The publisher deferred, then signed a quarter later for a narrower scope worth C$4.2 million. We paid C$1.3 million in credits against a contractual minimum of C$420,000. The next three launches on the service ran cleanly, including one that peaked at 640,000 concurrent players. Availability reached 99.96 per cent across 2025, pages per on-call shift fell from 5.8 to 1.6, and Fallowdeep renewed for three years. Error budgets agreed with product leaders now decide when feature releases pause.
I would have made the corrected development kit a mandatory update rather than a recommended one: six months later, 40 per cent of live titles still used the old retry logic. And I would have asked our chief revenue officer to help design the error budgets, not presented them to him finished.
Buying GPU servers for the floor of our demand and renting the peaks
By late 2023 we were renting more GPU capacity by the hour than our subscriptions could support. Our hosted lighting-bake and asset-processing service, together with model training for our animation tools, had taken cloud GPU spending from C$6.1 million in 2021 to an annualised C$19.4 million. Studios near a milestone sometimes waited nine hours for a bake because our cloud region was short of capacity, and the cost was taking four points off subscription gross margin.
Three options reached the executive team. Our chief financial officer favoured a three-year committed cloud reservation of about C$41 million, at roughly a third off, with no asset on our balance sheet. Our vice-president of infrastructure preferred to stay on demand and negotiate. I proposed buying servers for the part of our demand that never went away, housing them in a Québec colocation facility with low-carbon power, and renting everything above that floor. Twenty-six weeks of hourly data by workload put the floor at about 60 per cent of median demand.
The objections were serious. Our infrastructure team had not run hardware at scale since 2014 and would need nine new specialists. The director appointed by Pelloway Growth Partners questioned putting C$19.5 million of capital into equipment that a new generation of chips could make look expensive within two years. In December 2023 the board approved the purchase on condition that I report utilisation and overflow every quarter.
The 128 servers went live in May 2024, two months late because network switches arrived after the servers. A scheduler fills our own fleet first and overflows to the cloud. In November 2024 a firmware fault took 30 per cent of the fleet offline for three days and cost C$310,000 in extra cloud hours. Through 2025 the owned fleet ran at 74 per cent utilisation. Cost per GPU-hour, including depreciation, power, colocation and staff, fell from C$3.10 to C$1.45, and GPU spending was C$14.2 million against C$27 million projected on demand. Waits before milestones fell from nine hours to forty minutes at the ninety-fifth percentile.
The Pelloway director's warning has partly come true: newer cloud instances have narrowed the gap. I would buy in two tranches nine months apart, depreciate over three years rather than four, and hire the data-centre team first.
Declining a clean break in the engine's core while studios were mid-production
Our chief architect proposed in February 2024 that version 6 of the engine should break compatibility with everything built on version 5. The engine's object model, which represents every entity in a game world, dated from 2005 and limited how well the engine used modern processors. He and about thirty senior engineers proposed a new data-oriented core, written cleanly over twenty-six months. Their case was strong: compatibility roughly doubled the work, and the old model caused a large share of our performance complaints.
Our customer data pointed the other way. Sixty-one per cent of active paying projects were on version 5, with release dates between 2025 and 2028. A studio cannot change engine generations in mid-production without risking its release, and many would have stayed on version 5 indefinitely, or left.
In April 2024 I decided against the clean break. We are building the new core behind a compatibility layer and moving the engine to it one subsystem at a time, with projects able to opt in, and we committed to a supported long-term branch of version 5 for five years. The costs were explicit in my paper. The programme is planned at thirty-eight months rather than twenty-six, the supported branch occupies fifty-two engineers at about C$9 million a year, and the compatibility layer means we capture about 70 per cent of the performance gain the clean design showed in benchmarks. Our chief financial officer challenged the branch's cost. The chief architect felt his design had been compromised, and in 2025 he left to found his own company with four of our engineers.
Telemetry from studios that opted in showed which of our 1,900 public interfaces were actually called: 340 accounted for 95 per cent of use, and we deprecated 610 that nobody used. By September 2026 three of the five subsystems had moved, 38 per cent of active projects had opted in, and processor frame time in our reference scenes was 31 per cent lower. Gross subscription retention was 94 per cent in 2025, against 91 per cent in 2023, though I would not attribute all of that to this decision.
I would publish the dated deprecation plan on the first day rather than six months in. And I would have given the chief architect ownership of the new core instead of asking him to review a compromise.