Set outage tolerances by the fifteenth and the thirtieth, not by monthly uptime

For four years I presented our board with a monthly availability figure for the bank's critical systems. It rarely fell below 99.9%, and it was accurate. It was also the wrong measure. My position now is that a bank should set its service tolerances for the journeys customers rely on, such as paying by QR code, sending or receiving an instant transfer and drawing cash, should measure them on the days when demand peaks, and should treat its change calendar as a control that protects those days. In the Philippines, those days are the fifteenth and the end of each month, when most private employers pay wages, and the weeks before Christmas.

Why 99.94% hid our worst afternoon

On Friday 15 December 2023, a payday in the busiest month of our year, QR payments and instant transfers from our app began failing at 11:50 and did not fully recover until 15:30. Load on our payments middleware reached 2.6 times the daily average, a connection pool ran dry, and the app's automatic retries multiplied the very traffic that had caused the problem. About 410,000 payment attempts failed. Some 19,000 customers saw money leave their accounts for payments that never completed. We reversed every one within two days, but a reversal on Monday does not buy groceries on Friday. We received 27,000 complaints, notified the Bangko Sentral ng Pilipinas that afternoon and sent a written account the following week.

Our availability report for December still showed 99.94%. The core banking system never stopped, the app stayed online and the failure was intermittent, so our monitoring recorded most of the afternoon as degraded rather than down. Averaged across a month of quiet nights and ordinary Tuesdays, three hours and forty minutes of partial failure barely moved the figure. Nobody had designed the report to mislead. It had been designed to show the health of machines, and the board had accepted it as a description of customers.

I should have seen this earlier. Two smaller payday failures earlier in 2023 had produced the same surge in complaints alongside the same reassuring report. I had read them as bad luck.

Journeys, not systems

Since early 2024 we have measured nine customer journeys instead of system uptime: QR payment, sending an instant transfer, receiving one, card authorisation, cash withdrawal, app login, bill payment, payroll credit and loan repayment. Each journey has a success rate, a time within which it should complete and an impact tolerance, meaning the longest disruption and the number of affected customers that the board risk oversight committee is prepared to accept. The tolerances are tighter on paydays, because the same failure harms more people and harms them more.

This approach is not original. Supervisors in several countries have pushed banks towards operational resilience framed around important services and the harm a disruption would cause, and I would not claim our version is the most sophisticated. What made the difference for us was narrower: putting the worst payday of each quarter in front of the committee, beside the average, on the first page. Averages describe a bank's systems. The worst day describes its customers' experience of them.

The change was uncomfortable. In the first quarter of the new reports, three of the nine journeys breached their payday tolerances. A director asked whether service had suddenly deteriorated. It had not; we had simply stopped averaging it away. Since then, payday QR success has risen from 99.2% across the paydays of 2023 to 99.86% in 2025, and we have had no payday incident lasting more than thirty minutes, against four in 2023.

The calendar is a control

Measurement alone would not have produced that improvement. We bought capacity for three and a half times average load and test it every month against a synthetic payday, at a cost of about ₱640 million in 2024. Our chief financial officer questioned that sum, reasonably, because the capacity sits idle for most of the month. I argued that idle capacity on the fourteenth is the price of working capacity on the fifteenth, and that the alternative was paying for failure in complaints, reversals and supervisory attention.

The more contested decision was the change freeze. Non-urgent production changes are now barred for two business days either side of the fifteenth and of month-end, which removes about eight working days a month from the release calendar. Our head of digital banking objected that it would slow every product team, and it did: production releases fell from twenty-two a month to seventeen in early 2024. We recovered to twenty-one in 2025 by making releases smaller and more frequent in the open weeks, which was better engineering practice in any case. I would make the same trade again. A release that ships two days late rarely hurts a customer; a release that fails on payday can hurt hundreds of thousands.

We also publish a status page and send an SMS notice within fifteen minutes of a confirmed disruption. Our communications colleagues worried that this would draw attention to failures. In practice it reduced complaints, because customers who know a payment has failed stop retrying, and fewer retries make recovery faster.

Some things I have not solved. Nine journeys were too many to introduce at once, and teams that owned two or three of them struggled to improve any. Our merchant view came late: a market stall owner whose QR payments fail on payday loses sales before we see a complaint, and we did not add merchant acceptance to the measures until 2025. The tolerances themselves remain judgements. I cannot prove that ninety minutes is the right limit for card authorisations on a payday; I can only show the committee what happened when we exceeded it.

My argument to other banks is practical. Ask for the worst day of the quarter, not the month's average. Define tolerances by what customers could not do and for how long. Then look at the change calendar and ask whether it protects the days when customers most need the bank, or treats every day as if it were the same.