Write the outage notice for the town before the ministry

Every operator I know can write a careful incident report for a regulator. It is precise about timings, causes, the number of subscribers affected and the corrective measures. Very few are as good at telling their own customers, in the first hour, what has happened and when it might end. My position is that a regional network should treat the public notice as an operating task with the same discipline as the regulatory report, issue it within an hour in honest ranges rather than promised times, and give its board the same account it gives the ministry.

Two audiences, one clock

At ten past one on the morning of 9 February 2024, a change to our subscriber provisioning system, made during a planned overnight window, stopped 162,000 of our broadband customers from reconnecting after routine session renewals. About 38,000 of those households also take their fixed telephone service over our network. Service began to return from half past five, and the last households were back shortly after eight. The incident was of a kind that Japan's telecommunications rules require carriers to report, and we reported it to the Ministry of Internal Affairs and Communications in the required form.

Our regulatory report was good. Our customers fared worse. The first notice on our website appeared two hours and forty minutes after the fault began. It said we were investigating a disruption and apologised for the inconvenience, and it gave no area, no cause and no estimate. The community channel carried nothing until morning. Our contact centres opened at nine to a queue of several thousand calls, many from older customers who had found their telephone silent when they tried to reach a relative.

When I reviewed the night, the delay had a simple explanation. The notice needed approval from three people, one of whom was asleep, and the engineers on duty believed, reasonably, that their first job was to fix the fault. Nobody owned the public clock. The regulatory clock had an owner, because we all knew what happens to carriers that report late.

Since then the public notice has an owner on every shift: the duty manager in our network operations centre, who may publish from a set of approved forms without further sign-off. The rule is a first notice within sixty minutes of any outage affecting more than 3,000 customers, on our website, by text message to affected customers who have registered a mobile number, and on the community channel's on-screen ticker for the areas concerned. The first notice must say which areas and services are affected and when the next update will come. It must not guess at a cause.

Ranges rather than promises

The second change concerns estimates. Engineers dislike giving restoration times because they are often wrong, and an operator that announces six o'clock and restores at nine has created a second grievance on top of the first. The usual response is to give no estimate at all. I think that is worse. A customer deciding whether to walk to a neighbour's house to make a call, or a shopkeeper deciding whether to run a card terminal on a mobile connection, needs some basis for the decision.

Our notices now give ranges, with a stated likelihood, and narrow them as we learn more: service expected back between five and nine, most likely before seven, with a further update at half past four. The ranges come from incident history, not from optimism. Duty managers keep a short table of how long similar faults have taken, and they are measured on whether the actual restoration fell inside the range they published, not on how fast it was. Since the change, restoration has fallen inside the first published range in 83% of qualifying incidents.

I learnt the cost of false precision earlier. After the 2016 earthquakes we announced fixed reconnection dates for 31 areas and failed to meet nine of them. Customers in those areas remembered the missed dates longer than the reconnection itself. Ranges would have served them better, and would have served us better too.

The report the board should read the next morning

The third change is internal. After the February outage, our audit and supervisory committee first received a two-page summary written for directors. It was accurate, and gentler than the report we had filed with the ministry. The ministry's version described the change that caused the fault, the review that should have caught it and the fact that the change had been moved into the overnight window at short notice without the usual second review. The summary described a configuration error.

I asked that the committee receive the regulatory report itself, with a covering note, and that this become the rule for every reportable incident. The committee's chair agreed at once. Some colleagues were uneasy, because a regulatory report is written plainly and names process failures that a board summary would soften. That plainness is the point. Directors are responsible for the systems that produce incidents, and they cannot oversee those systems through a filtered account. A report fit for the regulator is fit for the board.

We also changed the process that caused the fault. Changes to provisioning systems now go first to a small share of customers, with an automatic stop if reconnections fail, and no change may move into an overnight window without the second review, whatever the business pressure. Those rules have delayed two product launches by several weeks. Our product team has said so plainly, and the board knows why.

There are fair objections. A notice issued within the hour will sometimes be incomplete, and occasionally it will describe an outage that resolves itself ten minutes later. Text messages at four in the morning annoy some customers, and a few have asked us to stop. We allow them to opt out of overnight messages, and we accept that an early notice will sometimes prove unnecessary. I would rather apologise for an unneeded message than for silence.

None of this is sophisticated. It rests on a judgement that customers and directors deserve the candour operators already give regulators, delivered on a faster clock. Regulatory reporting exists because the public has an interest in dependable networks. An operator that is candid with the ministry and vague with the town has misunderstood whom the ministry is there to protect.