A high accuracy score can conceal a difficult alert queue

## Read the counts before celebrating the percentage

The candidate's overall accuracy was 94.6%. That was the number proposed for the opening slide of a 2025 maintenance-alert review. It was not an arithmetical error. The problem was what the audience might infer from it before seeing the underlying counts. The evaluation contained 10,000 nonoverlapping component assessment episodes with a defined future observation window. Only one hundred contained the specified event. Most episodes did not, so correctly leaving many of them unflagged could dominate an overall percentage without explaining how useful the alerts would be.

The candidate produced six hundred alerts. Eighty corresponded to confirmed events and 520 did not. Among the 9,400 episodes without an alert, twenty contained events and 9,380 did not. Those figures reconcile to the original population and to the 94.6% score. They also describe two practical problems that the percentage alone concealed: a substantial queue of false alerts and a set of missed events. Service colleagues would have to understand both before deciding what kind of response the output could support.

Alert precision was eighty out of six hundred, approximately 13.3%. Recall was eighty out of one hundred, or 80%. I wanted those rates shown with their counts, definitions and limits. They were findings for the reviewed population, not universal attributes of the candidate. A different mix of equipment, event prevalence or observation conditions could change the result. The evaluation also depended on the quality of outcome confirmation. A neat classification table cannot compensate for an ambiguous definition of what counts as an event.

One comparison made the weakness of accuracy particularly clear. An always-no-alert rule would have been correct for the 9,900 episodes without an event, yielding 99% accuracy while missing all one hundred events. That would not make it a useful warning system. The example did not prove that the candidate was ready either. It showed why the release decision needed a more relevant account than choosing the largest overall percentage. A metric can be correctly calculated and still be a poor summary of the practical question.

The sponsor wanted a simple message because the programme had already consumed effort and colleagues needed a decision. I understood the pressure. The answer was to make the decision clearer, not to make its evidence smaller. A board or executive committee should be able to see what the proposed system would ask people to do and what it would fail to reveal. The counts gave the conversation a concrete starting point that the attractive headline had postponed.

## Connect evaluation to a feasible response

A false alert is not automatically harmless. Someone may spend time investigating it, move a crew, interrupt another task or become less attentive to later warnings. The size of those consequences depends on the intended workflow and the equipment involved. I would not translate 520 false alerts directly into a rupee loss without that context. I would ask the service owners what response was proposed and what capacity it required. The data function can describe the queue; it cannot decide that every investigation has the same cost or importance.

The twenty missed events also require more than a rate. Their consequences, detectability through existing arrangements and relationship to the proposed use matter. A warning tool supplementing an established inspection process raises different questions from an automated system replacing part of it. In our case, the candidate did not replace existing safety or maintenance arrangements. We kept it in a bounded shadow study while qualified owners examined the operating implications. That distinction should remain explicit whenever the study is described publicly or internally.

We spent ₹56 lakh on evaluation and review. Engineering and service colleagues considered follow-up capacity, event definitions and the design of a prospective assessment. I supported that work without presenting a universal acceptable threshold for missed events or false alerts. Such a threshold would need the actual operating context and the authority of the people accountable for its consequences. A CDO should resist both overstating the model and borrowing safety authority from the fact that its outputs are numerical.

The observation unit mattered too. We used nonoverlapping assessment episodes with a specified future window, rather than silently treating every repeated daily record as an independent opportunity. If several records concern the same underlying event, a large row count can exaggerate the apparent amount of evidence. The study needs to state how episodes were constructed and how related histories were handled. A count of rows is not automatically a count of independent decisions, and an executive summary should not invite that assumption.

I would bring service colleagues into the evaluation design earlier in a future programme. They can help explain which distinctions affect the response before analysts choose the presentation. That does not mean fitting a metric to whichever conclusion an operating team prefers. It means making the intended use and its constraints visible before a convenient score becomes the de facto decision rule. Agreement on the question is a stronger starting point than an argument over the winning percentage after the experiment is complete.

## Keep the shadow study distinct from an operating result

A shadow study can produce useful evidence without changing physical operations. Its outputs can be compared with later observations and reviewed by authorised specialists while existing procedures remain in force. That is what made the next step proportionate in this case. It allowed the company to investigate the candidate further without pretending that the retrospective score had already justified an operating release. The study's value lay in answering a more precise question, not in creating an immediate success story for the programme.

We did not claim that eighty correctly flagged episodes represented eighty failures prevented. A prediction and an effective intervention are different events. Preventing a failure would require a feasible response, timely action and evidence about what would otherwise have occurred. None of that follows merely from agreement between an alert and a later outcome. The same caution applies to avoided downtime and financial return. A retrospective classification result does not supply those measures by itself, even when the event being predicted is commercially significant.

Prospective assessment would need to preserve the declared population and observation conditions, record how the output was handled and examine changes over time. I would want the evaluation account to retain uncertainty and any departures from the agreed design. A result can become less representative as equipment, service practices or the mix of customers changes. That is a reason to define ongoing review, not to assume that one successful study establishes permanent performance. The operating owners should know what evidence would cause the programme to be reconsidered.

For directors, the useful challenge is whether management has connected the analytical result to a credible operating proposition. What work does an alert create? Which events remain undetected? What is the proposed role alongside existing arrangements? What would justify expanding the use? These questions do not require directors to tune a classifier. They require management to explain why the chosen evidence supports the chosen commitment and where specialist judgement remains necessary. That is the level at which an oversight discussion can be both informed and properly bounded.

I left the review with a different view of what a strong opening slide should do. It should make the decision intelligible before it makes the programme impressive. In this case, six hundred alerts, 520 false alerts and twenty misses were a better beginning than 94.6% alone. The figures did not settle the release question, but they exposed the question we actually needed to answer. A leadership account becomes more useful when it shows that practical burden clearly, including the work still required before a promising analytical output can earn a place in a real service.