A prediction must be tested with the past it could have known

## Put the decision moment before the score

The repair record looked like a useful source because it was complete. It contained the component episode, the work that followed and the eventual closure account. Our analytical extract brought those fields together, and the validation routine divided 1,200 labelled episodes into 900 development records and 300 test records. The reported test score was 91%. In 2022 I endorsed the result as support for a predictive-maintenance proposal. I had asked whether the test records were separate. I had not asked whether the model's inputs belonged to the moment at which a warning would have been useful.

Ritu Nezvethi challenged the repair-closure fields. They were completed after the event we intended to predict. Their availability in a retrospective extract did not make them available before that event. The model had been given part of the answer through the later history. Separating episodes did not remove that knowledge. The result might describe performance on the assembled retrospective task, but that was not the task the operating proposal claimed to solve. The distinction was material even if the calculations had been performed exactly as specified.

A prediction has an intended moment of use. Someone needs to know something while there is still a decision to make. The data specification should state what exists at that point, how quickly it arrives and which later revisions are unavailable. A field called a status can be particularly misleading because its current value may conceal a sequence of changes. A completed work order can include facts that were unknown when the order opened. Reading the final record backward is not the same as reconstructing the earlier evidence.

I do not think an executive needs to reproduce every modelling step to ask this question. The executive does need an account of the task that can be understood without relying on the apparent sophistication of the method. What is being predicted? At what time? For which population? Which inputs would the user actually possess? What event later establishes the outcome? Those questions determine whether the score has a relevant meaning before a conversation about its size begins.

The mistake was mine because I approved the interpretation. It would have been easy to describe it as a technical preparation problem discovered by engineering. That description would have concealed why the problem reached the investment discussion. I had treated a recognisable validation arrangement as sufficient evidence that the practical question was sound. The review needed to examine the relationship between the records and the proposed decision, not merely confirm that a familiar analytical procedure had been followed.

## A revised dataset requires a revised claim

The programme sponsor argued that the model could lose the obvious closure fields during implementation. That might have been a reasonable next development step. It was not a reason to preserve the existing 91% claim. Once the inputs change, the evaluation concerns a different task. A favourable result obtained with later knowledge cannot be carried across as though removing that knowledge were a minor presentation edit. The useful idea in the work deserved another examination; the existing performance claim did not deserve protection from the consequences of the correction.

We authorised ₹67 lakh of reconstruction and held the ₹2.8 crore rollout proposal. The reconstruction specified the intended prediction cutoff and traced when each proposed input could have been available. Where timing remained uncertain, the uncertainty stayed in the account. We did not replace the withdrawn score with an invented lower number to make the story feel complete. Nor did we declare the entire approach worthless. The supported conclusion was narrower: this evaluation did not justify the operating claim attached to it.

That distinction matters when explaining the cost. The ₹67 lakh was spent on work that had to be done. The ₹2.8 crore was a proposal held back, not cash already paid and then recovered. Calling the decision a saving would have implied an equivalent approved service had been avoided without consequence. In fact, the company still had an unmet analytical ambition and further work before it could decide whether a credible implementation was available. A corrected claim does not make the earlier effort free.

Other forms of separation also needed examination. Repeated records from the same component, closely related episodes or shared site conditions can make a test less independent than its row count suggests. A later evaluation must be designed for the actual intended use, with technical specialists deciding the appropriate grouping and timing. I would not prescribe one universal split as a remedy for every maintenance dataset. The relevant question is whether the assessment asks the model to perform the future task under conditions it could actually encounter.

Engineering and safety colleagues retained authority over physical decisions. A valid prediction study would still leave questions about what action an alert should trigger, how errors would be handled and whether service capacity could support the response. Correcting leakage is necessary for an honest claim; it is not a complete operating case. I wanted the programme to return with those boundaries visible, so that improvement in one part of the evidence did not silently become permission for a much wider intervention.

## Design the next evaluation around actual use

For the next evaluation I would begin with the person receiving the proposed warning. What decision is available to that person, and how much time remains to make it? A signal that arrives after the service crew has already acted may still be useful for analysis, but it should not be presented as a warning that enabled the action. Likewise, a prediction with no feasible response may be analytically interesting without supporting the proposed commercial outcome. The practical setting should shape the evaluation before the dataset is assembled.

The evidence record would distinguish original observations from subsequent updates. It would preserve the declared cutoff, the source and timing of candidate inputs, the outcome window and the population selected for assessment. These details are not an invitation to expose confidential customer material on an executive website. They are the underlying discipline that makes a public case intelligible. A leader can explain the judgement and its limits using authorised, appropriately generalised detail without publishing the raw maintenance record.

I would also agree how the team will describe an inconclusive result. A programme should be allowed to discover that its present evidence is insufficient without manufacturing a success metric for the next presentation. That does not remove accountability for expense or delivery. It makes the learning question explicit: what was established, what remained unknown and what commitment is now justified? In our case the immediate output was a corrected task definition and a held rollout, not a proven reduction in downtime.

For directors, the oversight question is whether management's performance statement describes the proposed use or an easier retrospective exercise. They should expect an explanation of relevant limitations and a credible route to examining them. They need not take over feature engineering or instruct technicians. Their contribution is to ensure that an investment recommendation does not borrow confidence from a score whose conditions have been omitted. Management remains responsible for designing, testing and operating the eventual service.

The strongest lesson I took from the episode was about sequence. We had put the impressive percentage before the question of what could have been known. Once that order was reversed, the decision became clearer, even though it became less comfortable. A predictive claim earns its place by surviving the conditions of its intended use. The future cannot be included in the inputs and then celebrated as something the model successfully anticipated. I want that simple proposition to remain visible whenever a polished performance result reaches my desk.