An accurate answer can arrive too late to be a prediction

The result was attractive enough to shorten my questions. A model had correctly classified 182 of 200 rental contracts in a retrospective test. Ninety-one per cent seemed to give us a strong basis for discussing early renewal action. I approved the result for management before establishing when each input could have been known. A later review found that a renewal invoice, created after the intended prediction date, was among the inputs. The model was answering a question with evidence from its future. My approval had allowed that timing problem to disappear behind the score.

We wanted the prediction thirty days before the rental ended. That timing was not a technical footnote. It defined the business opportunity: what could a colleague know early enough to choose a useful action? A field that exists in the completed contract file may be entirely legitimate for later reporting and still be unavailable for that earlier decision. The invoice did not become a bad record because it was unsuitable for prediction. Its role in this particular evaluation was wrong.

Put the decision on a calendar

I now want the intended decision described before the data extract is praised. Who will use the output, at what point, and to consider which action? Those questions determine which observations are eligible. They also expose ambiguities that a single label such as renewal can hide. Predicting an event before it occurs differs from recognising it after an administrative record has been created. Both may support useful work, but their performance claims cannot be treated as interchangeable.

The strongest argument for retaining our 91% figure was that it showed an underlying relationship worth pursuing. I agreed with the second part. The project might still be valuable. But a promising research direction did not make the published score an honest estimate of early predictive ability. Keeping it as the headline while explaining the timing problem in a note would ask readers to carry two conflicting interpretations. We withdrew the claim and allocated ₹41 lakh to rebuilding the evaluation.

The new work restricted inputs to those available at the intended prediction date and used a separate prospective shadow cohort. It correctly classified 154 of 200 contracts, or 77%. The word shadow matters: we were observing the model's account rather than allowing it automatically to refuse a rental, change a price or determine customer treatment. Commercial and operating owners retained those decisions. The evaluation could then be examined without turning an unproven score into an intervention whose consequences customers would carry.

The two percentages invite a simple story about a fourteen-point decline. I do not use that story. The retrospective and prospective cohorts were different, and the evaluation conditions changed. Subtracting the percentages gives an arithmetic difference, but it does not isolate the effect of removing one input. A senior reader should be able to understand that distinction without needing to reconstruct the experiment from a technical appendix. The correction made the new result more relevant to its intended use, not directly comparable in every respect with the earlier claim.

Ask what the score permits the business to do

Even a properly timed 77% result is not a complete commercial argument. We need to know how a suitable simple alternative would perform, which mistakes the model makes and what those mistakes mean for the proposed action. A correct classification may have little value if it arrives too late or prompts a response the customer does not need. An incorrect one may carry very different consequences depending on whether it triggers a courteous reminder or restricts access to a service.

This is where the data function must stay connected to colleagues outside it. Commercial owners understand the relationship in which an action would occur. Operating colleagues know what the service can support. Finance examines the resources and consequences. Qualified legal and privacy owners assess the proposed use of customer data. The model does not absorb those responsibilities simply because its output appears in the decision. My role is to make the evidence usable for their judgement and to identify where it remains insufficient.

There is a cost to slowing a promising project. A team may have spent months assembling data, and a sponsor may have planned a programme around its expected result. Withdrawing a score can create uncertainty about the whole effort. That concern deserves more than a lecture about technical purity. The response should explain which claim failed, which useful work remains and what the next evaluation can establish. Otherwise an honest correction can be misunderstood as proof that no analytical approach is worth trying.

The opposite risk is to protect the programme by softening the explanation until nobody understands why the claim was withdrawn. In our case, the input's timing was decisive. I had approved a result that could not support the early-decision interpretation attached to it. Naming that error was necessary even though the team could rebuild. The organisation needed to distinguish confidence in continuing the investigation from confidence in a particular performance statement. They were different judgements with different evidence.

Make correction part of the leadership record

A CDO's account should include the point at which personal approval contributed to the problem. It would be easy to describe this as a successful detection of data leakage and leave my earlier decision outside the case. That would make the story cleaner and less useful. The lesson concerns how an attractive result can change the quality of executive questioning. I had allowed the score to determine the pace of the discussion before making the decision setting explicit.

I would now establish the calendar, eligible inputs, comparison and proposed actions when the evaluation is commissioned. I would also decide how a correction should be communicated if later review changes the interpretation. These are not demands for a model that never fails. They are ways to ensure that failure can be understood before the business builds a larger commitment around a claim. The appropriate response may be further research, a narrower use or no deployment at all, depending on what the evidence supports.

For a board, the useful question is not which algorithm produced the highest number. It is whether management can connect its evidence to the decision it proposes and explain the consequences of being wrong. Directors need not redesign the analysis to ask that question. They can require a clear account of timing, alternatives and responsibilities. The model becomes more credible when its limits remain visible, and the data leader becomes more useful when an earlier approval can be examined with the same care as a later correction.