A project can become attached to a quality measure before anyone has agreed what the measure proves. In our 2024 review, two annotators independently described one hundred selected sporting sequences. They agreed on ninety-two. The proposed project account called that ninety-two-percent accuracy. It was an attractive figure, easy to communicate and apparently strong enough to support a wider release. A domain specialist asked why agreement between two descriptions was being treated as evidence that either description was correct. The question changed the review because both people could apply the same ambiguous instruction and arrive at the same wrong answer. Consistency was present; correctness had not yet been established.
Agreement answers a narrower question
I value agreement measures. They can reveal whether instructions are understood consistently and where reviewers interpret material differently. The problem was not collecting the figure. It was naming it as something more conclusive than the method supported. Two independent annotation attempts can still share assumptions, training or an unclear definition. Their matching outputs do not provide an external reference merely because two people produced them. A data leader needs to understand what the comparison is between before treating its result as a statement about the underlying sporting sequence. The measure's usefulness depends on keeping that question attached to it.
The delivery sponsor wanted to proceed and examine the eight disagreements later. That seemed efficient: focus scarce specialist attention where the annotators had already signalled difficulty. I asked for examination of the agreed group too. The specialist review found fourteen shared mistakes within those ninety-two descriptions. Those were not disagreements waiting to be resolved; they were cases in which the same interpretation had been reproduced. If we had treated agreement as the acceptance condition, the comfortable-looking group would have carried a problem into the release while the review concentrated on the smaller and more visibly uncertain set.
I withdrew the accuracy claim and supported ₹49 lakh of work on definitions, source review and revised evaluation. The cost and slower release discussion were material consequences. We did not describe the review as proof that the whole service had a particular error rate. The one hundred sequences were a selected set, and the findings needed to remain connected to that selection. Nor did we simply subtract fourteen from ninety-two and publish a new universal accuracy percentage. The work showed that the evaluation design had missed a class of shared mistake and needed a more defensible reference for assessing the descriptions.
Review the comfortable group as well
The practical change was to examine why the shared mistakes occurred. Domain practitioners could point to ambiguity in the instructions and examples that made the intended distinction clearer. The data team could then work with those definitions rather than ask annotators to become more careful under the same unclear conditions. A request for greater care would have been easier to issue, but it would not necessarily have changed the interpretation producing the error. The review needed to connect the finding to the work process and the meaning of the label, while retaining the source evidence used to assess the sporting sequence.
This experience made me more cautious about selecting review material only from obvious exceptions. Difficult cases deserve attention, but a process can be consistently wrong in a way that does not produce disagreement. The apparently settled group may therefore contain the evidence most likely to challenge the evaluation method. That does not mean every record requires the same level of manual inspection. It means the sampling and review design should be capable of finding the failure the current measure would otherwise miss. The question is not simply where the system admits uncertainty, but where it may be confident for the wrong reason.
I also wanted to preserve the contribution of the people doing the annotation. The response should not turn a weak definition into a story about careless individuals. They had worked independently within the process supplied to them. When both applied the same interpretation, that was evidence about the instructions and evaluation as well as the outputs. Accountability still mattered: we needed to understand what happened and improve the work. But assigning blame without examining the shared condition would have made it harder for the team to surface similar problems later. A data function needs people willing to challenge a favourable metric without fearing that the challenge will be treated as a personal failure.
Let the practitioner challenge the release story
The specialist's question arrived after the ninety-two-percent figure had begun to carry commercial momentum. That made the correction more difficult than it needed to be. I would now bring domain review into the evaluation design before an attractive result becomes the release story. The practitioner should help establish what constitutes a correct description and which examples would expose ambiguity. The data team should then explain how the chosen measure relates to that reference. Neither group should have to rescue the meaning after the result has already been labelled for management. The order of the conversation affects how readily the enterprise can accept an uncomfortable finding.
At board level, the lesson is not that directors should inspect sporting sequences or choose annotation rules. It is that a quality claim should have an intelligible basis. Management ought to be able to say whether a reported measure concerns agreement, completeness, observed error or another property, and how it relates to the service promise. Those properties can inform one another, but they are not interchangeable. A high number should not close the discussion if the enterprise has not established the link between the measure and the conclusion. Directors can ask for that link without becoming the operational quality team.
The review left us with a less flattering account and a clearer task. We retained the original agreement figure, the fourteen shared mistakes and the cost of changing the evaluation. I did not claim that the correction had already improved every annotation or generated a financial return. The useful change was that the service could no longer call consistency accuracy without supporting that interpretation. For me, leadership in data includes protecting the space in which a practitioner can explain why a good-looking number is answering the wrong question. Two matching answers can be a useful observation. They become trustworthy evidence only when we understand what they actually establish.