The 2016 ticket-classifier test contained 10000 examples: 9600 routine tickets and 400 severe tickets. The model correctly classified 9400 routine tickets and 200 severe tickets. That produced 9600 correct classifications overall, or 96% accuracy. I presented the aggregate as evidence that the model was ready to support automatic severe-incident routing. Clara Fenwicke challenged the conclusion by asking how many of the severe tickets had been found. The answer was 200 out of 400, giving 50% severe recall. The headline was arithmetically correct but inadequate for the decision I had attached to it.
I had allowed the much larger routine class to dominate the account. Clara's objection was not that overall accuracy had no meaning. It was that the proposed routing use depended on detecting the severe cases, and half of those had been missed in the test. My first presentation did not make that consequence visible enough. Adding the recall figure beneath the same readiness statement would not repair the interpretation. I needed to withdraw the statement and reconsider what evidence the use required, rather than defend the aggregate because it remained technically true.
We withheld automatic routing and supported US$650000 review. The revised account included the class counts and correct classifications so that a recipient could follow both calculations. Delivery and operational owners retained responsibility for any later routing proposal, with model-validation expertise separately involved. I did not claim that the review produced a later performance result or that an improved model was eventually deployed. The documented consequence was the withdrawn readiness claim, the withheld use and the additional work needed to assess the issue properly.
The decision delayed a capability I had advocated and required me to explain why my original account had been too broad. I accepted that cost. The case changed how I review a model summary: the measure must be connected to the consequence the proposed use cares about. A strong aggregate may coexist with a weak result in a smaller but important class. The public narrative retains that failure because it makes my later judgement assessable. It is a fictional example, not a universal acceptance threshold for classifiers, and it does not replace specialist evaluation of any real system's intended use.