A model that can screen component images quickly is an attractive prospect for a manufacturer. It may direct attention to material worth examining and help people work with a volume of images that would otherwise be difficult to review. Those possibilities deserve investigation. They do not make a test score equivalent to a decision about a product. As CIO, I want the proposal to explain the intended use of the signal, the errors observed and the operating responsibility that remains. The speed of producing an answer matters only in relation to the decision someone proposes to make from it.
Our surface-image pilot used a held-out pool of 18,000 labelled images. The model flagged 432. Of these, 360 corresponded to the agreed defect labels and seventy-two did not. A further twenty-eight labelled defect images were missed, and 17,540 images were correctly left unflagged. Reporting the full account changed the discussion. Colleagues could see the additional review work created by false flags and the limitation represented by misses. A single aggregate measure would have been dominated by the many non-defect images and would have made those different consequences harder to examine.
Read both kinds of error
A false flag and a miss are not interchangeable simply because both are errors in a table. A false flag may require inspection effort and interrupt a workflow. A miss matters differently depending on the defect, the process and the other controls in place. The people responsible for quality need to assess those consequences. My team can provide the technical result and help investigate its conditions, but it should not assign an operating acceptability merely by choosing a convenient metric. The evaluation needs the judgement of the function that understands what the proposed use would mean.
I ask for the underlying counts because percentages can become detached from the material examined. In this case, the pool contained 388 labelled defect images in total: the 360 found and the twenty-eight missed. That is a different denominator from all 18,000 images. We did not need to turn every count into a promotional score. We needed to understand which questions each count could answer. The same care applies to the seventy-two false flags: their burden depends on the proposed review process, not only on their proportion within the whole image pool.
Labels also deserve examination. They are an account against which the model is judged, produced through a defined process and with possible limitations of their own. I want to know how they were established and whether the difficult cases received the relevant review. A held-out pool can help keep the evaluation separate from development, but the phrase alone does not establish that the material represents every operating condition. The enterprise needs an intelligible description of how the pool was selected, what it covers and which conditions still require examination before a wider claim is justified.
Move from an image pool to a working process
The factory adds conditions that may not be present in a controlled exercise. Lighting can change, component families can vary and the position of an item may affect the image. The operating sequence also determines when a signal is useful and who can act on it. I therefore ask the sponsors to describe the proposed process before extending the technical conclusion. Will the signal direct an inspector's attention? Will it hold an item for review? What happens when the model cannot produce a dependable result? Those are different uses with different demands, and they should not be hidden behind the general word automation.
For our JPY110 million pilot, I supported a decision-support role with human quality authority retained. The controlled image results did not support automatic product release. That was a boundary for the next stage, not a declaration that the model had no value. We could investigate whether the signal helped reviewers and under which conditions, while keeping the existing decision responsibility clear. A bounded next step can be worthwhile when it is designed to learn something specific. It becomes misleading when the public account implies that the ultimate operating benefit has already been demonstrated.
The review burden belongs in the proposal. If seventy-two images are flagged unnecessarily in the examined pool, somebody must assess what a comparable workload would mean in the intended process. We should not assume that image-processing speed translates directly into available labour. People perform several tasks, and the timing of review may matter more than its average duration. A staffing claim would need evidence about the actual work and the decision to use any released capacity. We made no such claim from this pilot. The first question was whether the signal could support a useful and controlled review process.
I also resist using the test to imply a production escape rate. The twenty-eight misses occurred within a labelled evaluation pool. They are important evidence about the model under those conditions, but they are not a count of defective products released by the factory. The surrounding process and other examinations matter. Conversely, the absence of a reported production event would not make the misses irrelevant. Keeping those propositions distinct allows a serious discussion of the limitation without exaggerating either success or harm. It helps colleagues decide what additional work is actually needed.
Keep authority attached to the decision
The CIO can fund the evaluation, maintain the technical environment and ensure the results are recorded honestly. Product disposition belongs with the authorised quality organisation. That boundary should be evident in the design of the process, not merely in a disclaimer at the bottom of a presentation. If the user interface, operating instructions or performance incentives cause people to treat the model as the decision maker, a formal statement that a human remains responsible may not describe the real workflow. The proposed use needs examination with the people who will actually perform it.
Continuing ownership is another part of the decision. Models and production conditions change. A useful proposal identifies who monitors performance, who investigates unexpected behaviour and who can restrict use when the evidence no longer supports it. It also preserves enough of the evaluation record for later teams to understand why the original scope was chosen. Those arrangements require resources. They should appear in the investment account alongside the attractive demonstration, rather than emerge as an unplanned obligation after the first deployment has been celebrated.
I want manufacturing technology to benefit from useful new tools without making the enterprise less precise about its own decisions. The complete error account gave our pilot a more credible route forward than a single favourable score would have done. It allowed quality colleagues, operators and technology specialists to discuss the same evidence from their different responsibilities. That is the standard I apply to an inspection proposal: the result should make the next decision clearer, including what it does not yet support, rather than ask an impressive number to carry more authority than the test has earned.