We evaluated a component-surface inspection model in November 2025 using a held-out pool of 18,000 labelled images. The model flagged 432 images. Review against the agreed labels identified 360 correctly flagged defects and seventy-two false flags. It also found twenty-eight labelled defect images that the model had missed, leaving 17,540 correctly unflagged images. These were results from a controlled image pool, not an estimate of defects escaping from an unreviewed production process.
The proposal initially emphasised the speed with which the model could screen images. I asked for the complete error account and the intended operating decision before supporting further use. False flags created review work, while misses raised a different question about reliance on the signal. A single overall accuracy number would have concealed much of that distinction because the pool contained many more non-defect images than defect images. We also needed to understand how the selected material related to changing lighting, component families and production conditions.
I supported the JPY110 million pilot as an aid to human inspection, with product disposition remaining under the quality organisation's authority. The alternative of automatic release was not supported by this exercise. We retained the test material and labels, documented the error categories and asked the sponsors to define a further evaluation covering the proposed use. We did not infer a staffing reduction from faster image processing or claim that the system detected every kind of safety-relevant defect. The model's purpose was narrower than that.
The pilot gave the enterprise a concrete basis for deciding what to investigate next. It showed useful signals and limitations in the same account, rather than asking colleagues to choose between enthusiasm for a new tool and rejection of it. My role was to make that account fit the decision. Quality colleagues determined whether and how the signal could assist their work, while my team maintained the technical evaluation and the resources for a bounded next step.
I would agree the consequences of false flags and misses with quality colleagues before selecting the model, ensuring the evaluation answers an operating question rather than rewarding one attractive aggregate.