A captioning programme can report that a text track is present on the material it examined. That is a useful technical fact. It does not establish that a person can follow the programme through the captions. A changed name, number or qualification can alter meaning. Text displayed at an awkward point can become difficult to connect with the speaker or event. The engineering account needs to distinguish those problems before deciding what a coverage figure means.
My argument is that caption evaluation should begin with the task of understanding the material, and that people who rely on captions should help shape the examination. This does not mean replacing specialist judgement with a vote on every output. It means recognising that a technically available feature can still fail the purpose for which it was built. The user perspective should influence the question before release is chosen, rather than appear only as a final endorsement of the work.
The distinction can be uncomfortable for an automation sponsor. Producing a track at scale is a substantial achievement and may remove a difficult bottleneck. It is reasonable to want that achievement recognised. The next question is whether the resulting service is ready for the uses being proposed. Recognition of the engineering progress and a careful release boundary can coexist. Neither requires pretending that the other consideration is unimportant.
Name the problem before counting it
In August 2025, Vensalith reviewed 120 selected clips from live material. Caption specialists and editors worked with twelve deaf or hard-of-hearing reviewers. Twenty-seven clips contained a meaning-changing caption error. Eighteen showed a display-timing issue. Nine belonged to both groups, leaving thirty-six unique affected clips. The overlap mattered because adding the two category counts would have implied forty-five separate affected clips and misdescribed the work required.
Those figures described an investigation set, not a prevalence estimate for all programmes. The clips had been selected to help us understand the proposed approach. A useful result did not need to represent every programme statistically, provided we retained that limit. It could reveal specific issues, direct a repair and help determine which uses required continuing review. The problem would arise if the selected findings were presented as a service-wide rate without the supporting design.
The categories also helped different specialists contribute. A meaning error might require attention to recognition, terminology or the editorial handling of a particular utterance. A timing issue could involve the way text was prepared, transported or displayed. Some clips needed both kinds of work. Describing every problem as caption quality would have obscured those differences and made the repair discussion less precise.
I wanted the team to preserve examples that made the consequence understandable. A count can show how much work was found, but it may not show why a changed word matters. Editors and reviewers could explain the effect on the programme without turning every error into the same severity. That allowed the sponsor to understand why selected formats should retain human review even while the technical approach continued to improve.
Let users influence the release question
The reviewers' involvement was valuable because they examined the feature as part of following the material. Their contribution was not a claim that twelve people represented every viewer with a hearing difference. It gave the team direct evidence about difficulties that an output-presence check could miss. We treated that evidence alongside caption expertise and engineering investigation, with the purpose of improving the decision rather than creating a symbolic consultation.
The automation sponsor preferred a broad release with continued tuning. Caption and content colleagues wanted human review retained for selected formats where meaning changes could have material consequences. I supported the narrower route and funded a timing repair. The ten-week programme cost CAD530,000. That decision accepted ongoing review effort and did not book a general saving from removing it. The operating account had to reflect the service we were actually prepared to recommend.
There is an important distinction between describing that decision and offering a universal rule for caption production. Different material, audiences and live conditions may require different arrangements. The fictional case does not establish a regulatory threshold or a preferred technical method. It shows how a sponsor can respond when the evaluation reveals that the proposed use is wider than the evidence currently supports.
The decision also protected the team's ability to learn. Once a broad automation claim has been made, unresolved cases can become awkward facts that nobody wants to emphasise. A narrower release with an explicit investigation plan makes those cases legitimate work. Engineers can examine them, content specialists can explain their significance and the sponsor can decide whether the remaining benefit justifies further effort.
Keep improvement separate from completion
After the changes, we repeated the thirty-six affected clips. Seven still had a meaning issue and four a timing issue, with two in both categories. Nine unique clips remained unresolved. The reduction was useful evidence about the examined set. It did not demonstrate that every other programme would be satisfactory, and it did not justify a claim of complete accessibility. The remaining exceptions stayed in the review process.
A repeat on known cases also has a particular purpose. It shows whether the identified problems have been addressed under the repeated conditions. It should not be confused with an independent examination of all future material. Further evaluation may be needed before expanding the use. Keeping those stages distinct makes it easier to understand what has improved and what question remains for the next decision.
I would now involve caption users and editorial colleagues when the first review categories and clips are selected. That would make the initial engineering report more useful and reduce the chance that coverage becomes the assumed definition of success. It would also help the sponsor budget for the specialist and human work that remains necessary around the automated output, rather than discover that work only after a release expectation has been set.
The broader leadership responsibility is to keep the purpose of the feature present in the investment discussion. Captions exist to help people understand a programme. Producing them is an essential stage, but their presence is not the full account. A strong recommendation describes meaning, timing, the people and material examined, and the unresolved work. That gives progress its proper value while keeping the decision grounded in the audience experience it is meant to serve.