In 2023 a proposed predictive product used 240 fictional specimens from forty material batches, with six specimens per batch. The initial split allocated 180 specimens to development and sixty to assessment. Thirty batches appeared on both sides, with four specimens in development and two in assessment; the remaining ten batches contributed all six specimens to development. The split was arithmetically correct, but the intended commercial use concerned a new batch. A model could therefore be assessed on related observations from batches it had already encountered, answering an easier or simply different question.
The analytical team had invested time in the original workflow. Commercial colleagues wanted to retain the assessment already prepared for a product discussion, arguing that the held-out specimens were still separate physical observations. That was true, but it did not establish that the assessment represented the future decision. Imran asked whether the proposed use could tolerate the difference between a new specimen from familiar conditions and a batch whose shared conditions were absent from development. I supported the need to make the validation unit explicit before promoting the result.
We reassigned complete batches: twenty-six batches, or 156 specimens, for development and fourteen batches, or eighty-four specimens, held apart. The ₹45 lakh validation programme and ₹7 lakh annual review provision funded the redesign and subsequent scrutiny. The revised allocation did not by itself establish model quality, and we did not invent a higher score to justify the delay. Preprocessing and model choices also needed to respect the separation. The qualified analytical team retained responsibility for the eventual assessment and the limits of any approved application.
The commercial consequence was that the earlier assessment could no longer support the proposed launch account. We accepted further work without knowing whether the later result would strengthen or weaken the business case. That is the trade-off I want a board to understand: improving the relevance of evidence can remove confidence rather than add it. The case is not an instruction that every model needs this exact split. It shows why the unit of independence should follow the intended use, with related observations treated honestly, before a convenient metric becomes a product promise.