A large test set can make a programme appear well examined. The number is easy to report, compare and carry into a release paper. Yet it says little about why the cases belong in the exercise or what work they represent. In 2024 a proposal relied on 650 production-derived fixture cases. The team valued their resemblance to actual operations and wanted to preserve the count. I wanted two questions answered separately: whether the material could appropriately be used, and whether the tests demonstrated the requirements that mattered.
Those questions can become confused because real operational material feels persuasive. It contains complexity that a designer may struggle to invent. But resemblance to reality does not settle the conditions attached to using it in another environment. Nor does a large collection of actual cases necessarily exercise the situations most important to a change. A familiar set can be convenient precisely because it avoids unusual paths that the new service needs to handle. Volume and authenticity are different from justified coverage.
Data owners and role specialists helped examine the proposal. We replaced the original set with 420 synthetic illustrative cases and 230 restricted metadata-only cases reviewed by the data owners. The latter were not described as anonymous or unrestricted. A business relationship or structure can remain sensitive even when direct personal details have been removed. The public story contains no actual fixture content or policyholder information; its purpose is to explain the management decision, not reproduce the material whose use was under review.
Resemblance does not settle permission
The first discipline was to explain why each kind of material was needed and under what conditions it could be used. A programme should not acquire permission merely because its work is internal or because testing has a legitimate purpose. The relevant owners need to assess the material and the intended environment. Technology can implement the agreed restrictions, but it should not invent a legal or professional conclusion from the fact that an engineer can technically copy a record into a test system.
The 230 metadata-only cases retained access conditions. That was a deliberate part of the account, not an awkward exception to a claim that the set had been sanitised completely. Metadata can describe relationships, timing or business structure that require care in their own right. The appropriate treatment depends on the actual circumstances and qualified assessment. Our fictional decision does not certify a method as compliant or establish that removing one category of detail makes every remaining item safe for unrestricted use.
The work cost ₹86 lakh and extended the release plan by three weeks. Those consequences belonged in the decision. Reusing the existing fixtures would have been easier in the short term, and the programme had already organised work around them. Changing the set required design, review and adjustments to the exercise. I did not present the response as a costless improvement that everyone should have welcomed immediately. The tradeoff was to spend effort establishing a more defensible basis for the evidence before relying on it for release.
Synthetic does not automatically mean representative
The 420 synthetic cases answered part of the material-handling question by using invented circumstances. They still needed to exercise meaningful work. A collection of tidy examples can pass easily while omitting the combinations, missing details or unusual sequences that make the real task difficult. Role owners therefore examined requirement coverage rather than accept the replacement cases merely because they were synthetic. The source of the material and the usefulness of the test remained separate judgements throughout the review.
This matters especially when a programme tries to preserve its earlier count. Four hundred and twenty plus 230 equals 650, but that arithmetic is not evidence that the revised set provides the same or better coverage. It tells the reader how the set is constituted. The programme still needs to explain which requirements the cases exercise and where the evidence is incomplete. A count can organise that explanation; it cannot replace it. I wanted the release discussion to understand the composition rather than celebrate the restored total.
The two groups also needed accurate labels. Calling the entire set synthetic would have obscured the restricted metadata cases. Calling it all production-derived would have obscured the invented material and its design work. A short management summary should retain the distinction when it affects use and interpretation. That is the same discipline I apply to a recovery test: the label should match the thing demonstrated, rather than acquire a broader meaning because the larger statement sounds more reassuring.
Explain the evidence rather than defend the count
The result was not a claim of perfect privacy, universal legal compliance or complete testing of every possible service condition. Qualified owners retained their responsibilities, and the programme retained limits on what the exercise established. Those boundaries do not make the work unsuccessful. They make the evidence useful. A decision-maker can ask for further work where the scope is insufficient, whereas a sweeping success label may conceal the very condition that needs attention before release.
There is a practical benefit to explaining the cases in terms of requirements. When a requirement changes, the team can ask which evidence needs revision instead of assuming the old set remains adequate because it is extensive. When a data-use condition changes, it can identify the affected material rather than treating the whole collection as one undifferentiated asset. The leadership contribution is to make those relationships understandable enough to maintain. A test set is part of an evolving service, not a permanent certificate obtained by reaching a large number once.
For a board or executive sponsor, the useful enquiry is what management's test evidence represents. Directors do not need to inspect every fixture or choose the technical method. They can ask how the programme established permitted use, how business owners judged coverage and what remains outside the result. Those questions should be answerable without exposing restricted material in the board paper. The account needs enough specificity to support judgement while preserving the same information responsibilities the testing process is meant to respect.
The three-week extension was easier to defend once we stopped treating the argument as a choice between useful real data and artificial examples. That framing concealed the different purposes involved. The revised approach used invented cases where appropriate and retained restricted metadata only under reviewed conditions, then asked whether the combined set exercised the work. It did not assume that either category was inherently sufficient. The decision became a question about justified evidence rather than loyalty to the original collection.
I now ask for the explanation behind a test count before accepting the number as a readiness signal. What does the set contain, why may it be used and which requirement does each part help demonstrate? The 650 cases remained a useful way to describe the population, but they no longer carried the whole argument. The leadership lesson is to separate permission from representation and both from volume. A more thoughtful account of the evidence can justify a release decision more effectively than a larger figure whose meaning nobody has examined.