Twelve shops were randomly assigned in a customer-explanation pilot, six to each arm. Across those shops, six hundred adult visitors were observed. The intervention was delivered through the shop setting. A preliminary account was nevertheless being described as though the six hundred visitors had each been independently randomised. The observation count was real within this fictional case, but the interpretation had quietly changed the experiment. We had assigned shops, not six hundred separate individuals.
At Qevzheti Music Rooms, the 2024 programme cost ₹58 lakh. I held the overconfident inference pending qualified cluster-aware review. This account reports no effect size, significance result or final rollout decision. Those omissions are deliberate. The leadership choice concerned the basis on which a conclusion could be stated, not a new statistical result I can supply from a few headline counts. The evidence remained useful, but its confidence needed to be assessed in relation to the design that produced it.
The intervention determines what can be assigned
A customer explanation delivered through a shop may naturally affect a shared setting. Staff, presentation and local conditions can be common to several visitors. Assigning the shop can therefore be a reasonable way to organise a pilot. It would be a mistake to treat group assignment as inherently inferior merely because a different design has a more familiar interpretation. The appropriate choice depends on the intervention and practical setting, with qualified design work before the programme begins.
Our issue came afterwards. The analysis needed to preserve the twelve assigned shops and the observations nested within them. Visitors in the same location might have experiences that are related through the setting they share. Counting each record as though it represented an independent assignment could overstate the strength of the evidence. The actual amount of dependence and its implications required analysis; it was not something I could settle by replacing six hundred with twelve in every calculation.
That distinction matters for executive discussion. A board or sponsor does not need a simplified rule that group experiments never work or that only the number of shops matters. It needs an explanation of which level was assigned, what was observed and how the inference accounts for the relationship. A careful analyst can use individual observations within an appropriate design-aware approach. The error is to ignore the design while borrowing the confidence associated with a different one.
More observations do not rewrite the design
The sponsor had a practical reason to emphasise six hundred visitors. The programme required money, staff effort and attention, and a substantial dataset seemed to justify moving quickly. I agreed that the work should inform the decision. I disagreed that the observation count alone settled how strong the claim could be. More records from a shared setting do not retrospectively make each visitor independently randomised, just as a large report does not erase the way its evidence was collected.
Holding the inference created delay. Colleagues wanted a clear result from the ₹58 lakh programme, and further review could feel like a challenge to the value of the entire effort. I needed to explain which interpretation was being withheld and which evidence remained. The pilot was not declared useless. The company still had observations from a deliberately assigned shop experiment. It needed analysis that respected that design before the result could carry the confidence proposed in the preliminary account.
Customer-facing owners retained responsibility for delivery and any eventual rollout. Qualified analysts retained the work of examining the design and inference, while privacy and legal colleagues retained appropriate-use and permission decisions. My role was to keep the executive claim aligned with those responsibilities and the evidence available. I did not become the sole judge of the intervention's customer value simply because I questioned its statistical presentation. Commercial usefulness and inferential strength are related considerations, but neither absorbs the other.
The corrected discussion also avoided declaring that the programme had no effect. A result that has been overstated does not automatically imply the opposite conclusion. It may require a wider uncertainty account, further work or a different decision threshold, depending on the actual analysis and consequences. None of those outcomes can be inferred from this fictional summary alone. The disciplined response is to reopen the claim at the point where the design was lost, rather than replace excessive confidence with an equally unsupported rejection.
A restrained conclusion can preserve useful evidence
I want an executive reader to understand why the conclusion has been restrained. Simply saying that statistics are complicated does little to help a sponsor decide what to do next. In this case the explanation was concrete: twelve shops were assigned, and six hundred visitors were observed within them. The inference had to retain both facts. That gave the review a clear purpose and allowed colleagues to distinguish a methodological correction from a general reluctance to act on evidence.
The learning should begin before the next pilot. The intended decision, assignment level and proposed analysis ought to be agreed while the intervention can still be designed coherently. The appropriate review also needs to consider what happens if the result is uncertain, rather than treat a confident headline as the only acceptable product of the investment. That does not mean every experiment must be large or elaborate. It means its ambition and claims should be proportionate to the evidence it is designed to produce.
There is a lesson for leadership communication as well. A profile could make this case sound stronger by inventing a corrected effect or a successful rollout after my intervention. Doing so would undermine the point. The actual contribution described here is holding an inference, accepting the delay and keeping useful evidence available for qualified review. The reader can see the tradeoff without being asked to believe a result that the case has not established. That is a complete account of a particular decision, even while the wider programme remains open.
Six hundred visitors deserve to remain six hundred observations in the story. Twelve shops deserve to remain twelve assigned settings. A reliable explanation does not choose whichever count makes the claim sound more impressive; it preserves their relationship. My responsibility is to ensure that the executive conclusion carries the experiment that actually happened, including the conditions on its confidence. Evidence becomes more useful when its design remains visible at the moment someone wants to rely on it.