The size of an experiment is not its transaction count

## Start with the unit that received the offer

An experiment can feel substantial when the transaction file is long. There are orders to inspect, customers to classify and a sales chart with a clear difference between two groups. That sense of substance becomes misleading when the offer was assigned to only a small number of commercial units. The transactions within one distributor catchment share local conditions and the same assignment. Treating each as a separate experiment can make the evidence appear much stronger than the design actually provides. The right starting question is where the decision to change the offer was made.

At the fictional Zelvani trial, sixteen catchments were allocated between two commercial routes, eight on each side. We spent ₹54 lakh testing a different reorder communication and service arrangement. The trial produced many transactions, but the allocation still contained sixteen catchments. A large customer order could move a catchment's result substantially. Local relationships, industrial buying cycles and the distributor's existing account mix also mattered. None of those conditions vanished because the final spreadsheet contained enough rows to make the comparison look comfortably large.

At the end of the agreed period, aggregate sales in the offer group were 8.6% higher than in the comparison group. That was an observation worth understanding. It was not, by itself, a precise estimate of what would happen if the business introduced the offer everywhere. Commercial colleagues wanted a decision, and their enthusiasm was reasonable: the trial had used money and attention, and distributors wanted to know whether the arrangement would continue. My responsibility was to explain the strength and limits of the comparison without treating that commercial urgency as an analytical mistake.

I had accepted the trial design and therefore shared responsibility for its limits. It would have been easy to arrive at the end with a technical objection and behave as though the commercial team had misunderstood a perfectly clear proposal. The original paper had not explained the distinction between many transactions and a small number of assigned groups well enough. It also had not examined connected buying behaviour closely enough before allocation. Those omissions belonged to the design discussion, not merely to the final presentation of results.

## Keep the inconvenient movement in the account

Eleven buyers traded across the assigned routes during the trial. Their behaviour was commercially plausible: customers do not organise their purchases to preserve an analyst's comparison. They may have relationships with more than one distributor or need availability that their usual route cannot provide. Once that movement appeared, a tempting response was to remove those buyers and present the remaining population as a cleaner experiment. But selecting exclusions after seeing behaviour can create a new bias. The supposedly cleaner account may describe a population chosen partly by the response to the offer.

We retained the original assigned-group account and described the cross-route activity separately. That did not solve every problem, but it preserved a clear statement of what had been allocated and what subsequently happened. Additional analyses could help explain behaviour, provided their purpose and limitations were visible. I did not want a sequence of alternative cuts ending with whichever chart most strongly favoured the rollout. Nor did I want the presence of crossover used to dismiss every useful observation from the trial. Both responses would substitute a preferred conclusion for an honest account of the design.

The practical question was whether the offer's boundaries matched the way this network traded. Catchments looked separate on a planning map but were not fully separate in customer behaviour. A future design might need a different grouping, different operating controls or a question that accepts connected trading as part of the policy being tested. Each possibility carries a cost. Larger groups can reduce the number of independent assignments available. Tighter operating boundaries may make the trial less like ordinary business. The design needs to acknowledge those trade-offs instead of presenting random allocation as a universal cure.

We also examined the influence of individual catchments and unusually large orders. With a small group count, a result can depend heavily on a few commercial events. That sensitivity is useful to report even when it makes the headline less stable. I want management to know whether the apparent difference is widespread across the network or concentrated in a small part of it. The answer does not automatically establish causation either way, but it changes how confidently we should generalise from the trial to a larger and possibly different commercial population.

## Choose the next commitment in proportion to the evidence

The decision was not between declaring the trial a success and declaring it worthless. We had learned about distributor administration, buyer movement and the feasibility of the offer. We had also observed a positive aggregate sales difference. Those findings could support a further bounded investigation. They did not require us to describe the offer as ineffective merely because the evidence fell short of a confident broad-rollout claim. The distinction matters to the team that did the work: acknowledging limits should preserve useful learning, not erase it with an all-or-nothing verdict.

I recommended redesigning the next trial rather than expanding immediately. The competing choice was to proceed on a promising signal while accepting the uncertainty and monitoring the commercial consequences. That would have been a legitimate management risk decision if properly described. My objection was to presenting it as the inevitable conclusion of decisive experimental evidence. A company can choose to act before evidence is conclusive. It should not need to overstate the evidence in order to explain why it is prepared to take that chance.

Meera agreed to the redesign. The ₹54 lakh cost remained part of the account; it was not retrospectively transformed into an investment with a demonstrated return. We had not established the commercial effect well enough to make that claim. The next paper had to state the question, assignment unit, observation period, treatment of crossover and decision standard more clearly. It also had to consider whether the business could support the required scale of trial. Sometimes the honest conclusion is that the available network cannot cheaply answer the desired question with the precision executives initially expect.

I would involve the people who understand customer routes earlier. A distributor manager may know that two supposedly separate territories share a buying office or that a large industrial customer routinely splits purchases. That knowledge is part of the design, not an anecdote to be added after the statistical plan is complete. Equally, the analyst must explain why that movement matters rather than simply requesting a tidier network. A workable experiment is jointly designed around commercial behaviour and an explicit inference, with neither side assuming that the other will repair the mismatch later.

For oversight, I would ask three plain questions. What was actually assigned differently? How many sufficiently separate units received that assignment? What happened that made the groups less separate than intended? The answers often reveal more than the length of the transaction file or the prominence of the uplift percentage. In our case they led back to sixteen catchments, eight on each route and eleven buyers crossing between them. That account was less exciting than a single sales number. It gave the business a firmer basis for deciding how much more it was prepared to commit.