In September 2024, our product team was preparing to assess a new browsing feature. The experiment had a planned equal allocation, yet the recorded analysis contained 92,000 control observations and 108,000 treatment observations out of 200,000. The treatment appeared to attract more clicks. A large recorded sample made the result look substantial, but the imbalance raised a prior question: whether the observations represented the experiment we believed we had run.
I backed an eight-week measurement investigation and preparation programme costing €410,000. Analysts and engineers examined assignment and capture separately. They found differential event loss in the reporting path. This did not mean that an uneven count, by itself, identified every possible cause; it gave us a discrepancy to investigate. Once the capture defect was established, the available click comparison could not support the causal reading the product proposal required.
The product sponsor argued that the observed improvement was too attractive to ignore and that the calendar favoured a decision. I rejected the proposal to promote the compromised result into a release justification. We repaired the capture path and prepared a fresh trial with its allocation, measurement period and decision criteria stated beforehand. The new trial was a separate piece of evidence. I did not claim that repairing the system retrospectively recovered the missing observations or made the original comparison trustworthy.
The immediate consequence was a deferred feature decision and work that did not produce a celebratory growth chart. We retained the investigation as an engineering lesson, including the point at which measurement quality should stop an interpretation. There is no subsequent conversion or revenue improvement asserted in this case. A click measure would in any event need an explicit relationship to the business question; a sound experiment on one interaction would not automatically establish a lasting commercial benefit.
I would now require allocation and capture checks to be reviewed before a team presents a winner narrative. That sequence matters because an attractive result creates pressure to explain away defects. The programme strengthened our ability to say that a decision was not yet supported. For a CTO, investing in that capability is part of responsible product development, even when its first visible result is a proposal that must wait for better evidence.