Our 2025 recommendation review contained 600000 eligible product-visitor pairs in a candidate log. The previous ranking had shown 420000 of those pairs; 180000 had not been shown. Behavioural observations were available where exposure occurred. An offline replay was being described as a measure of preference across the eligible catalogue. I asked the team to retain the selection that had made some interactions observable and left other alternatives unseen.
The product sponsors wanted to use the existing record to move quickly. It was substantial, came from ordinary shopping activity and could support useful investigation without waiting for another assessment. I agreed that the record should be used. My objection concerned the scope of the claim. A visitor’s response to a displayed product did not, by itself, tell us how they would have responded to every product that the earlier ranking had withheld.
We committed EUR 3.4 million to evaluation work and withheld the all-catalogue preference claim from the offline account. The candidate, exposure and interaction records were kept distinct. Merchandising and experimentation owners took responsibility for designing the next supervised assessment, with privacy and technical colleagues retaining their respective decisions. This case reports no randomised result, accuracy score or conversion increase. It describes a limit recognised before the evaluation was promoted into a broader commercial conclusion.
The trade-off was a slower path to the decision the sponsors wanted. A more qualified report could feel less useful than one comparative score, and further assessment required attention from teams already supporting live retail work. I did not argue that every unseen alternative would perform differently or that the previous ranking was necessarily poor. Those would be additional claims. I wanted the institution to understand which observations its existing system had made possible before deciding how much confidence to place in the replay.
The counts and system are fictional, and no customer-level information belongs in this public example. A real account would require authorised evidence and appropriate technical review. My contribution was to preserve the circumstances of observation in a recommendation story, so that a large dataset did not become a claim to have observed choices the retailer had never placed in front of the visitor.