Keep the question intact through the calculation

A historical forecast exercise must distinguish information available at issue time from observations that became available later. Otherwise the evaluation can reward a model for knowledge the real forecaster did not possess. I ask the team to reconstruct the input boundary and preserve uncertainty about missing historical sources rather than quietly substitute later observations. Our 2019 review found eighty-six of 120 windows with that problem. The correction required withdrawing the performance claim, not merely improving a footnote. A later evaluation can support only the conditions and cases it actually examines. I do not assume that chronological separation alone solves every leakage risk; the provenance and availability of each input also matter. The CTO should make those questions part of product acceptance before a benchmark becomes a promise to customers.

An optimisation result is useful only if its status and constraints are interpreted correctly. I ask whether a stated limit is genuinely enforced or has been represented as a preference that the model can trade away for a better objective score. The 2024 review found four of forty proposed plans outside a declared customer limit. The attractive ranking did not make those plans feasible under the problem we had agreed to solve. We withheld them and corrected the product’s interpretation with qualified reviewers. The remaining unresolved case stayed outside release after further work. This is a product-validation question, not authority to relax a customer’s operating limits. Customers and their professionals must define the constraints and approve any practical decision, while the software must faithfully report whether its proposed answer satisfies them.

Hourly energy can conceal a shorter demand peak. Four quarter-hour energy values of 25 kWh each give 100 kWh over the hour and a constant 100 kW during those intervals. A different profile of 50, 20, 15 and 15 kWh has the same hourly total but a first-quarter average demand of 200 kW. A hypothetical 150 kW planning limit therefore distinguishes the profiles even though the hourly average does not. I use such examples to ask whether the product’s resolution matches the decision it is supporting. They are illustrative quantities, not operating instructions for an installation. Professional users must define suitable boundaries and constraints. The CTO’s responsibility is to prevent aggregation from silently removing a feature that makes one planning answer unsuitable.

Technical teams can feel pressure to deliver a complete recommendation for every scenario. I want the product to make an unsupported or infeasible result visible rather than improve its appearance by hiding the limitation. That requires clear definitions of evaluation conditions, constraint ownership and the status of an output. Quality colleagues need authority to challenge a claim even when a commercial demonstration has already created expectations. My operating envelope funds research and engineering, but it does not give me permission to assume a customer’s dispatch or safety responsibility. In board oversight I would use the same experience to test whether management’s technology claim matches its evidence, without choosing algorithms or certifying a model personally. A useful limitation should change release scope or decision language, not remain detached from the answer people are invited to use.