I have seen an AI prototype described as inexpensive because each response cost very little to produce. That statement can be arithmetically correct and commercially unhelpful. A user does not need a response merely to exist. The response must do something useful under the conditions of the task, and an unusable answer can create additional review or reconstruction work. I therefore begin a product evaluation by asking what a person can accept and how we will recognise it. Only then does expenditure become a meaningful comparison. Even that comparison remains narrower than the economics of an operating service, which includes people, development and continuing support as well as inference.
Define what the user can accept
Acceptance needs a task, not a general impression of intelligence. In a document-assistance exercise, we required factual support from supplied material, specified fields and an explicit refusal where the source was insufficient. Those criteria reflected the work the prototype was meant to help. They would not automatically suit creative exploration or another document process. The reviewers agreed them before comparing routes. That order matters. If the criteria are adjusted after seeing an attractive result, the evaluation can quietly reward the system for doing what it already does rather than examine whether it serves the user's original need.
I also want reviewers to distinguish a correct refusal from an unsupported attempt to be helpful. A system that answers every request may look more productive while creating more verification work. Conversely, a system that refuses too readily may satisfy a narrow safety measure and fail to provide useful assistance. The product needs an account of both behaviours. In our exercise, an accepted output could be a properly supported draft or a refusal that followed the agreed insufficient-evidence condition. We did not let a confident tone substitute for source support. Nor did we count a refusal as acceptable simply because it reduced the chance of an incorrect factual statement.
The people making those judgements need a procedure they can apply consistently. We used subject reviewers who did not see which route had produced each answer. That reduced one possible influence on their assessment, but it did not make every judgement objective or eliminate disagreement. Ambiguous cases required a shared interpretation of the criteria. I would rather retain a small account of those disagreements than report an acceptance percentage as if it were measured by an infallible instrument. Product leaders should understand what the reviewers were asked to do and where reasonable people could still see the task differently.
Keep the numerator and the missing work
Our comparison used the same 1,000 permitted requests, with one answer per request from each route. The existing route incurred CNY 2,240 in measured inference charges and produced 800 accepted answers. The alternative incurred CNY 2,100 and produced 875 accepted answers. The resulting ratios were CNY 2.80 and CNY 2.40 per accepted output. I report the underlying figures because a ratio alone can obscure how it changed. The second route both cost less in the measured run and produced more accepted answers. A different result could reduce the ratio by changing only one of those factors, with a different implication for the product.
The full inference charge belongs in the numerator, including expenditure on unaccepted outputs. Removing those charges would make the usable work appear cheaper by pretending the unsuccessful attempts never happened. The unaccepted group also remains part of the service problem. In the second run it comprised 125 outputs. A live product would need an appropriate route for such work, whether that means review, a different attempt or a clear statement that the task has not been completed. I do not infer the cost of that route from the inference bill. It is a separate question that must be answered before extending the experiment into an operating promise.
Several other costs were deliberately outside the narrow ratio. Reviewer time was not included. The six-week evaluation and implementation programme cost CNY 1.9 million, which was not divided across those 1,000 requests as if they represented the product's eventual volume. Continuing monitoring and support were also outside the inference charge. Those exclusions do not make the ratio useless; they define what it can tell us. The mistake would be to rename it total cost per completed customer task. A bounded measure is valuable when its boundary is visible and the investment discussion includes the other accounts needed to make a decision.
I supported a limited trial because the result justified learning more about that task. The commercial sponsor wanted to extend the cost expectation to every document workflow. That would have converted a defined comparison into a forecast about populations we had not examined. Source length, task difficulty, required response structure and the proportion of insufficient-evidence requests could all affect both cost and acceptance. The next product decision therefore needed a stated population and a way to observe whether its work resembled the experiment. A useful result should earn the next question, not exempt the programme from asking it.
Ask a fresh population to challenge the result
The most important limitation was our familiarity with the test set. Engineering knew its requirements while developing the routing approach. Repeating the same population made the before-and-after calculation comparable, but it did not provide a clean account of performance on unseen work. I would reserve a separate final evaluation population earlier in the next programme. That set should reflect the intended task while remaining outside the development cycle. The point is not to create a ceremonial pass mark. It is to give the product an opportunity to discover whether an attractive result depends too heavily on examples the team already understands.
An unseen population can still be unrepresentative. It may omit uncommon but consequential document structures or contain a different mix of work from the intended service. I therefore ask how requests were selected, what permissions govern their use and which categories remain absent. Technical teams should not infer a new right to use client material merely because additional examples would strengthen an experiment. Where the available set is limited, the conclusion should say so. Responsible evaluation sometimes produces a narrower product scope rather than a larger confidence claim, and that can be a commercially useful outcome.
The longer-term discipline is to keep acceptance, cost and operating context connected as the product changes. A new model, a revised prompt, different routing or a different source collection can alter the meaning of an earlier result. The organisation needs a proportionate way to re-examine consequential changes without treating every small adjustment as an entirely new research project. I want the team to know which assumptions support the current decision and what would make them doubtful. Counting accepted work helps because it puts the user's task back into the account. It does not remove judgement; it gives that judgement a clearer object and a more honest set of numbers.