In November 2024, a document-assistance prototype looked cheaper when reported as expenditure per model response. That denominator counted useful drafts, unusable drafts and answers that a reviewer would have to rebuild. The product team wanted to select the cheaper route quickly. I asked for a comparison at the level of work a user could actually accept, while keeping the experiment narrow enough to explain.
We assembled a fixed set of 1,000 permitted test requests with representative source documents. Each route produced one answer per request. Before comparing them, subject reviewers agreed criteria for factual support, required fields and an explicit refusal when the supplied material was insufficient. The reviewers did not see which route had produced an answer. The set and criteria remained unchanged across the two runs; no client document was made available for a different purpose merely because the experiment was useful.
The existing route incurred CNY 2,240 of measured inference charges and produced 800 accepted answers. The proposed routing approach incurred CNY 2,100 and produced 875 accepted answers. Dividing the full charge by accepted outputs gave CNY 2.80 and CNY 2.40 respectively. We reported all four underlying numbers. The lower figure did not include reviewer time, development expenditure or the cost of later operating support, and the 125 unaccepted outputs in the second run did not disappear from the report.
I supported a limited product trial rather than broad adoption. The commercial sponsor wanted to extrapolate the observed cost to every document workflow. I declined that forecast because the tested set represented one task, and the routing choices had been developed with knowledge of its requirements. The six-week evaluation and implementation work cost CNY 1.9 million, a separate investment from the measured inference charges.
The trial made a useful unit of comparison available without claiming general model superiority. A different mix of requests, longer sources or changed reviewer criteria could reverse the result. Next time I would reserve a separate, unseen evaluation set before engineering begins. Keeping the development and final evaluation populations distinct would provide stronger evidence than repeating a familiar set, even when every calculation within that set is correct.