I think enterprise software companies should judge automated output by the work a customer can complete without having to repeat it. An answer produced in seconds may still require a trained employee to inspect every field, locate the source and correct an error that is difficult to recognise. Counting outputs treats that employee's effort as somebody else's cost. The product may be useful, but its value cannot be established honestly until the work surrounding the output is included in the comparison.
This changes the executive question. Instead of asking whether the system can process more documents, I ask which document classes it handles, what errors remain and what the user must do next. The answer may justify a narrow release with strong practical value. It may also reveal that an impressive demonstration is not yet a sound commercial proposition. Neither conclusion should depend on whether the technology is fashionable. The customer is buying a usable change in work, even when the invoice is expressed through another unit.
Follow the task beyond the generated answer
Our document assistant initially looked convincing on clean examples. It extracted fields and organised material faster than a person performing the same first pass. The limitation became clearer when customer operations colleagues supplied difficult documents and described their checking habits. Some errors were obvious and inexpensive to correct. Others appeared plausible, which meant a user had to compare the output carefully against source material. Those differences mattered more than a single aggregate accuracy figure because the consequence of trusting a mistaken field was not uniform.
We therefore defined the task through the user's decision, not the model's response. What did the customer need to establish before moving the document into the next workflow? Which fields required confirmation? What happened when an exception could not be resolved? The evaluation included review time and correction, as well as the initial processing step. This required more work than timing a demonstration. It also produced a more credible basis for deciding where the feature could help and where it could merely make uncertain work move faster.
Product teams should be explicit about the population being evaluated. A result for two well-understood document classes does not establish performance across every file a sales demonstration might suggest. The same caution applies to language, formatting and the condition of source material. I want the commercial team to know those boundaries well enough to explain them without reaching for a broad assurance. A customer who understands the intended use can make an informed choice. A customer who discovers the boundary through an expensive error has been given a different product from the one they thought they bought.
The user's review step must also be realistic. It is not enough to add a message asking people to check the output if the workflow encourages rapid acceptance or hides the source they need to verify. We considered how uncertain fields were presented, how a correction was recorded and when a person could return to the original material. These are product decisions, not merely communication decisions. The design of the work determines whether human oversight is a usable action or an instruction that is unlikely to survive everyday pressure.
Price the correction as well as the computation
Commercial discussions can become distorted when software usage is easy to count but customer effort is not. A company may know precisely how many documents it processed and much less about the time users spent correcting them. If pricing rewards volume alone, both parties can end up focusing on an activity that is only loosely connected to value. I do not think every contract needs a complex outcome formula. I do think the executive team should understand the gap between its billing unit and the work the customer actually benefits from.
That understanding affects our own economics too. A feature can carry attractive direct processing costs while generating a large support burden. Unclear boundaries produce questions, rework and disputes about expected behaviour. Some costs appear in product operations; others appear in account management or customer success. If those functions report separately, the feature's apparent margin may depend on work being absorbed elsewhere. I ask finance to examine the full operating consequence before treating an early usage result as evidence of a repeatable business model.
There is also a question of accountability when output quality changes. Customers need a meaningful account of what has changed and whether previously approved workflows require review. The company needs an internal process for assessing the effect of an update on relevant tasks. I do not regard that as an argument against improving the system. It is a reason to make change management part of the product obligation from the beginning. A useful automated feature still belongs inside the customer's operating environment, with responsibilities that persist after launch.
In our controlled rollout, median handling time for the two approved document classes fell by 34% after review and correction were included. That result supported a narrower commercial claim than our initial plan. It also gave the account team something more concrete to discuss than a processing-speed comparison. We could explain the workflow, the review requirement and the boundaries of the observation. The three other document classes remained outside the release. Their exclusion was part of the value proposition's accuracy, not a detail to conceal until a customer asked.
Release a boundary the customer can understand
The decision deferred US$3.2 million of planned first-year bookings. I mention the figure because release discipline is easy to praise when its cost remains abstract. Sales colleagues had invested in the wider proposition, and the revised scope affected their plans. We changed the forecast and the demonstrations together. It would have been unreasonable to narrow the product formally while leaving the commercial team rewarded for selling the broader promise. The executive decision needed to reach the incentives and materials through which customers encountered the feature.
A bounded release should have a route to reconsideration. We agreed what additional evidence would be needed before another document class entered scope: the pattern of errors, the effort of review and the consequence of a mistaken output. Those conditions were more useful than a date chosen to fit a launch calendar. They also gave engineering and product teams a clearer basis for prioritisation. The work was not simply to increase an aggregate score; it was to remove the obstacles that prevented a defined customer task from being completed responsibly.
I would encourage another chief executive to observe a user completing the full task with the proposed feature, including an awkward exception. Watch where the user hesitates, what they verify and what they do when the output is plausible but wrong. Then compare that observation with the commercial description and the internal business case. The differences will not necessarily invalidate the product. They will identify work that the organisation needs to own before asking a customer to rely on it. That is a stronger basis for growth than treating every generated answer as a completed piece of value.