A link is not the relationship it appears to represent
An archive assistant can produce a fluent answer and attach a reference that appears reassuring. The item exists, the title looks relevant and the link opens successfully. Yet the cited edition may not contain the claim being made, or the relevant passage may belong to a different version of the publication. The visible citation has done enough to encourage confidence without necessarily providing the relationship that confidence assumes. For a publisher, that is a central design question rather than a minor presentation defect.
I approach it as the CIO responsible for an internal research service, not as the editor who decides what should be published. Staff can benefit from help locating material across a large archive, but the tool needs a purpose and a boundary. Finding a possible source is different from establishing that a proposition is supported. Establishing support within a passage is different again from deciding that it is accurate, current and appropriate for a new article. An assistant should make those distinctions easier to examine rather than combine them in one polished paragraph.
Our initial review examined 240 generated answers against their cited source and edition. Editorial reviewers accepted 180 as supported under the examination criteria. Sixty failed: thirty-six made a claim the cited material did not support, and twenty-four relied on the wrong edition. The failures were not all broken links or missing documents. Some of the most misleading responses were accompanied by material that was real within the archive but did not justify the particular answer. Counting citations would have missed the point of the examination.
A wider staff launch with a caution to verify every answer was an attractive alternative. The innovation sponsor wanted colleagues to experience the potential of the service and discover useful applications through use. There is value in learning from real work, but a general warning can become a way to transfer a known design problem to every user. If the system makes evidence look settled when it is not, asking the user to check everything does not remove the responsibility to improve how the service presents and limits its answers.
An abstention can be an informative result
I supported twelve weeks of work costing AUD950,000. We improved edition selection, the visibility of supporting passages and the circumstances in which the assistant should decline to answer. The service remained an internal research aid, with no authority to publish automatically. Editorial reviewers judged the relationship between the response and its source. My team was responsible for the system changes and the examination record, while the editorial decision about using material in a publication remained where it belonged.
Re-examining the same 240-question pool produced 220 supported responses and twenty abstentions. The twenty were not secretly counted as correct answers. They represented occasions on which the service did not provide the requested synthesis under the revised conditions. That could still be useful behaviour if the user received a clear explanation and a route to inspect possible material. It was preferable to presenting a confident account that the available evidence could not carry. The result gave us a more discriminating description than a single number labelled accuracy.
Abstention also has a cost. A staff member may need to continue the search manually, consult a colleague or decide that the question cannot yet be answered. If the tool declines too readily, it may offer little benefit over an ordinary search. I would therefore examine the reasons for refusal, the work left to the user and whether the service helps them take a sensible next step. The aim is not to celebrate a refusal count. It is to make the boundary of a useful answer visible and operationally manageable.
Edition matters because an archive preserves change as well as content. A later item may correct, qualify or replace an earlier account. A historical question may legitimately require the older version, while a current summary may need the newer one. Technology should help the user see which question they are asking and which edition the response relies upon. It should not silently choose whichever version produces the most convenient sentence. The archive's value includes its ability to preserve those distinctions, even when a synthesis interface encourages the reader to expect one simple answer.
A known question pool cannot establish every future answer
The improved result came from the same examination pool. That is useful for checking whether the identified problems have been addressed, but it is not a complete account of future performance. The system and its supporting rules may have become particularly well suited to those examples. Unseen questions, different editions and new combinations of material can expose other failures. I kept that limitation beside the result and required a broader fresh examination before treating the known-pool improvement as a reason to expand the claim.
I would also separate source support from other qualities an enterprise may care about. A response can accurately reflect a cited passage while omitting important context elsewhere. The passage itself may contain an outdated assertion. A suitable internal research summary may still be inappropriate to publish. These distinctions make evaluation more demanding, but avoiding them makes the account less useful. The executive team should know what the reviewers examined, what they accepted and which conclusions remain outside that specific exercise. A single reassuring percentage cannot substitute for that explanation.
The commercial discussion needs similar restraint. We spent AUD950,000 on a defined improvement programme; we did not establish an equivalent labour saving merely because the assistant could produce a paragraph quickly. Staff still needed to examine the material and decide whether it served their purpose. Any later claim about time, quality or productivity would need observation of actual work. The business case can recognise potential without treating a demonstration's speed as a realised operating benefit or a reason to remove the people whose judgement makes the archive useful.
What I want from the service is a better encounter between the user and the evidence. The answer should make its source relationship inspectable, its edition clear and its limits understandable. Where support is inadequate, the tool should stop at a point that helps the user continue responsibly. That is a more modest proposition than an assistant that appears able to answer everything, but it is also more relevant to a publisher whose work depends on distinguishing what a source says from what someone wishes it would say. A citation is valuable when it enables that examination, not when it merely decorates a confident response.