An analysis archive becomes more useful when another authorised person can run the stored work again. The code, data and relevant environment give the reader a way to inspect what was done rather than relying only on a finished report. I value that capability. I also want its meaning stated precisely. Replaying a process is not the same as independently validating its analytical choices or scientific conclusion. A programme can demonstrate the first while leaving the second to a different form of expert assessment.
The fictional Seravune Therapeutics exercise of 2025 attempted to replay sixty-four research analyses from stored code and data. Forty-nine ran again. Fifteen lacked a required dependency version and could not be replayed in the exercise. The first draft of the programme account leaned toward describing the archive as reproducible, with the remaining items presented as technical clean-up. I wanted the 49/15 split retained prominently. I also wanted the successful forty-nine described without implying that their scientific conclusions had received independent validation through the act of running the stored process.
We funded S$2.9 million remedial work and kept the two states separate. The case reports no later resolution of all fifteen. It records a useful capability, an unresolved group and a decision to address it. Scientists, statisticians and technology owners retained their respective responsibilities. The data office helped explain what the archive could demonstrate and what material was missing. That account was more qualified than a broad completion statement, but it was better suited to a decision about the next work and the confidence the institution could reasonably place in the current archive.
State what the replay exercise actually tested
The question was whether the retained material supported another run of the stored analysis. That is a practical and important question. A report can be difficult to examine if its process cannot be reconstructed from the available record. Yet a successful run may repeat the same assumptions or analytical choices as the original. It does not automatically test whether those choices were appropriate for the scientific question. I want the programme account to describe that boundary so the technical result receives credit without being used as an assurance it did not provide.
The code and data also need their relevant context. A dependency version can influence whether the stored process executes as expected. In the fifteen unresolved cases, that required version was absent from the retained material. The issue was therefore not captured by asking whether a code file and dataset existed somewhere. The exercise exposed a gap in the account of the environment needed to replay the work. I would want the responsible owners to understand and address that gap rather than let an inventory of files stand in for demonstrated capability.
This does not mean a public leadership story should publish code, participant data or restricted research methods. The case can explain the scope, the missing context and the decision without exposing the underlying material. A real account would need permission and careful review. Its value lies in showing that the leader understood what the exercise had and had not tested. The reader should be able to distinguish a data-management contribution from a scientific conclusion, even when both concern the same body of research work.
The unresolved environment belongs in the archive account
The fifteen analyses were not merely an awkward remainder after the successful majority. They identified work the archive programme still needed to do. Keeping them visible required continued ownership and resources. It also allowed the institution to judge whether the unresolved cases mattered to the uses proposed for the archive. A percentage of successful runs might be useful, but it should not replace the count and nature of what remains. The next decision depends on the actual gap, not just the attractiveness of the completed share.
The S$2.9 million commitment was a response to that gap, not evidence that it had been closed. I resist the habit of allowing funded work to acquire the status of achieved work in a retrospective profile. The ending of this case remains the reported replay result and the remedial programme. That is enough to show a leadership decision if the reasoning and trade-off are clear. An invented final success would make the narrative smoother while removing the very distinction the programme account was meant to preserve.
Colleagues who preferred the broader reproducibility statement wanted to recognise substantial progress. I agreed that forty-nine successful replays deserved recognition. The disagreement concerned whether that progress could describe the whole archive without qualification. We could acknowledge the completed work and still state the fifteen unresolved cases. The cost was a less concise headline and continued attention after the first exercise. I considered that cost reasonable because the archive was intended to help future readers examine evidence, and its own description needed to meet the same standard.
A technical success should keep its proper meaning
Successful replay and scientific assessment can support one another, but they answer different questions. A replayable process makes its steps more inspectable. An expert may then examine whether those steps and assumptions are appropriate. The data programme should not collapse that sequence into a single assurance word. I would ask which aspect has been demonstrated, who is responsible for the next judgement and what evidence that judgement requires. This keeps the institution from borrowing authority from a technical success while still valuing the practical foundation it provides.
The distinction connects with the other research-data decisions in my career. The shared reported unit in a cross-study view did not establish method comparability. A populated field would not turn missing source material into a normal finding. A row count did not make replicate readings independent samples. In each case, a convenient data property could be mistaken for a broader evidential conclusion. The situations required different specialists and different work, but they shared a need for the management account to retain the boundary between preparation and interpretation.
For a board, I would ask management to describe the archive's demonstrated capability in ordinary language. Which analyses can be replayed from the retained material? Which cannot, and why? What does successful replay leave unassessed? How is the next work owned? These questions can help directors judge a programme without operating its software or deciding its science. The CDO's role is to provide a clear account and seek the relevant expert input, not to turn board communication into a simplified certification that the underlying exercise never attempted.
I want research archives that help people revisit and question earlier work. That ambition is best served by an honest account of their present state. The 2025 exercise showed forty-nine replayable analyses and fifteen missing dependency versions. It also showed why the word reproducible needed a precise explanation. A technical process running again is a worthwhile result. Keeping that result within its actual meaning leaves the institution better able to carry out the further scientific scrutiny that the archive exists to support.