A dataset is fit for a question, never fit in general

About once a month a sponsor or a data intermediary asks Aurevalle some version of the same question: is your data research-grade? Sometimes the request arrives with a quality framework attached, asking for completeness percentages across a list of variables. Sometimes it asks for a certificate. I understand the request, because for four and a half years I made it myself, from Copenhagen, to hospitals across Europe. I now think it rests on a mistake, and that hospitals do sponsors no favours by answering it on its own terms.

A dataset is not fit or unfit in general. It is fit or unfit for a particular question. The same records can be excellent evidence for one purpose and misleading for another, and the difference usually lies in details that a general quality score cannot see.

The certificate sponsors keep asking for

Quality frameworks for real-world data are useful. They ask sensible things: are values complete, do they conform to expected formats, are they plausible, are they consistent over time? When we first assessed our warehouse against such a framework, it exposed real problems, and we fixed several of them.

But consider smoking status. Across our whole warehouse it is recorded for 58 per cent of adult patients, a figure that looks poor. For patients diagnosed with lung cancer it is recorded at diagnosis for 91 per cent, because oncologists ask and note it as part of their assessment. A study of lung cancer outcomes by smoking history can rely on our data. A study of smoking among patients admitted with fractures cannot. A single completeness score averages the two and describes neither.

I learnt this first on the sponsor's side. In 2019 at Tornvig Pharma we planned to compare a rare-disease medicine's single-arm trial with patients from a European natural-history registry. The registry was careful, well-run academic work, and by any general standard its data were sound. But it recorded lung function about every fourteen months, against every three months in our trial, and baseline values were missing for nearly half its patients. For describing the course of the disease it was valuable. For estimating a rate of decline to set against our trial it was unfit. We withdrew the comparison, and it cost the company in two countries' assessments. No quality certificate would have warned us, because the registry's quality was never the problem. The fit between the data and the question was.

Clinicians' habits are the real data model

Hospital records are written by people treating patients. Their content reflects what a clinician needed to note at that moment, for that patient, in that system and under the rules that applied then. That is why fitness cannot be judged from the structure of the database alone.

Three examples from our own warehouse make the point. First, French hospitals code every stay for the PMSI, the hospital discharge database that drives activity-based payment. Those codes are accurate for the job they are designed to do, but a diagnosis recorded as a secondary code because it affects payment is not the same thing as a diagnosis a clinician has confirmed for research. Second, laboratory results are among our most reliable data, because analysers produce them and they are checked before release; yet reference ranges and units changed at two of our hospitals when their laboratories merged in 2017, and an analyst unaware of that will discover an epidemic of abnormal results that never happened. Third, the date of disease progression in cancer, which many sponsors want, is not a field in our records at all. It lives in clinicians' letters and imaging reports, and extracting it reliably requires either manual abstraction or language processing that must itself be validated.

None of this is visible in a data dictionary that lists tables and formats. It is visible to the data stewards who sit with clinical teams and record how each variable is born. That provenance, more than any completeness score, is what makes hospital data usable as evidence.

Regulators and health technology assessment bodies increasingly ask the same thing of real-world evidence: not whether a dataset is good in the abstract, but whether it is relevant and reliable for the question in hand. Europe's joint clinical assessments, which began in 2025 with new cancer medicines, will put further weight on that distinction. A sponsor who arrives at a hospital asking for a certificate is in effect asking us to answer the assessor's question in advance, without knowing what it will be.

A fitness note for every request

Since 2024 Aurevalle has answered every data request with a fitness note of two or three pages, written by a steward and reviewed by a statistician. For each variable the study needs, the note says how it is recorded and by whom, how complete it is in the relevant population and period, what changed over time, and what we would not trust it to show. It ends with a plain recommendation: proceed, proceed with stated limits, or redesign.

The notes take about three working days each, and we charge for them. Some sponsors resisted paying for what they regarded as diligence the hospital owed them. Most now ask for the note before they write their protocol, which is when it is most useful. Twice in 2025 we advised a sponsor not to proceed, because the outcome it needed was not reliably recorded in our hospitals. Both accepted the advice, and one redesigned its study around an outcome our records could support.

The notes have changed our own behaviour too. Writing down what our data cannot show has made us more careful about what we claim for it, including in our own researchers' publications. It has also made our stewards' work visible to the clinicians whose habits they describe, and several departments have changed how they record a variable once they saw what its gaps cost a study.

For directors of companies that rely on real-world evidence, the practical question is simple. When management presents a study built on routine data, ask whether someone wrote down, before the analysis began, why those particular records could answer that particular question. If the answer is a quality certificate, the work has not been done. If it is a fitness note, it probably has.