A processor that trains on its clients' data has changed its job

I believe an IT services company should not train a shared AI model on the content of its clients' data without each client's separate, informed and revocable agreement, even where its lawyers can construct an argument that the contract allows it. My reason is not mainly legal, although the law matters. A provider that trains on what it holds for others has changed what kind of company it is, and its clients are entitled to decide whether they want to do business with that company.

The pressure to do otherwise is considerable, and I understand it. Services firms hold some of the richest operational text that exists: millions of service-desk tickets, incident reports, configuration notes and emails about broken systems. A model trained across all of it could route requests faster, draft better replies and diagnose faults that no single client sees often enough to recognise. In 2023 our product team estimated that such a model could save Lumivara €14 million a year in service-desk effort. The estimate was credible, and a competitor was already advertising something similar.

What the processing agreement actually says

For most of the personal data in those tickets, Lumivara is a processor. Under the GDPR a processor acts on its client's documented instructions, and our contracts describe those instructions in terms of the service: receive the ticket, resolve it, keep records for a defined period. Training a model that serves every client is a different purpose. If we decide why and how that processing happens, we are acting as a controller for it, and we need our own lawful basis, our own transparency to the people concerned and our own answers to their rights. The people concerned are our clients' employees, who wrote to a service desk about a frozen laptop and had no idea their words might teach a machine.

There is also a technical problem that contracts rarely address. Language models can reproduce fragments of their training data. A ticket from one client's finance department, describing a salary payment that failed for a named employee, should never be able to surface in a reply drafted for another client. Redaction reduces that risk. It does not remove it, because names, account details and medical references appear in forms that no redaction pattern catches reliably.

Some colleagues argued that anonymisation solves both problems. I do not think it does for free text at this scale. Anonymised data falls outside the GDPR only when individuals can no longer be identified by means reasonably likely to be used, and a ticket that says the only Swedish-speaking payroll clerk in a particular municipal office has been off sick since March identifies someone, whatever has been removed from it. I spent my first job learning that lesson from a statistical table, and text is far less forgiving.

Metadata first, text by invitation

The approach we adopted has two tracks. Every client benefits from a routing model trained on structured metadata: ticket category, affected system, time of day, priority and resolution group. That metadata is still processed on clients' behalf, but it carries far less risk and stays close to the purpose for which we hold it. It routes 71% of tickets correctly, against 58% under the rules it replaced.

Training on ticket text happens only for clients who sign a separate addendum. The addendum says what the model will learn from, who can use the result, how text is redacted first, how the client can withdraw and what withdrawal costs us both. By the end of 2024, forty-seven clients had opted in, contributing about 1.9 million tickets a year, and their model routes 86% of tickets correctly. Before release we ran 4,000 prompts designed to draw one client's data into another client's session. None succeeded, which tells me our safeguards worked against the attacks we thought of, and nothing more.

The commercial result was half of what our product team had hoped for. The saving reached €6.1 million in 2025 rather than €14 million. Our head of managed services thought we had given a competitor a year's start. Perhaps we did. I would still rather explain a smaller saving to our board than explain to a municipality why its employees' sick-leave tickets helped draft replies for a bank.

Build the exit before the model

The weakest part of our programme, in hindsight, was withdrawal. We promised that a client could leave the opt-in and have its data removed from the model, but we built the retraining process only when the first request arrived. It took three weeks, during which we were in breach of the spirit of our own addendum if not its letter. The second withdrawal took nine days. I would now refuse to launch any model trained on client data until the removal path has been tested end to end and its cost is known.

The addendum itself needed more care than I gave it. Our first draft was written by lawyers for lawyers, and about 300 clients sent questions before deciding. The questions were good ones: whether a model trained partly on their data could be sold to a competitor, whether retraining would change the service they received, whether their own employees could object. A clearer document, drafted with a few clients' data protection officers, would have produced more informed decisions and probably more of them.

The obvious objection is that clients benefit from shared models and that consent processes favour the cautious over the useful. I accept part of it. Some clients who declined would have gained from a better model, and in principle a cross-client model trained with strong safeguards might serve everyone well. My answer is that the choice belongs to the client. A provider that makes it on their behalf, however carefully, has decided that the data it holds is partly its own.

For boards of services companies the practical test is short. Ask management which AI models are trained on client content, under what agreement, and what happens when a client says no or changes its mind. If the answers rely on a general clause about improving services, ask whether the clients who signed that clause would recognise the use. If management hesitates, so would the clients.