Let the assistant answer the parcel, not the fault

Our customer contact team handles about 3.4 million contacts a year, and four in ten ask some version of the same question: where is my order? A large retailer of televisions, laptops and washing machines also receives a steady stream of harder questions. The dishwasher stopped draining after fourteen months. The laptop screen has a line across it. The manufacturer says the warranty has expired, so what will the store do? When we evaluated a generative AI assistant for chat and messaging in early 2024, the difference between those two kinds of question decided almost everything about the design.

The vendor's proposal was capable and confident. It projected that the assistant would resolve 65 per cent of contacts without a person, including returns and fault claims, saving about A$9.5 million a year. Our general manager of customer contact was keen, and so was our chief financial officer. I was keen too, on parts of it. We launched a narrower assistant than the one proposed, and I want to explain why, because I suspect many retailers are about to make the opposite choice.

Containment flatters the vendor

The usual measure of a customer-service assistant is containment: the share of conversations that end without a transfer to a person. It is easy to count and easy to improve. An assistant can raise its containment by answering confidently, by making the transfer a little harder to find, or by ending conversations that customers abandon in frustration. None of those helps the customer, and the last of them looks like success on a dashboard.

We replaced containment with a stricter measure. A contact counts as resolved only if the same customer does not come back about the same order or product within seven days. In the first quarter after launch our assistant's containment was 44 per cent, while its resolution on our measure was 31 per cent. The difference was customers who had received an answer, left, and returned to ask a person the same thing. Had we reported containment, we would have overstated the benefit by about a third and missed the cost of the recontacts entirely.

Measured honestly, the assistant is useful. It resolves 31 per cent of chat and messaging contacts end to end, mostly delivery questions, store stock, opening hours and product specifications, and it saves about A$3.6 million a year. Customers rate the conversations it resolves at 4.2 out of 5, slightly above comparable conversations handled by people, largely because it answers in seconds at eleven at night.

Where consumer guarantees change the design

Before launch we wrote 1,200 test conversations with our frontline agents, who are better than any engineer at imagining what customers actually say. Two hundred and forty of them were fault claims. In nine, the assistant told the customer that the product was outside the manufacturer's warranty and that there was nothing more we could do.

In Australia that answer can be wrong. Under Australian Consumer Law, consumers have guarantees that goods will be of acceptable quality, and those consumer guarantees sit alongside a manufacturer's warranty rather than ending with it. Whether a particular fault entitles a customer to a repair, a replacement or a refund depends on the product, its price, how it has been used and how long such goods should reasonably last. Trained people make those judgements with the customer, and sometimes with the manufacturer. A fluent sentence that sounds like policy, delivered by a retailer's own assistant, can mislead a customer about rights they hold. The ACCC treats misleading statements about consumer rights seriously, and so should every retailer.

Nine in 240 is under 4 per cent. A vendor might fairly reply that people make mistakes too, and some of ours do. But a person's mistakes are scattered, noticed and coached. An assistant's mistake is repeated at scale, in the same words, until someone finds it. We decided that the assistant would not answer fault, refund or warranty questions at all. If a customer mentions a fault, a return or a warranty, the conversation passes to a person.

Writing the handover before the answer

Narrowing the assistant made it more valuable in a way we had not expected. Because every fault conversation now reaches a person, we invested in what that person receives: a short summary of what the customer has said, the order, the product's age and history, any earlier contacts and the store or carrier involved. Average handling time on fault claims fell by 22 per cent, and fault-claim complaints escalated to our customer relations team fell by 17 per cent. Some of the assistant's most useful work happens in conversations it does not finish.

We applied the same discipline to delivery dates. In testing, the assistant occasionally offered a plausible date that no carrier had given us. It now answers delivery questions only from carrier tracking data, cites that source in its reply and says plainly when it does not know. Each week we sample 400 conversations, and an agent checks every sampled answer against the order record.

The narrower design, together with the stricter measure, left us about A$5.9 million a year short of the original proposal. That is a real number, and our chief financial officer was right to press me on it. My answer was that a retailer's credibility on faulty products is worth more than the cost of the people who handle them, and that we could widen the assistant later if testing showed it deserved trust. Since launch we have widened it once, to rescheduling deliveries. We have not widened it to faults, and I do not expect to soon.

For other retailers I would offer three suggestions. Write your test conversations with the people who answer customers today. Measure resolution by whether the customer comes back, not by whether the conversation ends. And before you buy anything, list the questions where a wrong answer would misstate a customer's legal rights. Those questions belong with people who can be accountable for the answer.