A better search average can make one reader's route worse

Our search candidate won the aggregate comparison and still gave us a reason to hold its release. On an 800-query evaluation it produced 624 acceptable results against 604 for the existing system. That was a real difference on the set we had reviewed. However, the candidate performed worse on transliterated queries, one of four groups we had deliberately kept equal in size. The decision was not resolved by declaring the aggregate wrong. It required us to explain what the aggregate left out.

The set contained 200 English-title queries, 200 original-script queries, 200 transliterated queries and 200 mixed descriptive queries. The candidate's acceptable-result counts were 186, 172, 116 and 150. The existing system's were 168, 154, 144 and 138. Adding the groups was arithmetically straightforward. Deciding whether the resulting improvement justified a weaker route to some books was a product judgement informed by evidence, not a conclusion the addition itself could supply.

## Decide what an evaluation set represents

We chose equal groups to make differences visible. That was useful for diagnosis, but it did not establish that one quarter of live searches belonged to each group. A balanced review set and a traffic-weighted estimate answer different questions. If we had presented the overall score as a precise account of the whole customer experience, we would have made a claim the sampling design did not support. The diagnostic value of a set should not be confused with population representativeness.

The distinction also affects how people discuss small groups. A group may be commercially important, poorly served or strategically relevant even when it accounts for a smaller share of current traffic. Conversely, equal representation in an evaluation does not by itself establish equal business weight. Those choices need an explicit rationale. I wanted colleagues to see both the result and the design that produced it, so nobody had to infer a release policy from the way the analyst happened to arrange a table.

An acceptable result also needs a definition. A query can reasonably refer to more than one work, and a title remembered imperfectly may still lead to a useful answer. Editorial reviewers agreed the criteria before comparing systems and resolved disagreements through review. That did not turn their judgement into an objective truth for every reader. It provided a consistent basis for this comparison and a record of where interpretation mattered. Without that discipline, a difference in reviewer expectations could be mistaken for a difference in search quality.

We did not use the exercise to infer characteristics of individual customers from the script they used. The query was the unit of evaluation, and the groups described ways of asking. One person can use several of those ways on different occasions. Treating a transliterated request as evidence of a fixed customer identity would have added an unnecessary and unsupported inference. The practical question was whether the catalogue could respond usefully to the request that had actually been made.

## Make the regression part of the release choice

Product colleagues had a reasonable case for proceeding. The candidate returned more acceptable results overall, and a released system could be improved through subsequent work. Holding it meant deferring benefits visible in other groups. I did not want the weaker transliteration result to become a rhetorical device that erased those benefits. The alternative deserved to remain on the page, with its likely advantages and the uncertainty around live behaviour stated plainly.

Omar's concern was that an important path to regional-language books would become less useful without a clear reason being offered to the reader. The existing system answered 144 of the 200 transliterated queries acceptably; the candidate answered 116. We could not explain that loss away by pointing to English-title gains. We recommended retaining the existing route while revising the candidate's handling of those requests. The accountable product and technology owners accepted the trade-off, including the improvements deferred elsewhere.

Our evaluation cost ₹18 lakh. That cost had been incurred whether we released the candidate or held it. It would be misleading to describe the decision as costless caution, just as it would be misleading to treat the stronger overall score as measured additional revenue. We had evidence about reviewed query results, not conversion, repeat visits or purchasing. Those outcomes might matter to a subsequent evaluation, but they should not be supplied retrospectively as if the original review had already established them.

I also recognised a weakness in my own preparation. We had specified the query groups and acceptable-result criteria, but had not made the treatment of a meaningful subgroup regression sufficiently explicit before the scores arrived. That left room for colleagues to think the decision standard had changed after a winner became visible. In a later review, I would agree which kinds of regression require additional investigation and who can accept them. A clear decision process protects the team from both premature release and an endlessly moving standard.

## Keep language coverage an ongoing responsibility

No single evaluation can certify that a catalogue understands every way a reader may ask. New titles, spelling habits, transliteration patterns and mixed descriptions continue to appear. A completed review therefore needs an owner for future evidence, not a permanent badge of language competence. The purpose of an evaluation is to support a decision within a defined scope and to identify what should be examined next. It should not close inquiry simply because its results fit on one chart.

The next iteration also needs fresh judgement about overfitting to the review set. Improving the exact 200 transliterated examples would be useful only if the method addressed a broader class of requests rather than memorising the exercise. That does not mean the original examples become irrelevant. They can remain regression checks while a separate set tests whether the improvement travels. The distinction should be planned and documented so a better score has a credible meaning beyond familiarity with yesterday's questions.

At board level, I would ask management how aggregate measures distribute benefits and losses across meaningful uses. The answer need not require directors to adjudicate individual search results. It should reveal whether the organisation knows which users or tasks may become worse served, whether the evidence supports that conclusion and who accepted the trade-off. A simple overall measure can remain useful when it is accompanied by those questions; it becomes misleading when it substitutes for them.

The lesson I carry from the search review is not that every regression must prevent every release. Different circumstances can justify a different decision. It is that the preferred average cannot decide which deterioration is acceptable on the organisation's behalf. The data team should make the trade-off legible, the responsible leaders should own it, and the next review should test what remains uncertain. That is a more durable contribution than announcing a winner whose stronger score has hidden the route by which another reader became harder to serve.