A test count cannot tell you what was left unasked

A large number of findings can look like a productive engagement. It is easy to put on a slide, compare with the previous month and use in a recognition discussion. Yet the number says little about the questions that were never investigated. It may reflect careful work, a poorly prepared system, many minor observations or a team that has learnt which activities earn recognition. Before I ask whether a testing team found enough, I want to understand what it was trying to establish and which uncertainties matter to the decision ahead.

Begin with the question the work must answer

At Belsori, a review of nine engagements exposed a problem with the count-based component of our recognition system. Teams could accumulate numerous minor findings while receiving little recognition for spending days on a difficult failure path. The count was a record of observations, but we had allowed it to stand for the value of the investigation. That was a management choice. It would have been unfair to criticise people for responding to a measure the company had explicitly rewarded.

We compared the engagement questions with the evidence in the reports. In some cases, important agreed risks had not been examined deeply enough despite impressive totals. The review was bounded. It did not establish that all our work had the same weakness, nor did it tell us that a particular release would fail. It gave us enough evidence to question an incentive that was influencing attention in the wrong direction. We needed to change the decision process without inventing a larger conclusion than the review supported.

The first step was to make the intended question visible. A client may need to know how a transaction behaves after interruption, whether a particular set of permissions holds under specified conditions, or whether an earlier finding has been adequately retested. Those questions require different evidence. A single count cannot explain their coverage. The team should be able to describe the conditions examined and the important paths remaining outside the work, so the client can judge the relevance of the conclusion.

That description must be agreed early enough to influence planning. If risk coverage is written only after the testing has finished, it can become a polished explanation of whatever happened to be done. I want engagement managers to discuss consequential questions before they allocate the team and before a commercial estimate hardens into a promise. Later changes can then be understood as changes to the work, rather than as an inconvenient discovery that the original plan did not answer the client's actual concern.

Make the cost of better judgement visible

Some managers defended the count because it appeared objective. Their concern was practical: a more qualitative review could be inconsistent, take longer and leave teams uncertain about how recognition would be decided. I agreed that those were real costs. Removing a poor measure does not automatically produce a good replacement. We therefore asked technical reviewers and delivery leaders to define the evidence they would examine, while keeping examples of actual work available for discussion.

We replaced the count-based component with a review of agreed risk coverage, supporting evidence and completed retesting. This did not create a universal score that could compare every team effortlessly. It required managers to understand the engagement and explain their judgement. I approved time for that work. Asking for a more demanding review while leaving every workload assumption unchanged would simply move the cost into evenings and hurried decisions, undermining the purpose of the change.

I also accepted the loss of a simple league table. Leadership teams often say they want nuance while continuing to demand one number for every comparison. That tension eventually falls on the people preparing the reports. If a measure cannot bear the interpretation we place on it, the answer is not a more elaborate presentation. We may need a different conversation, supported by a smaller number of well-chosen records and a clear explanation of the judgement being made.

The replacement still needs scrutiny. Teams could select easy risks, write persuasive narratives or treat completed retesting as a box to tick. No recognition system removes the need for managers to look at the work. We retained periodic reviews of actual engagement evidence and asked whether the criteria were directing attention usefully. The purpose was not to find a perfect metric that would make leadership unnecessary. It was to reduce a known distortion and improve the quality of the discussion.

Protect the conclusion from the commercial hierarchy

The same issue appears at the end of an engagement when a sponsor wants a stronger conclusion than the work supports. A team can have done its job well and still be asked to turn a scoped report into a broad endorsement. If senior leaders reward only a smooth account relationship, the written testing standard may offer little protection. The company has to make it possible for a qualified result to survive the final commercial conversation.

In one engagement, a client linked a ₹42 lakh milestone to a release endorsement beyond our examined conditions. I had sponsored the account, which meant my own influence could discourage challenge even without an explicit instruction. I moved the commercial discussion to another executive and left the technical conclusion with the specialists. We retained the qualifications and made the exclusions useful to the client's decision, rather than presenting them as distant fine print.

Payment arrived seven weeks later. That was a real working-capital consequence, and the board saw it. We did not claim to have prevented a failure, because the client retained its release decision and we could not know the counterfactual. The evidence supported a narrower statement: we had refused to represent unexamined conditions as examined and had accepted a delayed payment while the contracted deliverable was discussed. That is enough to explain the decision without making it heroic.

For a board, these examples raise questions about how conclusions are produced. Which incentives shape the evidence management receives? Who can challenge a reassuring report? Can an unresolved finding remain visible when an important customer or senior executive wants closure? Directors do not need to become test managers to ask those questions. They need confidence that the company's reporting process does not systematically remove the facts most likely to interrupt a convenient narrative.

My own practice is to ask for the limits alongside the result. What was left unasked, what remains unresolved, and what decision can reasonably be supported now? Those questions make a report more useful rather than less confident. A testing company earns trust by being precise about its contribution. The count can remain a useful operational fact, but it should never be allowed to answer a question it was not designed to answer.