A clean rerun should invite a different challenge

When engineers repair a fault, the first obligation is to show that the correction addresses the conditions that exposed it. A clean rerun matters. It gives the team evidence that the known failure no longer occurs under the recorded circumstances. The mistake is to let that result answer a larger question without further thought. A parser that handles the messages used to diagnose it may still fail on another malformed input. The next challenge should examine something beyond the examples that have already shaped the repair, rather than merely repeat a reassuring result.

At Pelvori, a device-message parser failed on eighty-four cases in a laboratory set of 12,000 malformed messages. The engineering team corrected the identified mechanisms and passed the set on rerun. A project review proposed closing the issue. We then used a separately prepared holdout of 4,000 messages, which exposed two additional failure cases. That did not make the first repair worthless. It showed that the two exercises answered different questions. The result gave us a chance to improve the implementation before the client accepted the software for its wider product programme.

## Separate repair evidence from wider confidence

A regression set preserves a known obligation. It should continue to show that a correction has not been lost during later changes. Its strength comes partly from knowing why each case matters. A broader challenge has another purpose: it tests assumptions not fully represented by the examples already understood. The distinction is practical, not a claim that one type of test is superior. A development team needs both a memory of known faults and a way to discover conditions that memory does not contain. Confusing the two can produce either repeated surprises or an ever-growing set with no clear rationale.

The report should say what constitutes failure. A malformed message might be rejected cleanly, consume excessive resources, interrupt another function or cause an unexpected restart. Those outcomes should not be treated as interchangeable simply because the input was invalid. The expected behaviour must be agreed for the interface and product context. In our exercise, the team recorded the observed failure paths and the rejection behaviour required after correction. This gave the client a more useful account than a single pass percentage whose meaning depended on assumptions hidden in the test implementation.

It is also necessary to state what the exercise does not cover. A message-parser test is not a complete security assessment of a device, and a clean input set does not demonstrate every possible operating condition. Other interfaces, resources and deployment arrangements may matter. I want engineers to be confident about the evidence they have produced without being asked to turn it into a universal assurance. That confidence is easier to maintain when the scope is explicit from the beginning, rather than added as a defensive qualification after a customer has heard a much broader promise.

## Protect the usefulness of the second test

A holdout needs enough independence to challenge the implementation rather than echo the developer's recent understanding. That can involve different preparation, different cases or a separate reviewer, depending on the project. The objective is not to conceal arbitrary surprises from the engineering team or create a contest between development and validation. It is to avoid using the same assumptions twice and calling the second result independent evidence. The team should understand the intended coverage and expected behaviour even when it has not seen every individual input in advance.

When the two additional cases appeared, the commercial lead was concerned about another delay. The project had already reported a clean rerun, and the client expected progress towards acceptance. I supported reopening the item and explaining the distinction between the sets. A further ₹31 lakh engineering and validation allocation funded three weeks of correction and testing. That cost should be visible. Discovering a weakness is useful, but it still consumes time and resources; treating the expense honestly helps the business decide how to plan similar evidence earlier in future projects.

There is a risk that teams learn to avoid difficult tests when every discovery is treated as a personal failure or an unwelcome disruption. A CTO should resist that incentive without excusing poor preparation. The question is whether the exercise was appropriately designed, whether the team acted on the finding and whether the programme had allowed time for a plausible result. If a test is only acceptable when it passes, it has become a presentation device. Our response was to preserve the finding, repair the new paths and keep the client acceptance owner informed of the revised evidence and schedule.

## Report the path to the final result

The revised parser passed both recorded sets under the laboratory configuration. We retained the earlier failures, their causes and the corrections in the engineering account. A future reviewer could therefore understand what the final status meant and why particular rejection behaviour existed. This history also supported subsequent regression work. Removing the failed runs from the account would have made the report tidier but less useful to the next team, especially if a later change recreated a condition that the final implementation had been designed to handle.

A summary should not imply that 16,000 messages represent all possible inputs. Nor should it add the two sets together without explaining their different roles. The original 12,000 cases guided and verified the first correction; the 4,000-message holdout challenged it and exposed another weakness. Both contributed to the final confidence, but their value lay in their design and the response to their findings, not simply their combined size. I prefer a report that a product owner can question intelligently to a large test count that discourages discussion through apparent precision.

For directors, the relevant issue is how management reaches confidence in a consequential technical claim. They can ask whether a clean result has been challenged independently, how new failures are reported and who decides that the remaining evidence is sufficient for the company's obligation. They do not need to inspect every malformed message. They should also understand the boundary between the supplier's software evidence and the manufacturer's complete-product acceptance. A board that rewards only uninterrupted progress may unintentionally discourage the discovery it most needs management to make before a release commitment becomes difficult to change.

The improvement I would carry forward is to plan the second challenge before the first repair is celebrated. That creates a more honest expectation of what closure requires and gives commercial colleagues a clearer basis for scheduling. It also makes the validation team a partner in the product decision rather than a late obstacle. A clean rerun should produce justified confidence in the correction, followed by curiosity about what has not yet been examined. That sequence is more useful than either distrust of every test or satisfaction with the first result that allows the programme to move on.