Skip to website
Arman ZelhaviChief Technology Officer · Essays & perspective
← All essays

Firmware recovery decisions5 min read

The update that must survive losing its next instruction

An interrupted update exposes a product and service decision. The recovery path, component cost and acceptance authority belong in the same discussion.

A firmware update is often demonstrated as a sequence with a beginning and an end. The new version arrives, the device restarts and the feature works. That is necessary evidence, but it leaves an important question unanswered: what happens if the device never reaches the next instruction? Power can disappear during the transition, and the installed unit may be far from the engineer who understands it. A CTO should make that interrupted state part of the design discussion before the successful demonstration becomes the basis of a product promise.

In one Pelvori project, seventeen of 200 scheduled power-interruption trials left a data recorder needing bench recovery. The ordinary update sequence had passed. The fault did not arise from a customer field incident; our validation exercise found it before acceptance. That timing gave us a choice. We could revisit the design and its cost while the manufacturer still had authority to decide, or allow the normal demonstration to dominate the release conversation and leave the recovery obligation poorly understood. I supported reopening the design, with the consequence made explicit.

## Describe the state left behind by an interruption

The engineering team needs to explain the relevant states in terms that support a decision. Which image is usable before the transition, what is being changed and how will the device recognise an incomplete attempt? What remains available after restart? The detail belongs with engineers, but the consequence should be understandable to the product owner. A report saying only that an update failed may conceal the difference between a retry the user can perform and a unit that must return to a service bench. Those outcomes have very different implications for the product's support model.

Our proposed response retained an earlier usable software image while the replacement was prepared and checked. That required additional memory and a clearly defined transition. I do not present this as a mechanism that should be copied into every device. Different hardware, software rights, capacity and operating requirements may demand another approach. The important discipline was to examine the interrupted condition and choose a recovery path that the manufacturer could accept. A design borrowed without its assumptions can create a new failure mode while appearing to solve a familiar one.

The test plan should examine meaningful interruption points rather than collect a large number of easy repetitions. The relevant question is which states and transitions the exercise covers, and why those conditions matter to the intended use. Engineers should also document the configuration so that a later result can be compared sensibly. A change in memory layout, update procedure or supporting component may alter the meaning of a previous pass. The board does not need the complete test script, but management should be able to explain why the evidence remains relevant to the release being considered.

## Bring the service model into the design review

The revised design added ₹180 to the client's estimated per-device component cost and required ₹48 lakh in our engineering work. The manufacturer understandably challenged both figures. A component increase repeats across units, while an engineering cost is incurred on a different basis. We kept those categories separate and discussed the recovery expectation alongside them. A cheaper device may still be the right decision, but it should not be selected on the assumption that physical recovery will be easy when the support arrangement has never provided for that work.

This is where commercial and engineering colleagues need one conversation. A technical team may view recovery as essential, while a product manager sees a cost increase against an already agreed price. Both perspectives are legitimate. The decision should consider where devices will operate, who can service them and what the manufacturer is prepared to promise. Our role was to provide evidence and an engineering option, not to assume the client's acceptance authority. Keeping that boundary clear made the disagreement more productive than presenting the redesign as a technical instruction the customer had no choice but to follow.

The handover moved by four weeks. That delay belongs in the account even though the redesign was justified. Other work depended on the planned date, and the manufacturer needed time to assess the change. I would not describe the intervention as a costless act of care or blame commercial colleagues for asking difficult questions. The better lesson is to raise interrupted-operation behaviour early enough that component selection, service planning and validation can be considered together. A late discovery may be preferable to a field failure, but it still reveals an opportunity to improve the development sequence.

## Keep the test result attached to its conditions

After the redesign, none of the repeated 200 scheduled trials required bench recovery. That was a useful result under the stated configuration and interruption matrix. It did not show that every future update would succeed, that every field condition had been represented or that the complete product had received a wider certification. We retained the scope with the result so that a shorter commercial summary would not accidentally broaden it. The manufacturer continued its own acceptance work, including considerations outside the software experiment we had performed.

A clean rerun should not erase the original failure. The earlier evidence explains why the design changed and what risk the change was intended to address. Keeping it also helps a future team understand which assumptions should be revisited if the product evolves. A record containing only the final pass can make a recovery mechanism look like unnecessary complexity to someone trying to reduce memory or simplify the update path later. The explanation of the original decision is therefore part of maintainability, not an embarrassing appendix to hide once the immediate issue is resolved.

Directors overseeing a material product programme should ask how the company knows that interrupted transitions have an acceptable outcome. They should connect the answer with the service commitment, the cost and the party authorised to accept the residual uncertainty. They should not choose a memory layout or prescribe a boot process themselves. The useful oversight question is whether management has examined the relevant condition and made the trade-off explicit. A demonstration that works once is not the same evidence as a recovery path tested against the conditions the company expects to support.

I would like the next project to make this conversation ordinary. The product requirements should describe what must remain possible after interruption, the plan should fund evidence for that behaviour and the release account should carry any remaining limit. That is a more durable improvement than celebrating one successful redesign. It gives engineers a reason to question the happy path, commercial colleagues a basis for explaining the cost and the manufacturer a decision it can own. The product becomes easier to trust because the organisation has thought about the moment when the expected sequence stops.