Skip to website
Irfan Qevzami

The journal · Missing observations in analytical software

The blank plot became a zero before anyone agreed what absence meant

A default value made a map look complete while changing the population behind its average. We restored the distinction between measured values and cloud-masked plots, keeping coverage beside the observed result.

The view contained 120 equal-area plots. Seventy-two had valid observations with a mean vegetation index of 0.64. Forty-eight were cloud-masked and unobserved for the reviewed period. The processing path inserted zero for those missing values, then calculated a mean across all 120 plots. The displayed result was 0.384. The arithmetic followed the numbers it had been given, but those numbers included an assumption that changed the meaning of the result.

Zero could be a valid index value. It could not also serve silently as the absence of an observation. We spent ₹36 lakh correcting the processing, review and client coordination. The revised view retained the 0.64 mean for the 72 observed plots and showed 60% coverage, with the 48 cloud-masked plots identified separately. We did not infer the values of the missing plots, assume they resembled the observed ones or turn the correction into advice about crop yield or land condition.

A default can answer a question nobody authorised

Software often needs a representation for a missing value. The danger is not the existence of a representation but the loss of its meaning as the value moves through the system. A default that helps a display or calculation proceed can become indistinguishable from a measurement. The next operation then treats absence as evidence and produces an orderly answer to a question the available observations did not support.

In this case, filling the 48 plots with zero made the map and the aggregate easier to produce. It also asserted something about those plots that had not been observed. The resulting mean was not merely a less precise version of the observed mean. It described a different constructed population in which the unobserved plots had been assigned a particular value. That assumption needed an explicit basis, and the workflow had none.

The interface sponsor argued that blank regions would confuse users. A fully coloured map would be easier to read and would make the product appear more finished. I understood the concern about usability, but the proposed solution changed the evidence to improve its appearance. Scientific-methods colleagues wanted the validity state to survive into the view. I supported their position and asked the team to make absence understandable instead of making it disappear.

That distinction required work across processing and presentation. It would not have been enough to add a note to the screen while the aggregate still used substituted values. Nor would a corrected calculation have been sufficient if the display continued to colour missing plots as measured zero. The meaning had to remain consistent along the path, so the user could understand both the reported statistic and the area for which observations were unavailable.

Preserve the observed population

The 72 valid plots still supplied useful information. Correcting the missing-value treatment did not require discarding them or refusing to report any result. Their mean was 0.64 for that observed population. The coverage was 72 out of 120, or 60%, under the equal-area definition in this case. Presenting those facts together gave the user a clearer account than either a complete-looking aggregate or a vague warning that some data might be missing.

The observed mean did not establish the mean of the whole area. We had no basis to assume that the cloud-masked plots resembled the visible ones. Missing observations can have a relationship to the conditions being studied, and the software should not silently choose an assumption about that relationship. Any estimation beyond the observed population would need an appropriate method, evidence and clear statement of uncertainty, with qualified owners assessing its suitability.

The equal-area condition also matters. It makes the simple comparison of plot counts and coverage understandable in this fictional example. A collection of unequal regions would require attention to what was being weighted and why. I would not turn this case into a general instruction to average every spatial dataset by counting records. The engineering habit is to preserve the population and interpretation, not to copy one calculation into a setting whose units differ.

We kept the ₹36 lakh correction as an expense. The work improved the account of observation status, but it did not prove an agricultural outcome or quantify economic harm avoided. The company supplies software; domain conclusions and customer decisions require their own evidence. The public story should not become more confident than the corrected product merely because a leadership profile benefits from a dramatic claim.

There is a legitimate question about how much missingness detail belongs in the main view. A screen overloaded with technical flags can make a product difficult to use. The answer should come from the decisions the user needs to make. Coverage and observation status belong prominently where they change interpretation, while deeper provenance can remain available through an appropriate explanation. Usability should organise meaning rather than replace it with an easier number.

A complete-looking map is not complete evidence

Maps carry a visual authority that can exceed the evidence behind them. Continuous colour, precise boundaries and smooth interaction encourage the impression that the underlying knowledge is equally complete. Engineering teams need to recognise that effect when choosing defaults and display conventions. An honest gap may look less polished, but it can be the most important information the system has about part of the area.

The correction also changed the questions I ask during review. Where does observation validity enter the system? Which operations preserve it? Can an aggregation distinguish a measured zero from a missing value? Does the final view explain the population behind its statistic? These are not requests to make every interface verbose. They are checks that the product has not acquired confidence by losing information on the way to a convenient output.

For directors, the broader issue is whether management understands the limits of an analytical claim. A board should be able to ask what was observed, what was inferred and what remains unknown. It need not inspect raster processing code to examine whether those categories remain distinct in the product and the commercial promise. The same question applies when an elegant dashboard presents a single figure whose missing population has quietly disappeared.

I would also ask how the company responds when a plausible output is challenged. If the initial reaction is to defend the calculation alone, the team may miss the assumption that entered before the calculation began. Here, the arithmetic was not the difficult part. The difficult part was admitting that a practical default had made an unapproved statement about the world and then carrying that correction through both processing and presentation.

A missing observation is not an embarrassment to be coloured away. It is a fact about what the system can presently support. The observed plots can remain useful while the unobserved plots remain unknown. I want our software to make that combination intelligible, so a qualified user can choose the next step with a clear view of the evidence rather than inherit a decision the interface made silently.