Systems that keep working when the highway floods

In February 2022 a flood closed the road to our Cairns depot for three days. That depot housed the recovery site for the corporate data centre in our head office, eleven kilometres away, and its generator relied on fuel deliveries that could not get through. Nothing failed. The episode still showed that a single cyclone or flood could disable both sites in the very week that mine operators and community councils most needed information from us.

I proposed moving corporate recovery to a commercial data centre in Brisbane, at a capital cost of A$3.2 million and A$0.8 million a year. The chief financial officer questioned the expense when the existing arrangement had never been needed. The executive general manager of operations objected for a different reason: if station monitoring failed over to Brisbane, the operators supervising forty-six stations would depend on long-distance links that a cyclone could also cut. Both arguments were sound.

We divided the decision. Corporate and commercial systems would recover in Brisbane. The backup operations centre stayed in our Darwin office, with a local copy of station-monitoring data so operators could keep working if the southern links failed. To stay within the approved budget, the finance and payroll systems moved from a four-hour to a twenty-four-hour recovery objective, which the finance team accepted reluctantly.

The first full test, in November 2023, took five hours and forty minutes against a four-hour objective. Call-routing scripts still pointed at the old site, and nobody from the customer desk had taken part. We corrected both. In March 2025 a flood cut head office's fibre links for thirty-one hours, and customer notices and invoicing ran from Brisbane throughout.

Recovery tests now include customer-desk and base staff, and the board receives measured recovery times rather than design objectives. I would bring the customer desk into the design from the first workshop. Leaving them out cost us a failed first test and some credibility with the committee.

In 2023 our technicians at six of fourteen regional bases still received job packs on paper and reported completion by radio or at the end of a shift. The supplier's standard mobile application assumed a constant connection. Our review of two years of work orders found 44 per cent of field jobs, by count, at sites without reliable coverage, with the share rising sharply in any week with a cyclone.

The field services manager wanted the standard product in place before the 2023–24 wet season, as planned. Procurement pointed out that configuration beyond the standard product would add A$1.6 million and that the supplier would not commit to a delivery date. I recommended waiting. An application that failed where crews most needed it would be abandoned, and paper would return with less discipline than before.

We required the application to hold a crew's whole work list for seventy-two hours, store photographs and readings on the device, and send them through satellite terminals fitted to 160 vehicles whenever no mobile signal existed. We removed the live crew-location map from the first release to keep within the A$1.6 million. Crew leaders from Mount Isa, Katherine and Kununurra wrote the rules for settling conflicting records.

Rollout began eight months late, in June 2024, and crews worked one more wet season on paper. By June 2026, 380 field staff used the application. Same-day job closure rose from 54 to 89 per cent, and records needing manual reconciliation fell from about 640 a month to 90. Battery life was the first complaint, and we bought vehicle chargers we had not budgeted for.

The company now tests field software in the worst coverage it serves rather than the best. I would run a pilot at two remote bases before procurement instead of after it. That would have shortened the argument and exposed the battery problem a year sooner.

In 2024 our contract with Pellacombe Managed Services came up for renewal. It covered hosting, networks, the service desk and application support for our 2004 contract-invoicing system, at about A$7.5 million a year. Invoicing support depended on three Pellacombe engineers. Each July, when our supply contracts index their prices, loading the new rates took nine weeks. In 2023 two configuration errors produced incorrect invoices for six sites, worth A$2.6 million in total, which we corrected with credit notes and explained to each customer.

I recommended taking invoicing support out of the contract and building a team of five inside Ollanvale. The chief financial officer opposed it. The change added about A$0.6 million a year, and Pellacombe offered A$0.9 million off a three-year renewal if the whole scope stayed together. The commercial director supported me because her team had to explain the errors to mining customers. The chief executive decided in favour of the split.

Pellacombe kept hosting, networks and the service desk, and we gave up its discount. Two of its invoicing engineers chose to join us, with retention payments and their employer's agreement, and we recruited three more. Together we documented 186 contract-pricing rules, 23 of which nobody could fully explain at first.

The July 2025 indexation round took four weeks. No configuration error reached a customer's invoice in the following twelve months. One of the engineers who joined from Pellacombe left after five months, which showed how thin our knowledge still was. We are also paying for overlapping staff longer than we hoped.

The company now treats knowledge of its pricing rules as something it owns and reviews each year, and the replacement business case starts from an accurate inventory. I would write staff-transfer terms into the original contract, years before renewal, so that the decision does not depend on goodwill at the end of an agreement.