Our plants are only as closed as our maintenance contracts

Ask a plant director in a Chinese machinery company whether the production network is connected to the internet, and the answer is usually no. In the sense intended, it is generally true. The robot controllers, PLCs and test benches on our lines do not browse websites or receive email. But a plant network is visited constantly. Robot integrators commission new cells. Machine-tool builders diagnose faults from their own offices, some of them abroad. Paint-line suppliers update recipes. Each visit opens a path, and in my experience it is those paths, not the corporate network, that decide how closed a plant really is.

I first learnt this as a junior engineer in 1999, when a machine builder's engineer dialled into a machining line over a modem, loaded a program written for another customer and stopped the line for two days. I learnt it again in November 2023, when an integrator's laptop carried ransomware into the welding shop of our Hefei loader plant. We stopped the plant for 58 hours, deferred RMB 96 million of revenue and discovered that 31 robot programs had no current backup. The integrator was competent and cooperative. Its contract simply said nothing about how its engineers' equipment should connect to ours, or about who held copies of the programs it wrote.

Where the outside world actually enters

After the incident we counted every outside party with any form of access to equipment in our eight plants. There were 52 vendors and integrators. Fourteen had permanent remote connections, some installed years earlier by the vendor and never reviewed. Nine reached our machines from outside China, which turned routine maintenance into a cross-border data flow our legal team had not assessed. Twenty-two brought their own laptops on site. Only six had a contractual duty to tell us about a security incident in their own systems.

None of these arrangements was reckless when it was made. A remote connection lets a builder in another time zone diagnose a fault overnight instead of flying an engineer in. An integrator's laptop holds the tools and licences needed to program a robot. The difficulty is that each arrangement was negotiated by a plant maintenance team as a service question, priced as a service question and never examined as a security one. Each plant network was therefore exactly as closed as the most permissive of its separate negotiations.

Technical controls matter, and we have spent RMB 78 million on them: zones by production line, a monitored boundary between each plant and the corporate network, a brokered service through which every remote session is requested, approved, recorded and ended, and scanning stations for any portable storage brought on site. But controls installed by my team can be undone by a contract signed at a plant. Durable protection has to be written where the relationship is defined.

Put access and backups into the contract

Our standard terms for equipment vendors and integrators now contain five obligations. Remote access is allowed only through our brokered service, for a stated task and a stated period. Equipment brought on site must pass our scanning station, or the vendor uses machines we provide. Every program written or changed for our equipment must be delivered to us, in a form we can restore without the vendor, within five working days of commissioning. The vendor must tell us within a day if a security incident in its own systems could affect our equipment. And access from outside China is permitted only for named purposes that our legal team has classified, so that a diagnostic session does not carry production data abroad without a basis.

Vendors pushed back. Two foreign machine-tool builders said their diagnostic tools needed a permanent connection; one accepted brokered sessions after six months, and the other's machines are now serviced on site, at a higher cost. Several integrators priced the backup obligation into their quotations. Plant maintenance teams lost some speed, because a session that once opened instantly now waits for an approval. I accept those costs. They are smaller than 58 hours of a stopped plant.

The backup obligation has proved the most valuable of the five. In its first year it brought in about 1,300 programs that had existed only on integrators' laptops, including programs for welding cells commissioned before 2020 whose original engineers had since left their firms. Had any of those cells failed, we would have been re-teaching robots from drawings while the line stood idle.

China's multi-level protection scheme sets baseline expectations for classified systems, including industrial control systems, and our plants are assessed against it. Meeting it is necessary. It describes a plant's own controls far better than it describes the obligations of the companies that visit, and the visitors are where our exposure was greatest.

The register I now show the board

Boards usually hear about plant security as an annual statement that controls are in place. I now give ours a register instead. For each plant it shows the number of outside parties with access, the sessions they opened in the quarter, how many came from outside China, the share of robot and controller programs with a verified offline copy, and the date and result of the latest restoration test. Each plant director signs the page for that plant. The register currently shows 4,100 programs with verified copies across eight plants and no remaining permanent remote connections.

The register has changed conversations I did not expect it to change. Procurement now asks for a vendor's access terms before its price. Plant directors compare their restoration times with one another, which my team could never have prompted. And the strategy committee asked, for the first time, which of our vendors are themselves dependent on a single engineer who holds all our programs on one laptop.

For directors of any manufacturing company, the useful questions are short. How many outside companies can reach your production equipment, and by what route? Who holds copies of the programs that run it? What does each vendor's contract oblige it to do if it is compromised? If management answers the first question with a network diagram and cannot answer the third, the plant is protected by luck and good will, and both eventually run out.