Count the factory AI pilots you stopped

Annual reports from Chinese manufacturers now carry a familiar paragraph about artificial intelligence. It lists smart-factory designations, the number of AI applications deployed and sometimes the number of models trained. Ruoshan's report has carried such a paragraph, and I helped to write it. I now think it describes our effort and says very little about our results, and that a board reading it learns almost nothing about whether the money was well spent.

The alternative I propose is simple to state. Report every pilot, including the ones that were stopped, with the condition under which each would be stopped, the result it produced and the operating measure it changed. A count of applications deployed rewards starting. A count of pilots stopped, with reasons, rewards judgement.

Eleven pilots, three survivors

Since 2022 we have run eleven AI pilots in our plants, our parts business and our service organisation, at a combined cost of RMB 48 million. Three have been scaled. A model at our Huai'an structural plant reads current and voltage traces from welding robots and flags boom and arm welds for ultrasonic inspection; it cut weld rework by 18 per cent in its first full year and now runs on 64 robots. A parts-demand forecast for overseas dealers, built on machine hours from telematics, helped raise first-pass fill from our hubs. And a service assistant for dealer technicians, once we restricted it to answers it could cite from our manuals, reduced hotline calls by 19 per cent in 2025.

Two pilots survive as narrower tools than intended. A scheduling optimiser for the small-excavator line in Changzhou was meant to sequence the whole line; planners now use it only to propose an order for the paint shop, where colour changes dominate cost, and they override it about a third of the time. An energy model for compressed air earned its place by finding leaks rather than by predicting demand.

None of the survivors looks like the pilot first proposed. The weld model began as a plan to pass or fail welds outright, and it became useful only when we accepted that it should direct inspectors to the welds worth checking and leave the verdict to them.

Six were stopped. They included a paint-film thickness predictor whose sensors drifted faster than the model could be recalibrated, a failure predictor for hydraulic pump test benches that had recorded only nine failures in three years, a camera check for missing fasteners that duplicated a cheaper torque-tool interlock, and a language-model assistant for plant engineers that answered questions about work instructions confidently and, too often, from the wrong revision. Together the six cost RMB 19 million.

What the stopped pilots taught us

The failures were not random. Most fell into three kinds, and each has changed how we choose pilots.

The first kind failed for lack of events. A model cannot learn to predict a failure that happens nine times in three years, however many sensors watch the machine. Before funding a predictive pilot we now ask how many examples of the outcome exist in the data. If the answer is a handful, we fund better condition monitoring instead and revisit the idea when the history is longer.

The second kind failed because a cheaper control already existed or could be built. The fastener camera worked tolerably well, but a torque tool that refuses to release a part until every bolt reaches its value does the same job with no model at all. A pilot should be compared with the best simple alternative, not with doing nothing, and that comparison belongs in the proposal rather than in the post-mortem.

The third kind failed on the quality of our own documents. The plant-engineering assistant and the first version of the service assistant both reproduced the contradictions in our work instructions and service bulletins. Fixing those documents was worth doing with or without AI, and in the service case it turned a failing pilot into a useful one. In the plant case it did not, because our work instructions change too often for any tool to keep pace, and we stopped.

I would add a fourth lesson, about time. Every pilot we stopped ran longer than it should have, on average about five months beyond the point at which the evidence was clear. Teams hoped one more data set or one more model would change the result. It seldom did. The extra months cost about RMB 6 million and, more seriously, kept capable engineers away from work that would have paid.

How I now report AI to the board

Each pilot at Ruoshan now begins with a one-page charter. It names the operating measure the pilot should move, the size of change that would justify scaling, the date by which that change must be visible, the condition that stops it and a business owner from the plant or function rather than from my team. Stopping a pilot on its own terms is recorded as a decision, not as a failure, and the people who ran it are not penalised for the result.

Twice a year I give the board's strategy committee a single table. Every pilot appears on it, with its cost to date, its status, the measure it was meant to move and the result. Stopped pilots stay on the table for a year after they end, with the reason. The owner of each surviving model also states once a year what would happen if we switched it off tomorrow, and two owners have found that harder to answer than they expected. The table shows, for instance, that the weld model's 18 per cent fall in rework is worth about RMB 9 million a year, and that the six stopped pilots together cost more than twice that.

The table is less flattering than the paragraph in the annual report. Directors have found it more useful, because it lets them judge whether the company is learning to choose better pilots, which is the only AI capability that compounds. Our recent charters are shorter, our stop dates are earlier and fewer proposals reach funding, which I count as progress. A manufacturer that can tell its board which models it switched off, and why, is far more likely to be running the right ones.