The weeks we switched off television told us more than our attribution model

I believe a large marketing budget should be tested by switching part of it off, under conditions agreed with finance in advance, rather than defended with an attribution model that cannot be shown to be wrong. Attribution models describe the path a customer took before arriving. They are useful for that. They are much weaker at describing what caused the decision, and in telecommunications, where a customer may think about changing operator for months before acting, the difference can move billions of yen in the wrong direction.

My position is not a rejection of models or of the people who build them. It is a view about what kind of evidence a chief marketing officer should bring to a board when asking for money. A claim that a channel works should be able to fail. If nothing we could observe would cause us to reduce a budget, we are not measuring it. We are describing it, and the description will tend to favour whoever wrote it.

What the model could see and what it could not

By 2023 Hinoura Communications spent ¥24 billion a year on consumer media, 62% of it on television. Our attribution model, built with our media agency, credited paid search with 44% of the customers who joined us from another operator and television with 11%. The finance director asked a reasonable question: if search was doing four times the work of television at a fraction of the cost, why were we not moving the money?

The honest answer was that I did not believe the model, and that my disbelief was not evidence. Search is usually the last thing a customer does before acting. Someone who has seen our advertising for weeks, heard a neighbour mention our price and decided to switch will often type our name into a search engine and click the first result. The model sees the click. It does not see the weeks. It therefore credits the step that was easiest to observe, and it rewards the channel that sits closest to the transaction.

Japanese mobile switching makes the problem sharper. Many customers change operator in the spring, when students start university, graduates start work and families rearrange their contracts together. The decision often forms in conversations at home rather than on a screen. Older customers, in particular, are more likely to have seen a television advertisement and walked into a shop than to have clicked on anything. A model built mainly from digital signals cannot see that customer at all. It does not count them as a failure of television. It simply does not count them.

Agreeing in advance what would move the money

Our head of marketing science proposed a withholding test. We would stop television advertising in three prefectures for eight weeks and compare switching, shop visits and search behaviour with three prefectures matched on population, age profile and recent switching rates. We would read the result by age group, because that was where we suspected the model was most wrong.

The objections were good ones. Our media agency argued that eight weeks was too short for television's slower effects to appear. Regional sales directors argued that their areas would lose customers to competitors who were still advertising, and that their own performance would be judged on the loss. We addressed the second objection by adjusting their objectives for the period, and the first by accepting that our result would understate any long-term effect. We also moved the test to September through November, away from the spring season, because losing customers in the busiest weeks of the year would have been both more expensive and harder to interpret.

The most important step was taken before the test began. Finance and marketing agreed in writing what result would move money. If television's measured effect on switching was less than half of what the model credited, we would move a quarter of the television budget to other channels. If it exceeded what the model credited, we would stop treating search as the main driver in our planning. Without that agreement, any result could have been explained away by whichever side it disappointed. With it, the test was a decision waiting for evidence rather than evidence waiting for a decision.

The cost of knowing

The test cost us an estimated 6,800 customers who would otherwise have joined during the period, roughly ¥330 million of first-year revenue. I put that figure in the board paper before the test and again after it. A measurement programme that hides its own cost invites the suspicion that it is a marketing department's way of protecting its budget, and that suspicion is often justified.

The results were mixed in a useful way. For customers aged fifty and over, television's effect on switching was about 1.9 times what the model had credited. For customers under thirty we could not distinguish television's effect from zero. When television stopped, search volume among older customers fell by about a third, which suggested how much of search's credit had been borrowed from the advertising that preceded it. For the middle age group the result was uncertain, partly because one of our control prefectures saw an unusual competitor promotion during the period.

We moved ¥3.6 billion. Some went from national television slots that skewed young into regional television and staffed shop events for older customers, and some went into online video for younger audiences. Neither the agency's position nor the finance director's original instinct was fully confirmed, which is usually a sign that a test was worth running.

The test also had limits that I state whenever I describe it. It covered one season, three regions and eight weeks. It measured switching, not long-term preference for our brand. It could not tell us how competitors would respond to a permanent change in our spending. If I were starting again, I would run it for twelve weeks and in more regions.

We now run at least one withholding test a year on a large budget line, and the board receives the confidence interval with the central estimate. I recommend the practice to any marketing leader whose largest budget is defended mainly by a model. Choose the line you believe in most. Agree with finance what result would cause you to reduce it. Then switch it off somewhere for long enough to find out. If you cannot bring yourself to do that, it is worth asking whether you believe the budget works or merely hope it does.