A New Reality: Multiple Sources of Truth

When translating incrementality measurement data into action, it is important to consider the nature of decision making with two sources of truth, incrementality and last touch, and the natures of those differing methodologies. For the most part, incrementality metrics can be used exactly how last touch metrics are used both strategically and operationally. Polaris can display incrementality and last touch metrics side by side to make this process much easier.

For example, if the day-to-day operational process dictates that campaigns with day 7 ROAS of less than 50% should have budgets reduced by 25%, incremental day 7 ROAS can be used in the same process as a drop-in replacement. This applies to most marketing processes. Simply put, ideally the marketing process remains unchanged while the input into the process is changed from last touch to incrementality.

Significant Optimization Swings

When large discrepancies exist between the two sources of truth, it's possible that the marketing process will dictate a very large swing in budget. Imagine that last touch data, the current input into the marketing process, claims that a channel is performing poorly, while incrementality data claims the opposite. Using incrementality data as a drop-in replacement could cause a significant increase in budget.

Use Experiments For Truth

Polaris automatically recommends experiments, which are prioritized based on the statistical information gain to risk ratio. Part of the information gain score is influenced by the magnitude of the discrepancy between incrementality metrics and last touch.



For a specific example, if incremental day 7 ROAS is 0% while last touch is reporting 100%, the operational process might dictate a full pause if the incrementality metric is trusted. Before pausing, many Polaris users will execute an experiment, which acts in this scenario as a “test pause” in order to obtain ground truth. If the experiment confirms the 0% incremental day 7 ROAS, you can pause permanently with confidence. Otherwise, you can execute whatever action is dictated by the operational process based on the true day 7 ROAS found, which will be reflected in Polaris' Analytics page immediately after experiment results are computed.

Use Gradual Optimization For Risk Mitigation and Learning

Sometimes, it's not practical to run an experiment, especially when there are a large number of channels and the model is relatively uncertain (large confidence intervals) due to few historical experiments thus far. In those cases, gradual optimization can both mitigate risk and help the MMM learn without pausing traffic.

Gradual optimization refers to placing caps on optimizations within set time windows. For example, there may be a cap on budget increases of 20% every 3 days. That means that the budget of any channel cannot increase more than 20% in a 3 day period. Many marketing processes already contain caps to mitigate risk even in last touch based optimization.

Use Confidence Intervals to Influence Caps

It can be useful to utilize confidence intervals to influence the caps. If the business has set the default budget increase cap to 20% every 3 days based on strategic risk tolerance, a very wide confidence interval (indicating low confidence) for a particular channel may decrease the cap to 10%, while a very small confidence interval (perhaps due to a ground truth experiment) may remove the cap altogether.

Increase Model Accuracy Without Pauses

Furthermore, these gradual budget shifts help decorrelate marketing metrics like media spend and impressions across traffic, which in turn increases the accuracy and confidence of the MMM as a whole. Imagine the following realistic scenario.

The MMM estimates poor performance for a certain channel, but with low confidence.
Budget is gradually decreased over a period of 1 week.
The MMM finds that the slow spend decrease was correlated during that week with an overall decrease in country level metrics in countries where spend was most significant.
The MMM reasonably rules out other potential causes for the country level metric decreases during that week and allocates more incrementality to that channel.
Incremental metrics for that channel increase with high confidence, causing budget increases.
Incremental metrics change for other channels as well, also with higher confidence.

Updated on: 18/08/2022