Experiment Result Actionability
The ultimate goal of executing an experiment is to inform action resulting in business value so it’s important to understand how to convert the ground truth incrementality insights into action. The key to remember on this topic is that experiment results are automatically fed into the econometric model that outputs the incrementality metrics. Therefore, you simply need to utilize the incrementality metrics available in the Polaris dashboard on the Analytics and other screens as well as the API in your normal marketing process.
When the model is refreshed after an experiment is completed, the ground truth results are used to calibrate it. That process ensures that it is 100% accurate and 100% certain of the incrementality metrics for the treated traffic on the dates when the experiment was running. It also helps to significantly increase accuracy and certainty throughout the rest of the model even for campaigns and countries that were not treated.
Recalling a previous example, let’s say a significant discrepancy between last touch day 3 ROAS, provided by a last touch MMP, and incrementality day 3 ROAS, provided by Polaris, has been identified on a mission-critical campaign. Before actioning the marketing process based only on the incrementality version of the metric, which would result in a large budget decrease (day 3 ROAS incrementality is low, but last touch seems to be overvaluing), you decide to execute an experiment to find the truth.
At the end of the experiment, the treatment is reversed. Three days later, the ground truth incrementality is known for all cohort day 3 metrics including day 3 revenue, from which day 3 ROAS is derived. You visit the Analytics screen in Polaris and find that the day 3 ROAS for the campaign has been updated to reflect the results of the experiment. The econometric model was directionally accurate, but was off by a small margin. Last touch attribution was far overvaluing the campaign. The experiment-validated incrementality day 3 ROAS is used in the marketing process with confidence and the campaign’s budget is sharply decreased.
To summarize, the incrementality metrics derived from the experiment-calibrated model and surfaced in the Polaris dashboard and API, should be utilized as the primary influence for marketing action in the same way that last touch metrics are utilized in the same process. The only difference is that last touch attribution isn’t designed to measure incrementality, which is all that matters, provides no information about how uncertain it is, and usually offers no way to validate its results. Polaris only measures incrementality in a privacy-centric way, quantifies uncertainty, and offers both validation and calibration via experimentation.
When the model is refreshed after an experiment is completed, the ground truth results are used to calibrate it. That process ensures that it is 100% accurate and 100% certain of the incrementality metrics for the treated traffic on the dates when the experiment was running. It also helps to significantly increase accuracy and certainty throughout the rest of the model even for campaigns and countries that were not treated.
Recalling a previous example, let’s say a significant discrepancy between last touch day 3 ROAS, provided by a last touch MMP, and incrementality day 3 ROAS, provided by Polaris, has been identified on a mission-critical campaign. Before actioning the marketing process based only on the incrementality version of the metric, which would result in a large budget decrease (day 3 ROAS incrementality is low, but last touch seems to be overvaluing), you decide to execute an experiment to find the truth.
At the end of the experiment, the treatment is reversed. Three days later, the ground truth incrementality is known for all cohort day 3 metrics including day 3 revenue, from which day 3 ROAS is derived. You visit the Analytics screen in Polaris and find that the day 3 ROAS for the campaign has been updated to reflect the results of the experiment. The econometric model was directionally accurate, but was off by a small margin. Last touch attribution was far overvaluing the campaign. The experiment-validated incrementality day 3 ROAS is used in the marketing process with confidence and the campaign’s budget is sharply decreased.
To summarize, the incrementality metrics derived from the experiment-calibrated model and surfaced in the Polaris dashboard and API, should be utilized as the primary influence for marketing action in the same way that last touch metrics are utilized in the same process. The only difference is that last touch attribution isn’t designed to measure incrementality, which is all that matters, provides no information about how uncertain it is, and usually offers no way to validate its results. Polaris only measures incrementality in a privacy-centric way, quantifies uncertainty, and offers both validation and calibration via experimentation.
Updated on: 28/07/2022
Thank you!