Methodology

In order to perform the performance impact analysis, we use the synthetic control method to establish an expectation of performance in each adoption country based on similar countries that are not participating in the adoption phase. That expectation represents the control. It is important that marketing continues to be driven based on last touch data in those non-participating countries. The observed change in each participating country's metrics during the adoption phase will be compared to the expectation or control, and the difference is the uplift attributable to the adoption of incrementality measurement.

This works for the same reason it works for our ground truth experiments: country-level metrics require no form of attribution or modeling to measure and are therefore always dependable. That's why we recommend that, during the adoption phase, incrementality be used to drive the marketing process for all traffic across one or more entire countries.

Best-Effort

While the methodology is very similar to the way we evaluate ground truth incrementality for lift experiments, it isn't exactly the same. During an experiment, we recommend that you keep everything business-as-usual to control for potential confounders. During the adoption phase, it is unlikely that you are able to avoid large budget swings and other abnormal events completely. Therefore, we are measuring the performance impact of the adoption phase using a best-effort method, not a foolproof one. That said, given context, the impact analysis is usually very clear, at least directionally.

Output

Adoption analysis data packages will be delivered that contain all data, including summaries, plots, and underlying details. Most of the data is in CSV format while the plots are PNGs. Summary files provide the most important information like the total and average daily impacts to the selected KPIs. Plots provide a graphical view of the impact. Underlying detail files provide extra information and enable validation of the methodology.

In addition, a presentation will also be delivered that presents the same information in a more easily consumable format. The presentation also highlights and distills the key takeaways, which can be helpful for executives.

Updated on: 28/07/2022