• Experiment Methodology

  First, a brief overview of our preferred incrementality experimentation methodology is in order. There are other documents and materials that delve deeper into the data science, so the purpose of this explanation is only to provide you with a high level intuition. The Old Way: Randomized Controlled Trials There are many ways to design incrementality experiments. The most popular has historically been a randomized controlled trial (RCT), usually called incrementality testing. This design is nFew readers

• Experiment Evaluation and Prioritization

  Experiment Requirements While there are no absolute requirements, the following should be considered when evaluating experiments. Share of total budget (if you are spending an extremely small share of the total budget on a channel, pausing it may have an impact that is difficult to detect) Country level performance volume (if the treatment country has very low install or revenue volume, pausing any traffic is unlikely to have a measurable effect on those metrics) Business as usual duriFew readers

• Statistical Experiment Evaluation Considerations

  For any automatically generated experiment, statistical considerations are quantified in the Experiments screen in the Ratio, Score, and Risk columns. The Ratio column is the ratio of Score to Risk. It is also the default sort order of the table, ranking the highest ratios at the top. The Score column represents the predicted information gain to be obtained by executing an experiment and the Risk column represents the estimated risk of executing an experiment. (https://storage.crisp.chat/useFew readers

• Business Experiment Evaluation Considerations

  The business perspective is just as important, often more important, than the statistical perspective. When evaluating experiments, consider the importance of the traffic to the business at both the strategic and tactical levels. The goal is to assess and weigh two types of business risk. The risk of believing the incrementality metrics of certain traffic The risk of treating the traffic with a pause The first risk can be assessed by examining two indicators. The discrepancy betwFew readers

• Experiment Result Analysis

  Accessing Experiment Results Understanding how to interpret the results of an executed experiment is crucial for confidence in the incrementality data. You can access the experiment results screen for any completed experiment by clicking the results button on the right side of the row in the Experiments screen. It replaces the edit button for all experiments that are in the Completed status. (https://storage.crisp.chat/users/helpdesk/website/e7f7cacc53e9c800/a9ff0a3e-a409-42d9-a5b3-f655abFew readers

• Experiment Result Actionability

  The ultimate goal of executing an experiment is to inform action resulting in business value so it’s important to understand how to convert the ground truth incrementality insights into action. The key to remember on this topic is that experiment results are automatically fed into the econometric model that outputs the incrementality metrics. Therefore, you simply need to utilize the incrementality metrics available in the Polaris dashboard on the Analytics and other screens as well as the APIFew readers