Experiment Requirements

While there are no absolute requirements, the following should be considered when evaluating experiments.

Share of total budget (if you are spending an extremely small share of the total budget on a channel, pausing it may have an impact that is difficult to detect)
Country level performance volume (if the treatment country has very low install or revenue volume, pausing any traffic is unlikely to have a measurable effect on those metrics)
Business as usual during experiment (ensure you can keep "business as usual" during the experiment to control for as many potential confounding variables as possible)

We typically require a minimum experiment duration of Seven(7) days with no Spend or <$10.00 minimum for Pause experiments. This timeframe is reasonable to obtain good results while minimizing business disruption and potential for confounding variables.

Evaluation Considerations

When evaluating experiments in Polaris, the primary motivation is to rank the priority of experiments by understanding the impact of each. To that end, there are two main types of considerations: statistical evaluation and business evaluation.

A mixture of both is often utilized in practice and the strategy can change day to day. One day, it may be critical to find the ground truth incrementality of an important channel. Next week, it may not be strategically feasible to pause any important channel so adding statistical accuracy to the media mix model as a whole by treating a smaller channel may be ideal.

To learn more about each consideration, use the following links.

Statistical Evaluation Considerations
Business Evaluation Considerations

Updated on: 16/05/2024