Business Experiment Evaluation Considerations
The business perspective is just as important, often more important, than the statistical perspective. When evaluating experiments, consider the importance of the traffic to the business at both the strategic and tactical levels.
The goal is to assess and weigh two types of business risk.
The risk of believing the incrementality metrics of certain traffic
The risk of treating the traffic with a pause
The first risk can be assessed by examining two indicators.
The discrepancy between last touch attribution and incrementality metrics of certain traffic
The spend/reach of the traffic
Where the discrepancy between last touch and incrementality is high, the marketing process would dictate very different actions depending on which version of the metrics is believed, elevating the risk of believing one or the other without strong evidence. Obtaining ground truth incrementality via an experiment prior to making such a potentially impactful decision provides that indisputable evidence for the incrementality metrics and therefore the proper action.
However, if spend is low, even a huge discrepancy represents very little risk to the business. In general, understanding the ground truth incrementality of an inconsequential channel in a low priority country is not nearly as valuable as a mission-critical channel with a significant budget.
It is also crucial to understand the risks of treating the traffic in an experiment. For example, If the traffic is so strategically important that any disruption could jeopardize the business, the experiment should be at least temporarily eliminated as a candidate for execution. Also, the nature of the channel should be considered. A dwindling number of ad networks still employ legacy algorithms that don’t handle disruptions well. Often, collaboration between marketing and data science teams is required to balance the statistical and business considerations.
The goal is to assess and weigh two types of business risk.
The risk of believing the incrementality metrics of certain traffic
The risk of treating the traffic with a pause
The first risk can be assessed by examining two indicators.
The discrepancy between last touch attribution and incrementality metrics of certain traffic
The spend/reach of the traffic
Where the discrepancy between last touch and incrementality is high, the marketing process would dictate very different actions depending on which version of the metrics is believed, elevating the risk of believing one or the other without strong evidence. Obtaining ground truth incrementality via an experiment prior to making such a potentially impactful decision provides that indisputable evidence for the incrementality metrics and therefore the proper action.
However, if spend is low, even a huge discrepancy represents very little risk to the business. In general, understanding the ground truth incrementality of an inconsequential channel in a low priority country is not nearly as valuable as a mission-critical channel with a significant budget.
It is also crucial to understand the risks of treating the traffic in an experiment. For example, If the traffic is so strategically important that any disruption could jeopardize the business, the experiment should be at least temporarily eliminated as a candidate for execution. Also, the nature of the channel should be considered. A dwindling number of ad networks still employ legacy algorithms that don’t handle disruptions well. Often, collaboration between marketing and data science teams is required to balance the statistical and business considerations.
Updated on: 28/07/2022
Thank you!