Basics

Model error refers to the difference between backtesting predictions of the model and the actual observed values. It is meant to provide some insight into how well the model is able to predict a future holdout test set with its current fit.

This is not a comprehensive evaluation of the model’s accuracy and cannot be used to validate a media mix model (MMM). Validation is only possible with ground truth incrementality, which is why experiments are so important. In fact, backtested normalized root mean squared error (NRMSE) is only one of several optimization objectives utilized in the training process. Model error can only give an important, but limited view of a model’s fit and correlative prediction power.

Errors File

The predicted column contains the backtesting predictions. The actual column contains the observed values. Ideally, those numbers are close. If they aren't, it could mean there are unmodeled performance factors. The abs_pct_error column contains the mean absolute percentage error or MAPE, which is another way to look at the difference.

Updated on: 22/08/2022