5.10. Evaluating the Forecasts

It is always expected that actual values will deviate from the forecast. Measuring this difference as a KPI and assessing it for the safety stock calculation is an important part of demand forecasting process.

There are many error measures used to evaluate the forecast. Among them:

Mean Squared Error (MSE);
Root Mean Squared Error (RMSE);
Mean Absolute Error (MAE);
Mean Absolute Percentage Error (MAPE).
Weighted Mean Absolute Percentage Error (WMAPE).

Streamline evaluates forecast based on MAPE. It is obvious that in order to evaluate the forecast the actual data for the forecasted period must be imported.

Viewing Forecast Quality

The results of the forecast evaluation are represented by the Model MAPE in the Model tab of the Panel (see figure below).

Model MAPE is MAPE calculated based on the model response for the data that was used to build the model. This response is indicated with the blue dots in the Plot (see figure above).

The MAPE shown in the figure above is the evaluation of periods (Jun 2018 - Dec 2019) for which actual data exists.

To see the forecasts evaluation for all of the items in one report, go to the Reports tab and select the Forecast performance report (see figure below).

The Absolute percentage error section of the report represents APE calculated for each item, location, and period. As we do not have actual data for periods starting with Jan 2020, the corresponding columns are empty in the report.

The table can have gaps or even empty rows. A gap occurs when there were no actual sales in the period. Streamline can’t calculate APE in this case. An empty row arises if either the item is inactive or an intermittent model is used.

The last two columns of the report represent MAPE and Weighted MAPE calculated across all of the periods for each planning item.

The Overall MAPE, shown in the toolbar, is MAPE weighted across all the items in the project. If the products’ prices are given this is a price-weighted MAPE measure that represents overall revenue percentage error. Otherwise, it is calculated based on the volume of sales. You can read more about WMAPE on Wikipedia.

You can also switch the measure of the forecast quality between error and accuracy (which is 100% - error ). To do this, we click the Settings button found on the toolbar and select the Accuracy option for the Measure of forecast quality (see figure below).

Now, the report shows the percentage accuracy of the forecasts for each period in the Percentage accuracy section and the average accuracy across the available evaluations in the last column (see figure below).

The Overall accuracy of the toolbar equals to 100% - Overall MAPE.

2018/04/02 14:55

Forecast Quality Measures

Streamline is able to evaluate forecast quality using two measures:

Error (MAPE) or
Accuracy (100% - Error).

The default measure is Error. You can switch between the measures by going to the menu File > Settings > Project tab > KPIs group and changing the Measure of accuracy parameter (see figure below).