Stop 11 · Time Series AnalysisTopics · 7/7

Forecasting Evaluation Metrics

In short

'Our forecast is 92% accurate' can mean almost anything. Picking the right error metric, and comparing against a naive forecast, is how you prove a forecast is actually useful.

Metrics specific to forecasting quality — MAPE (mean absolute percentage error) is intuitive but breaks down near zero values, RMSE and MAE from Phase 7 also apply directly to forecasting.

The mascot holds up a pocket watch on a railway platform as a train pulls in.

Think of it like this

Judging train arrival predictions

The station display predicts each train's arrival. Being 5 minutes off on a long-distance train is fine. Being 5 minutes off on a local that runs every 3 minutes is useless. The same size of error can be big or small depending on scale, so you need the right ruler.

Watch it happen

Two families of forecast error

MAE / RMSE

In the data's own units
Easy to explain: 'off by 7 units'
Can't compare across products

MAPE

Error as a percentage
Compares across scales
Explodes when actuals near zero

Report MAE or RMSE in units people understand, use percentages only when actuals stay well above zero, and always compare with a naive forecast. MASE does that comparison for you.

The big ideas

What you need to know

Beat the naive forecast

'Tomorrow equals today' or 'same as last week' is your baseline. A model that can't beat it isn't worth deploying.

MASE scales by naive

MASE divides your MAE by the naive forecast's MAE. Below 1 means you beat the naive method.

MAPE's trap

An actual of 2 and a forecast of 10 is a 400% error. One tiny day can wreck the average.

Score by horizon

Errors usually grow further ahead. Report them separately for 1 day, 1 week and 1 month out.

See it in code

Walk through the code

python

1import numpy as np2from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error3 4actual   = np.array([120, 135, 128, 2, 140])5forecast = np.array([110, 140, 130, 10, 150])6 7print("MAE ", mean_absolute_error(actual, forecast))8print("MAPE", mean_absolute_percentage_error(actual, forecast))   # 0.84 means 84%9 10naive = actual[:-1]                                  # yesterday's value as forecast11mase = mean_absolute_error(actual, forecast) / mean_absolute_error(actual[1:], naive)12print("MASE", round(mase, 2))

1/4Five days of actuals and forecasts. Day 4 had a near-zero actual.

MAE is 7 units, but MAPE comes out around 84%, almost all from the day with an actual of 2.

Save yourself the pain

Common mistakes

Oops

Quoting MAPE on data with zeros or near-zero days.

Do this instead

Use MAE, RMSE, MASE or weighted MAPE, which divides total error by total actuals.

Oops

Reporting only an average error, with no baseline next to it.

Do this instead

Show the naive forecast's error alongside yours, so the improvement is clear.

Your mission

Score three forecasts honestly

~25 min

Quick check

A forecast has MASE = 1.3. What does that tell you?

Go deeper

1Calculate MAPE, MAE, and RMSE for a forecast and compare what each emphasizes.
2Learn why MAPE becomes unstable or undefined when actual values are near zero.
3Practice presenting a forecast's expected error range, not just a single predicted number.