7 Model Evaluation

Learn how to evaluate regression and classification models, interpret common metrics, and choose measures and thresholds that reflect real-world error costs.

Use data splits for evaluation

Model evaluation measures how well a trained model predicts data it has not seen. Choose measures that reflect the task and the consequences of errors.

Use training data to fit the model, validation data to compare models and tune settings, and a separate test set for a final, relatively unbiased estimate of performance. Repeatedly using the test set to make choices can make its score overly optimistic.

Measure regression performance

Regression predicts a numerical value. For actual values $y_i$ and predictions $\hat{y}_i$ , the is $e_i = y_i - \hat{y}_i$ . Regression measures summarize these errors in different ways.

Absolute and squared errors

is the average absolute error:

\text{MAE} = \frac{1}{n}\sum_i |e_i|

It is expressed in the outcome’s units. For example, an MAE of $3$ means predictions are off by $3$ units on average. MAE is less affected by a few very large errors than squared-error measures.

is the average squared error:

\text{MSE} = \frac{1}{n}\sum_i e_i^2

Squaring errors penalizes large misses more heavily, and the result is in squared outcome units. is the square root of MSE:

\text{RMSE} = \sqrt{\text{MSE}}

RMSE is expressed in the outcome’s units and, like MSE, gives extra weight to large errors. When a few extreme misses are especially costly, RMSE may be more informative than MAE. When typical error matters and extreme values should not dominate the score, MAE may be preferable.

Comparing with a mean-value baseline

The coefficient of determination ( $R^2$ ) compares prediction errors with the variation around the mean outcome. A score of $1$ indicates perfect predictions; $0$ means performance is equivalent to predicting the test-set mean; and a negative score means performance is worse than that baseline. Unlike MAE and RMSE, $R^2$ is not an error measured in the outcome’s units.

Interpret regression measures in context and compare them against a simple baseline.

Read a

Classification predicts a category, such as positive or negative. In binary classification, a counts four kinds of outcomes:

True positives (TP): positive cases correctly predicted positive.
False positives (FP): negative cases incorrectly predicted positive.
True negatives (TN): negative cases correctly predicted negative.
False negatives (FN): positive cases incorrectly predicted negative.

These counts distinguish different types of correct and incorrect predictions and are used to calculate classification measures.

Choose classification measures

is the fraction of predictions that are correct:

\text{Accuracy} = \frac{TP+TN}{TP+FP+TN+FN}

It can be misleading when classes are imbalanced. For example, if only one case in every hundred is positive, predicting every case as negative gives $99\%$ while finding no positive cases.

is the fraction of predicted positives that are truly positive:

\text{Precision} = \frac{TP}{TP+FP}

Prioritize when false alarms or unnecessary interventions are costly. Recall, also called sensitivity, is the fraction of actual positives found:

\text{Recall} = \frac{TP}{TP+FN}

Prioritize recall when missing a positive case is costly. is the fraction of actual negatives correctly identified:

\text{Specificity} = \frac{TN}{TN+FP}

It is useful when false positives are a particular concern.

The is the harmonic mean of and recall:

F_1 = \frac{2PR}{P+R}

It summarizes their balance, but does not account for true negatives and may not match the costs of a particular application. With imbalanced classes, report class-specific measures rather than relying on alone.

Evaluate thresholds, rankings, and probabilities

Many classifiers produce scores or probabilities, then assign labels using a . Changing the changes the and usually trades against recall. Select a on validation data to meet the application’s needs, then evaluate the final choice on the test set.

summarizes how well a classifier ranks positive examples above negative ones across thresholds; it does not assess one chosen . For rare positive classes, a can be more informative about the trade-off between finding positives and generating false alarms.

If predicted probabilities themselves matter, evaluate probability quality as well, using measures such as log loss, and check . For example, a forecast of $0.8$ should correspond to about an $80\%$ event rate when probabilities are well calibrated.

Select and report metrics

Start with the decision the model supports and ask which kinds of errors cost most. Choose a primary metric that reflects those costs, then inspect complementary measures so an improvement in one does not hide a serious weakness in another.

The task can guide which metric to emphasize. A medical screening system may emphasize recall while monitoring ; a spam filter may place greater weight on to avoid hiding legitimate messages. Consider whether the application needs accurate probabilities, a ranking across cases, or labels at a fixed operating , because these call for different measures.

Evaluate on data representative of the intended use and compare performance with a simple baseline. Report metric definitions, the chosen where relevant, and test-set results. A single score is not a complete description of model quality.