7 Model Evaluation

Purpose of model evaluation

Model evaluation estimates how well a trained model will perform on data it has not seen. A score is useful only when both the evaluation data and the metric reflect the model’s intended use.

Evaluation data and procedure

A common procedure separates labeled data into three parts. The training set is used to fit the model, the validation set is used to select models or tune settings, and the test set is reserved for a final, unbiased estimate of performance. Repeatedly choosing models based on test-set results leaks information from the test set into model selection and can make the reported score overly optimistic.

Cross-validation and appropriate splits

When data is limited, k-fold cross-validation on the training data can support model selection. Divide the data into $k$ folds, train on $k-1$ folds, evaluate on the remaining fold, and repeat until each fold has served as the validation fold. Average the fold scores to summarize performance, and consider their spread to understand how much results vary across splits. Keep a separate test set for final evaluation when possible.

Use split procedures that reflect the data structure and intended use. For classification, stratified folds approximately preserve class proportions. For repeated observations from the same person or group, keep groups separated across folds. For time-dependent data, train on earlier observations and evaluate on later ones. Otherwise, information can leak across splits or the evaluation can fail to reflect future use.

Fair comparisons and leakage prevention

Compare models using the same data splits and a metric chosen before inspecting test results. A simple baseline helps show whether a model adds value: for regression, predict the training-set mean; for classification, predict the most frequent class.

Check for data leakage during preprocessing. Any preprocessing that learns from the data, such as scaling or feature selection, should be fitted using only the training portion of each split.

Regression metrics

For true values $y_i$ and predictions $\hat{y}_i$, common measures summarize different aspects of regression error.

Mean absolute error (MAE)

Mean absolute error (MAE) is the average absolute difference between true and predicted values:

It reports average error in the target’s units and is less affected by large errors than squared-error metrics.

Root mean squared error (RMSE)

Root mean squared error (RMSE) is the square root of the average squared difference:

It is also in the target’s units, but penalizes large errors more heavily than MAE. If unusually large prediction errors are especially costly, RMSE may be more informative. If a typical error is the priority and extreme errors should not dominate the summary, MAE may be preferable.

Coefficient of determination

The coefficient of determination, $R^2$, compares squared prediction error with the error from predicting the test-set mean. A score of $1$ is perfect, $0$ means no improvement over that mean baseline, and a negative score means the model performs worse than the baseline.

Always report target units where applicable: an RMSE of $5$ means little without knowing what is being predicted and its scale. MAE, RMSE, and $R^2$ answer different questions, so compare models using the measure that matches the actual cost of errors.

Classification metrics

A confusion matrix counts actual versus predicted classes. In a binary task, it records true positives ($\text{TP}$), false positives ($\text{FP}$), true negatives ($\text{TN}$), and false negatives ($\text{FN}$). It reveals which kinds of mistakes a model makes, not just how many predictions are correct.

Measures of classification performance

• Accuracy is the fraction of predictions that are correct. It can mislead when classes are imbalanced: a model that labels every example as the majority class can have high accuracy while missing nearly all minority-class cases.

• Precision is the fraction of predicted positives that are truly positive: $\text{TP}/(\text{TP}+\text{FP})$. It matters when false alarms are costly.

• Recall is the fraction of truly positive examples found by the model: $\text{TP}/(\text{TP}+\text{FN})$. It matters when missing positive cases is costly.

• F1 score is the harmonic mean of precision and recall. It summarizes their balance, but does not reflect true negatives and may hide whether precision or recall matters more.

• Balanced accuracy averages recall across classes, giving each class equal weight. It can be more informative than ordinary accuracy when class frequencies differ.

Choose metrics in light of the consequences of errors. In a medical screening task, a missed case may be more consequential than an extra follow-up, so recall may be emphasized. In a system that sends costly alerts, precision may matter more. Inspect per-class scores as well as any aggregate score.

For multiclass tasks, precision, recall, and F1 can be averaged across classes. Macro averaging weights classes equally, while weighted averaging weights them by their frequency; state which averaging method is used.

Thresholds and score-based evaluation

When a classifier produces scores or probabilities, its predictions depend on a decision threshold. A ROC curve plots true-positive rate against false-positive rate across thresholds, and ROC AUC summarizes ranking performance across those thresholds. These measures can help compare score-based classifiers, but they do not select the right threshold for a particular application.

Choose a threshold using validation data and the relative costs of the possible errors, then report the resulting test-set metrics.

Putting evaluation into practice

Evaluate models on held-out data that matches the intended use, and prevent information from crossing between training and evaluation. Use cross-validation for model selection when appropriate, while reserving test data for a final check. Compare models on the same splits, against a baseline, and with measures aligned to the cost of mistakes. For regression, MAE, RMSE, and $R^2$ capture different aspects of performance; for classification, combine confusion-matrix counts with metrics such as precision, recall, F1, or balanced accuracy as needed.