8 Generalization and Overfitting

Assessing performance on unseen data

A model's training performance alone cannot show whether it learned patterns that apply broadly or merely memorized details of its training data. Its performance on a held-out test set estimates performance on future data, provided the test examples are relevant to the setting where the model will be used.

Training, validation, test, and future data should be sufficiently similar. If the real-world data distribution changes, model performance may change too.

Recognizing underfitting and overfitting

Overfitting occurs when a model fits the training data very closely, including random noise or peculiarities that do not recur, and performs substantially worse on new data. It often appears as strong training performance and weaker validation performance. For example, a high-degree polynomial might pass through every training point in a small, noisy dataset but bend sharply between points, producing poor predictions for new observations.

Underfitting occurs when a model is too limited or insufficiently trained to capture important patterns. It performs poorly on both training data and new data; for instance, a straight-line model may underfit a strongly curved relationship.

Use the score pattern as a diagnostic clue, not an automatic verdict:

• Poor training and validation performance suggests underfitting.

• Strong training performance and much weaker validation performance suggests overfitting.

• Good training and validation performance with a modest gap suggests a useful fit that may generalize.

Judge performance with a metric appropriate to the task, and check that the validation data represent the intended use. Learning curves and validation curves can help reveal underfitting or overfitting as training-set size or model settings change.

Choosing a targeted remedy

Match changes to the evidence. If a model underfits, try a more expressive model, useful features, or a better training procedure. Check that training has been adequate and that the chosen metric reflects the goal.

If a model overfits, consider reducing model complexity, removing unhelpful features, or using regularization. Regularization penalizes complexity; L2 regularization, for example, discourages excessively large weights. During iterative training, early stopping can help: stop when validation performance stops improving rather than continuing to optimize training fit.

Additional training examples can help when they cover the cases the model must handle. Adding data that does not represent real use may not solve the problem.

Tuning without compromising evaluation

Tune models using validation data or cross-validation: compare candidate models and settings on data not used to fit them. Keep a separate test set for the final evaluation. Repeatedly adjusting a model based on test results gradually turns the test set into part of the tuning process, making the final evaluation less meaningful.

Prevent data leakage by splitting data before learning preprocessing steps such as scaling, imputation, or feature selection. Fit those steps on training data only, then apply the learned transformations to validation and test data. Pipelines help keep this separation consistent.

A practical evaluation workflow

Establish a simple baseline, inspect training and validation performance, make a targeted change, and compare results using the same validation method. After selecting an approach, evaluate it once on the held-out test set.

In summary, generalization concerns performance on unseen data. Overfitting means training performance is much better than validation performance, while underfitting means both are poor. Use validation results and learning curves to diagnose the problem, then adjust model complexity, regularization, training, features, or data as appropriate. Keep test data separate from fitting and model selection so the final evaluation remains meaningful.