3 Training and Testing

Why data is divided

A model should be judged on examples it did not use to learn. Divide the available data into separate training, validation, and test sets before fitting any data-dependent steps.

The three data sets

The training set teaches the model: it is used to learn model parameters from examples and their targets. Preprocessing steps that learn values from data, such as a scaler's mean and standard deviation, are fitted on the training set too.

The validation set supports development. Use it to compare models and choose settings, features, or other design choices. Since those choices are guided by validation results, a model can gradually become tailored to the validation set.

The test set is held apart until development is complete. Use it for a final estimate of performance on unseen data, not to choose or tune the model. Repeatedly checking test results and adjusting the model based on them makes the test set part of development and weakens that estimate.

There is no single correct split proportion for every dataset. Keep enough training examples to learn effectively and enough validation and test examples to make useful comparisons and estimates. Partitions should reflect the examples the model is meant to encounter.

For a small dataset, cross-validation can help use development data efficiently: train on some folds and validate on the remaining fold, repeating the process. Keep a separate test set for final evaluation when possible.

Fit models and preprocessing safely

The word fit means using data to learn something. A model's fit operation learns predictive parameters; a preprocessing step's fit operation learns transformation values. After fitting a transformation on training data, apply that same learned transformation to validation and test data with transform. Do not fit or refit preprocessing on those held-out sets.

For example, a scaler should learn a feature's average and spread from the training set only. If it uses the test set to calculate those values, information from test examples has influenced the modeling process, even if their labels were not used. This is data leakage and can make evaluation scores overly optimistic.

A pipeline helps keep preprocessing and model fitting together. Each step is learned from the data supplied for training and then applied consistently to held-out examples:

For model or hyperparameter selection, use a validation set or cross-validation on training or development data; do not select settings based on test performance. Pipelines are also useful in cross-validation because preprocessing is refitted separately within each training fold.

Design splits to prevent leakage

Data leakage can happen beyond preprocessing. A feature may contain information that would not be available when a real prediction is made; for example, a later medical test result could be used to predict an earlier diagnosis. Leakage can also arise when duplicate or closely related examples appear in both training and test sets, or when future observations are allowed into training to predict the past.

Choose the split to match how predictions will be used. For independent examples, a randomized split may be suitable. When several records belong to the same person, household, or other group, keep those groups together across partitions. For time-dependent prediction, train on earlier data and evaluate on later data. These choices help prevent information from crossing the split in ways that would not be possible in real use.

A practical evaluation checklist

Use this checklist when preparing an evaluation:

1. Decide what counts as one independent example and remove or group duplicates appropriately.

2. Split the data before fitting scalers, imputers, feature selectors, or other learned preprocessing.

3. Fit the full pipeline on training data; use validation data for development decisions.

4. Keep the test set untouched until the model and its settings are finalized.

5. Ensure features use only information that would be available at prediction time.

In short, training data teaches the model, validation data guides development, and test data provides a final independent check. Fit models and data-dependent preprocessing only on training data, apply learned transformations consistently to held-out data, and design splits to prevent related, future, or target-derived information from leaking across them.