3 Training and Testing Free Online FlashCards

What is the training set used for?

The training set is used to fit the model’s parameters from examples and their targets.

In machine learning, what does “fit” mean?

To fit means to use data to learn something, such as model parameters or a preprocessing transformation’s values.

When does a feature cause leakage through prediction timing?

A feature leaks information if it uses facts unavailable at prediction time, such as a later medical test result used to predict an earlier diagnosis.

What is the validation set used for?

The validation set helps compare models and choose settings, features, and other design choices during development.

What is the test set used for?

The test set provides a final estimate of performance on unseen data after development is complete.

How should you choose the proportions of training, validation, and test sets?

There is no universally correct proportion. Keep enough examples in each set for effective learning, useful comparisons, and reliable estimates.

How can cross-validation help with a small dataset?

Cross-validation trains on some folds and validates on the remaining fold, repeating the process to use development data efficiently.

Which data should be used to fit learned preprocessing steps?

Fit data-dependent preprocessing, such as a scaler or imputer, on training data only.

How should a fitted preprocessing step be applied to held-out data?

Apply the transformation learned from training data to validation and test data with transform; do not refit it on those sets.

Why is it leakage to fit a scaler using test data?

Using test examples to calculate a scaler’s values lets test information influence modeling, causing data leakage and potentially optimistic scores.

How does a pipeline help prevent inconsistent preprocessing?

A pipeline keeps preprocessing and model fitting together, helping steps be learned from training data and applied consistently to held-out examples.

Why should you avoid tuning a model using repeated test-set results?

Repeatedly adjusting a model based on test results makes the test set part of development and weakens its value as an independent estimate.