Page 06 of 14 · Core techniques

Underfitting and overfitting: too simple, too clever

A model that scores perfectly on its training data can still be useless. The goal is not to memorize the examples, but to learn the pattern underneath them well enough to handle examples it has never seen. That ability is called generalization.

Sinclair College - Dr. Mike Libassi

underfit

Too simple to capture the pattern. Bad on training data and new data. Like a student who skimmed the chapter.

good fit

Captures the real pattern and ignores the noise. Similar scores on training and new data.

overfit

Memorizes every quirk, noise included. Great on training data, poor on new data. Like a student who memorized the answer key.

Turn up the model's flexibility

The white points are 12 training examples drawn from a smooth curve plus random noise. The model is a polynomial; raising its degree lets it bend more. Watch the training error and the validation error as you slide.

polynomial degree

Show validation points Show the true curve

training error (MSE)

validation error (MSE)

diagnosis

Error versus degree

training errorvalidation error

Bias and variance

Two kinds of error pull in opposite directions:

bias

Error from wrong assumptions, such as forcing a straight line onto a curve. High bias means underfitting.

variance

Error from being too sensitive to the particular training sample. Retrain on slightly different data and the model changes wildly. High variance means overfitting.

Making a model more flexible lowers bias but raises variance. The best model sits at the bottom of the validation curve, where the two balance out.

Reading the symptoms

Training error	Validation error	Diagnosis	What to try
high	high	underfit	More flexible model, better features, train longer
low	much higher	overfit	More data, simpler model, regularization (page 07)
low	close to training	good fit	Confirm once on the test set, then ship it

Why this matters in security

An overfit malware classifier can learn shortcuts that happen to separate the training samples, such as a compiler timestamp or one packer used by a single malware family, instead of real malicious behavior. It scores 99% in the lab and misses the first new variant in the wild.

Rule of thumb: if a model's lab results look too good to be true, check for overfitting and data leakage (page 02) before celebrating.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

for degree in range(1, 12):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X_train, y_train)
    print(degree, model.score(X_train, y_train), model.score(X_val, y_val))

Underfitting: too simple, high error everywhere. Overfitting: memorizes training data, fails on new data.
Always compare training error with validation error; the gap tells you which problem you have.
Choose model complexity at the bottom of the validation error curve.