> ML SERIES :: 07 regularization

Page 07 of 14 · Core techniques

Regularization: keeping a flexible model honest

On page 06, the degree-11 polynomial memorized the noise. One fix is a simpler model. Regularization is another: keep the flexible model, but charge it a penalty for using large weights. The model can still bend, but only when the data truly demands it.

Sinclair College - Dr. Mike Libassi

Loss=MSE+λ · penalty(w)

λ (lambda) sets how much the penalty counts. λ = 0 is no regularization. A huge λ forces the weights toward zero and the model toward a flat line.

Tame the degree-11 polynomial

Same 12 training points as page 06, same wildly flexible model. Raise λ and watch the curve calm down, the validation error fall, and the weights shrink.

strength regularization strength λlambda;
training error
validation error
weights exactly zero
diagnosis

The 12 weights (log scale)

Weight wk multiplies xk. Bar height is log10(1 + |w|), so each grid line is ten times larger than the one below it. Cyan bars are positive, amber negative.

L2 versus L1

L2 / ridge
λ Σ wj²

Penalizes the square of each weight. Big weights shrink a lot, small ones barely. Weights get small but rarely exactly zero. The most common default.

L1 / lasso
λ Σ |wj|

Penalizes the absolute value. Pushes unhelpful weights all the way to zero, so it also performs feature selection. Useful when you have many features and suspect most are noise.

Combining both is called elastic net. The intercept w0 is normally left unpenalized, since shifting the whole curve up or down is not overfitting.

Other ways to regularize

early stopping

Watch validation error during training and stop when it starts rising, even if training error is still falling. Cheap and very common with neural networks.

dropout

For neural networks (page 11): randomly switch off a share of neurons on each training step, so no single neuron can memorize. All neurons are used at prediction time.

more data

Not a penalty, but the most reliable cure. Augmentation, such as flipping or cropping images, creates extra training examples.

Choosing λ: λ is a hyperparameter, a setting you pick rather than one the model learns. Try several values and keep the one with the lowest validation error, never the lowest test error. Cross-validation (page 09) automates this.
from sklearn.linear_model import Ridge, Lasso, RidgeCV

ridge = Ridge(alpha=0.01).fit(X_train, y_train)    # sklearn calls lambda "alpha"
lasso = Lasso(alpha=0.001).fit(X_train, y_train)
print((lasso.coef_ == 0).sum(), "features dropped")

best = RidgeCV(alphas=[1e-4, 1e-3, 1e-2, 1e-1, 1]).fit(X_train, y_train)
print(best.alpha_)                                  # lambda chosen by cross-validation