Page 05 of 14 · Core techniques

Optimizing training: getting downhill faster

Gradient descent always heads downhill, but how quickly it gets to the bottom depends on the step size, the shape of the landscape, and how much data you look at per step. Small changes here can turn hours of training into minutes, or into a model that never converges at all.

Sinclair College - Dr. Mike Libassi

Explore the loss landscape

This is a top-down view of a loss surface for a model with two weights, like a contour map of a valley. The lowest point is the center. Each dot is one gradient descent step. Change one setting at a time and compare how many steps it takes.

learning rate α

Scale the features Use momentum (β = 0.9) Mini-batch noise

Things to try: push the learning rate up until training explodes; turn on scaling and watch the valley become round; add momentum to a slow run.

The learning rate is the most important setting

too small

Every step is safe but tiny. Training takes forever and may stop early, before reaching the bottom.

about right

Big steps at first, then steady progress into the valley floor.

too large

Each step overshoots to the other side of the valley, higher than before. The loss grows until it becomes infinite (shown as NaN in Python).

Many training runs also use a learning rate schedule: start with a larger rate, then shrink it over time so the final steps are precise.

Why feature scaling helps

On page 04, the user count ran from 0 to 10 while response times reached 250. When features live on very different scales, the loss valley becomes a long, narrow canyon. The gradient points mostly across the canyon instead of along it, so gradient descent zig-zags. Rescaling every feature to a similar range makes the valley round, and the gradient points straight at the bottom.

x_scaled=(x − μ) / σ

Standardization: subtract the mean μ (mu) and divide by the standard deviation σ (sigma). Every feature then has mean 0 and spread 1. Compute μ and σ on the training set only, or you leak test data (page 02).

How much data per step?

The true gradient uses every training row, which is expensive when there are millions. In practice, each step uses a sample:

batch

All rows per step. Smooth and accurate, but each step is slow.

stochastic (SGD)

One random row per step. Very fast steps, but a noisy, jittery path.

mini-batch

32 to 512 rows per step. The standard compromise, and a good fit for GPUs.

One full pass through the training data is called an epoch. With 10,000 rows and a batch size of 100, one epoch is 100 steps.

Smarter optimizers

Momentum lets each step keep part of the previous step's direction, like a ball rolling downhill. Zig-zags cancel out and progress along the valley adds up. Adam combines momentum with a separate, automatically tuned step size for each weight. It is the default choice for training neural networks.

In Python

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDRegressor

scaler = StandardScaler().fit(X_train)       # learn mu and sigma from training data only
X_train_s = scaler.transform(X_train)
X_test_s  = scaler.transform(X_test)         # reuse the same mu and sigma

model = SGDRegressor(learning_rate="invscaling", eta0=0.01, max_iter=1000)
model.fit(X_train_s, y_train)

The learning rate controls step size: too small is slow, too large diverges.
Scale features so the loss surface is round and gradient descent heads straight to the minimum.
Mini-batches make each step cheap; momentum and Adam make the steps smarter.