> ML SERIES :: 04 linear regression

Page 04 of 14 · Foundations

Linear regression: your first model that learns

A straight line is the simplest model there is, and it already contains every idea in machine learning: parameters, a measure of error, and a procedure that adjusts the parameters to shrink that error.

Sinclair College - Dr. Mike Libassi

ŷ=m·x+b

x is the feature, ŷ is the prediction, and the two parameters are the slope m and the intercept b. Training means finding the m and b that fit the data best.

Fit the line yourself

A web team logged the average response time of their server at different loads. Drag the sliders to fit a line through the points, then let gradient descent try. The red segments are the residuals, the gap between each point and the line.

slope m
intercept b
mean squared error
equation
gradient descent steps
0

Measuring "how wrong": the loss function

To improve a model you need one number that says how bad it is. For regression, the standard choice is the mean squared error: square every residual, then average them.

MSE=(1/n) Σ (yi − ŷi)²

Read: "the average, over all n points, of the true value minus the prediction, squared." Squaring makes every error positive and punishes big misses much more than small ones.

Gradient descent: walking downhill

Picture the loss as a landscape where every location is a choice of m and b, and the height is the error. Gradient descent stands somewhere on that landscape, checks which way is downhill (the gradient), and takes a small step. Repeat until the ground is flat.

m←m − α · ∂MSE/∂m
b←b − α · ∂MSE/∂b

α (alpha) is the learning rate, the size of each step. ∂MSE/∂m reads "the partial derivative of the error with respect to m": how much the error changes if m moves a little.

Notice: in the demo, gradient descent fixes the slope quickly but crawls toward the right intercept. That is because the two parameters live on very different scales. Page 05 shows why, and how feature scaling fixes it.

In Python

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)              # X_train shape (n, 1): hundreds of users
print(model.coef_, model.intercept_)   # the learned m and b
model.predict([[12]])                   # predicted response time at 1,200 users

With more than one feature, the same idea becomes ŷ = w1x1 + w2x2 + … + b, called multiple linear regression. The code does not change at all; X simply has more columns.