> ML SERIES :: 08 logistic regression

Page 08 of 14 · Core techniques

Classification: from a score to a yes or no

Regression predicts a number. Classification predicts a category: spam or not, malicious or benign, compromised or safe. Logistic regression, despite its name, is the classic classification model, and it is built from a linear model plus one S-shaped function.

Sinclair College - Dr. Mike Libassi

The sigmoid squashes any number into a probability

A linear model can output any value from minus infinity to plus infinity. The sigmoid function maps that score, called z, into the range 0 to 1, which we can read as a probability.

score z
σ(z)=1 / (1 + e−z)
σ()=

Draw the decision boundary

Each point is a login. The model combines two features into a score, z = w1x1 + w2x2 + b, then applies the sigmoid. Wherever the probability is exactly 0.5 lies the decision boundary, a straight line. Adjust the weights by hand, then let training find them.

w1 (failed logins)
w2 (location risk)
bias b
normal login (circle)compromised account (square)shading: predicted probability of compromise
accuracy
log loss
training steps
0

The loss for classification: log loss

Mean squared error works poorly for probabilities. Classification uses log loss (also called binary cross-entropy), which punishes confident wrong answers very harshly.

loss=−[ y·log(p) + (1−y)·log(1−p) ]

If the true label is y = 1 and the model says p = 0.99, the loss is about 0.01. If it says p = 0.01, the loss is about 4.6. Averaged over every example, this is the number gradient descent minimizes.

More than two classes

To sort malware into five families instead of two, logistic regression generalizes with the softmax function: one score per class, turned into probabilities that add up to 1. The same idea appears again at the output of neural networks.

In Python

from sklearn.linear_model import LogisticRegression

model = LogisticRegression().fit(X_train, y_train)
print(model.coef_, model.intercept_)       # w1, w2 and b
model.predict_proba(X_test)[:, 1]          # probability of class 1 (compromised)
model.predict(X_test)                       # 0 or 1, using a 0.5 threshold
Note: scikit-learn's LogisticRegression applies L2 regularization by default (page 07). The setting C is 1/λ, so a smaller C means stronger regularization.