Page 10 of 14 · Core techniques

Unsupervised learning: finding structure without answers

Labels are expensive. Someone has to review each example and decide what it is. Unsupervised learning works with no labels at all: it groups similar items together and points out the ones that do not fit anywhere. In security, those misfits are often exactly what you are hunting for.

Sinclair College - Dr. Mike Libassi

k-means clustering, one step at a time

Each point is a host on a network, described by how many connections it makes per minute and how much data it sends. Nobody has labeled them. k-means tries to sort them into k groups by repeating two steps:

1. assign

Each point joins the cluster whose center (centroid) is closest.

2. update

Each centroid moves to the average position of the points assigned to it.

number of clusters k

Flag anomalies

iteration

next step

inertia (spread)

How many clusters? The elbow method

k-means needs you to choose k. Inertia is the total squared distance from each point to its centroid, a measure of how tight the clusters are. It always drops as k grows (with k equal to the number of points, it reaches zero), so look for the elbow, where adding another cluster stops helping much.

Each value is the best of 10 random starts, because k-means can get stuck in a poor arrangement depending on where the centroids begin. Press "New random start" above a few times with k = 3 and you may catch one.

Anomaly detection

Once you know what normal groups look like, anything far from every centroid is suspicious. Turn on Flag anomalies in the demo: the red rings mark hosts that sit far from their nearest centroid, measured in units of that cluster's typical spread. A host sending large amounts of data with few connections could be exfiltrating files; a host making thousands of tiny connections could be scanning.

Unsupervised does not mean automatic. The model only says "this is unusual." A new backup server is unusual too. An analyst still decides whether an anomaly is malicious, and those decisions can become labels for a supervised model later.

Other unsupervised tools

DBSCAN

Finds clusters of any shape based on density, and labels sparse points as noise automatically. No k needed.

PCA

Principal component analysis compresses many features into a few that keep most of the variation. Great for plotting high-dimensional data.

isolation forest

Built for anomaly detection: outliers are easier to isolate with random splits, so they get high anomaly scores.

autoencoders

Neural networks (page 11) that learn to compress and rebuild normal data; inputs they rebuild badly are anomalies.

from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.ensemble import IsolationForest

X_s = StandardScaler().fit_transform(X)        # distances need features on the same scale
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X_s)
print(km.labels_[:10], km.inertia_)

iso = IsolationForest(contamination=0.02).fit(X_s)
flags = iso.predict(X_s)                         # -1 = anomaly, 1 = normal

Unsupervised learning finds structure, such as groups and outliers, without labels.
k-means alternates assigning points and moving centroids; choose k with the elbow method and scale your features first.
Anomaly detection flags the unusual, not necessarily the malicious; a human still makes the call.