Page 02 of 14 · Foundations

Every ML system is a pipeline, not just a model

The model gets the headlines, but most of the work in a real ML project happens before and after training. Data has to be collected, cleaned, turned into features, split, and watched after the model goes live.

Sinclair College - Dr. Mike Libassi

Follow an intrusion detector through the pipeline

Select a stage. The running example is a model that flags malicious network connections.

Rows, features, and labels

Almost every supervised dataset is a table. Each row is one example (here, one network connection). Each feature column describes the example, and the label column holds the answer the model should learn to predict.

duration_s	src_bytes	dst_port	protocol	failed_logins	label
0.4	512	443	tcp	0	normal
12.9	48,210	22	tcp	7	attack
0.1	64	53	udp	0	normal
0.0	0	3389	tcp	0	attack
2.3	1,980	80	tcp	0	normal

Select a column header to see what role it plays.

Select a column header above.

In math notation the feature columns form a matrix X and the labels form a vector y:

X:n rows × d features y:n labels

This table has n = 5 and d = 5. Real intrusion datasets have millions of rows and dozens of features.

Train, validation, and test sets

You cannot grade a model on the same data it studied, any more than you can grade students with the exact questions from the practice exam. So the data is split three ways:

training set

The model learns its parameters from these rows.

validation set

Used while building, to compare settings and choose the best version.

test set

Locked away until the very end. One final, honest score.

Split 100 logged connections

The raw log is in time order, and the attack campaign started near the end of the collection window. Each square is one connection; A marks an attack.

validation15%

test15%

Shuffle before splitting

Data leakage: the silent failure

Leakage happens when information from outside the training data sneaks into training, so the model looks brilliant in testing and then fails in the real world. Common causes:

A feature that is really the answer in disguise, such as a blocked_by_firewall column that was only filled in after an analyst confirmed the attack.
Scaling or cleaning the full dataset before splitting it, so statistics from the test rows leak into training.
Duplicate rows that land in both the training and test sets.
Shuffling time-ordered data so the model trains on tomorrow's attacks and is tested on yesterday's.

In security work: threats change over time, so the honest test is often a time-based split. Train on older traffic and test on newer traffic, the same way the model will be used after deployment.

In Python

scikit-learn does the split in one line. Page 03 walks through this toolkit step by step.

from sklearn.model_selection import train_test_split

# 70% train, 15% validation, 15% test, keeping the attack ratio the same in each
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=42)

A model is one stage in a pipeline that runs from data collection to monitoring.
Features go in, labels come out. Choosing and preparing features often matters more than choosing the algorithm.
Keep training, validation, and test data separate, and watch for leakage between them.