Page 09 of 14 · Core techniques

Evaluating models: why 95% accuracy can be useless

An intrusion detector sees 1,000 connections, and 50 of them are attacks. A "model" that labels every connection as normal is right 950 times. That is 95% accuracy, and it has caught zero attacks. Good evaluation means choosing metrics that match what you actually care about.

Sinclair College - Dr. Mike Libassi

Move the threshold, watch the trade-off

The model gives each connection a probability of being an attack. The threshold decides where "suspicious enough" begins. Page 08 used 0.5, but nothing says it has to be.

threshold

Swap in the "always normal" model

950 normal connections50 attacks (bars scaled up 5× so you can see them)

Confusion matrix

	predicted attack	predicted normal
actual attack
actual normal

TP true positive: attack caught. FN false negative: attack missed. FP false positive: false alarm. TN true negative: normal traffic left alone.

ROC curve

The metrics, in plain words

Metric	Formula	Answers the question
accuracy	(TP+TN) / all	What share of all predictions were right? Misleading when classes are imbalanced.
precision	TP / (TP+FP)	When the model raises an alert, how often is it a real attack?
recall	TP / (TP+FN)	Of all real attacks, how many did the model catch? Also called sensitivity or detection rate.
F1 score	2·P·R / (P+R)	One number that balances precision and recall; low if either one is low.
false positive rate	FP / (FP+TN)	What share of normal traffic gets wrongly flagged?
ROC AUC	area under ROC	How well does the model rank attacks above normal traffic across all thresholds? 1.0 is perfect, 0.5 is a coin flip.

Which metric matters?

It depends on which mistake costs more. Decide which metric each team should prioritize.

Cross-validation: a more reliable score

A single validation split can be lucky or unlucky, especially with small datasets. k-fold cross-validation splits the training data into k parts (often 5), trains k times with a different part held out each time, and averages the k scores. You get a more stable estimate plus a sense of how much the score varies.

from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.model_selection import cross_val_score

probs = model.predict_proba(X_test)[:, 1]
y_pred = (probs >= 0.3).astype(int)           # a custom threshold
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))  # precision, recall, F1 per class
print(roc_auc_score(y_test, probs))

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
print(scores.mean(), scores.std())

Alert fatigue is a real cost. A detector with 99% recall and a 5% false positive rate sounds great until it scans a million events a day: that is 50,000 false alarms for analysts to dismiss. In a SOC, precision is often what decides whether a model gets used at all.

Accuracy hides failures on rare classes. Use precision, recall, and F1 for imbalanced problems.
The threshold trades precision against recall; choose it based on the cost of each kind of mistake.
ROC AUC measures ranking quality across every threshold; cross-validation gives a more stable estimate.