ML concept: Reading a Confusion Matrix
Every classification metric in this unit is computed from four counts: true positives (TP, correctly flagged malicious), false positives (FP, legitimate wrongly flagged), true negatives (TN, correctly passed), and false negatives (FN, malicious wrongly passed). Unit 2's confusion matrix was built from exactly these four counts.
| Metric | Formula | What it answers |
|---|---|---|
| Precision | TP / (TP + FP) | Of everything flagged malicious, how much actually was? High precision means few false alarms. |
| Recall | TP / (TP + FN) | Of everything actually malicious, how much did the model catch? High recall means few misses. |
| F1-score | 2 · (Precision · Recall) / (Precision + Recall) | The harmonic mean of precision and recall, a single number balancing both. |
These are the three metrics every security classifier should be reported with, alongside accuracy, never instead of it, but never only accuracy either.
precision_recall_f1.py
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score
y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall: {recall_score(y_test, y_pred):.3f}")
print(f"F1-score: {f1_score(y_test, y_pred):.3f}")
Caution: Precision and Recall Trade Against Each Other
Lowering a classifier's decision threshold catches more true positives (raising recall) but also lets in more false positives (lowering precision), and raising the threshold does the reverse. There is no threshold that maximises both at once; the right balance depends on the cost of a miss versus the cost of a false alarm in your specific deployment, a judgement call no metric makes for you.
Tool in focus: Precision-Recall Curve, Not Just ROC
Unit 2 used the ROC curve (true positive rate against false positive rate). On severely imbalanced data, ROC can look deceptively good, because the false positive rate stays low even when the raw count of false positives is large, relative to the small number of true positives. A precision-recall curve, plotting precision against recall across thresholds, is the more honest picture when the positive class is rare, exactly the situation fraud and intrusion detection are almost always in.
precision_recall_curve.py
from sklearn.metrics import precision_recall_curve, average_precision_score
proba = model.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, proba)
ap = average_precision_score(y_test, proba)
print(f"Average precision: {ap:.3f}")
Try it yourself: Find Your Own Operating Point
Using the thresholds array from the code above, find the threshold that gives at least 90% recall, then report the precision at that exact threshold. This is the real workflow a fraud or intrusion-detection team uses to set a production decision threshold: start from the recall the business can tolerate, then see what precision, and therefore what false-alarm rate, that recall costs.