Unit Summary and Quick Reference
| Topic | Core idea |
|---|---|
| Why unsupervised matters here | Most security telemetry has no label; intrusion detection leans on structure and anomaly, not classification |
| k-means | Partitions into $k$ clusters by minimising distance to centroids; needs scaled features and a chosen $k$ |
| DBSCAN | Density-based clustering; finds arbitrary shapes and explicitly flags sparse points as noise, no $k$ required |
| PCA vs t-SNE | PCA: fast, deterministic, meaningful distances. t-SNE: slower, stochastic, visually cleaner but not metrically faithful |
| Purpose-built detectors | Isolation Forest, One-Class SVM, and Local Outlier Factor each define "anomalous" differently |
| Unsupervised NDR in practice | Commercial tools like Darktrace build a learned baseline of normal and flag deviation, catching novel attacks at the cost of a baseline that can itself drift wrong |
Self-Study Questions
Q1. Explain the difference between clustering and anomaly detection as goals, even though the same algorithm can sometimes serve both.
Clustering groups similar points together to describe the overall structure of a dataset, such as discovering that a network carries three broad traffic patterns. Anomaly detection instead asks a narrower question about each individual point: how unusual is this one relative to everything else? A point can simultaneously be the sole member of its own tiny cluster and the exact anomaly a security team cares about.
Q2. Why must features be standardised before running k-means, and what goes wrong if they are not?
k-means measures distance using raw Euclidean distance across all features at once. If one feature is measured in the thousands (such as bytes transferred) and another in single digits (such as TCP flags set), the large-scale feature will dominate the distance calculation entirely, effectively making k-means ignore the smaller-scale feature regardless of how informative it actually is. Standardising every feature first puts them all on a comparable scale.
Q3. Give one concrete advantage DBSCAN has over k-means for intrusion detection specifically, and one disadvantage.
Advantage: DBSCAN explicitly labels sparse, isolated points as noise (the -1 label) rather than forcing every point into the nearest cluster, which is exactly the outlier flag a security analyst wants, and it needs no pre-chosen number of clusters. Disadvantage: its eps and min_samples parameters need to be tuned per dataset, and a poor choice can merge genuinely separate clusters or fragment one real cluster into many.
Q4. Why would you choose PCA over t-SNE before a production pipeline, even if t-SNE's plot looked visually more convincing?
PCA is fast, deterministic, and each of its components has a precise mathematical meaning as a weighted sum of the original features, so the same data produces the same projection every time and the result can be reasoned about numerically. t-SNE is slower, stochastic (a different run can look different), and its distances are not meaningful between widely separated clusters, which makes it unsuitable for anything beyond a one-off visualisation, not a pipeline a production system depends on.
Q5. The real Isolation Forest result on resampled KDD data scored only 28% precision and 28% recall on attacks, at 95.7% overall accuracy. Explain what this result does, and does not, tell you about Isolation Forest as an algorithm.
The result tells you that Isolation Forest run out of the box, on raw numeric features with no feature engineering, catches barely more than a quarter of real attacks in this specific setting, and that a high overall accuracy number is misleading when the attack class is rare. It does not tell you that Isolation Forest is a weak algorithm in general; the course explicitly notes that the same algorithm, given better-engineered features (Unit 5's subject), performs substantially better, so the result is primarily an argument for feature engineering, not a verdict on the algorithm itself.
Q6. Describe the rarity-scoring approach used in UEBA products, and explain why it is normally combined with a second signal rather than used alone.
Rarity scoring maintains a historical count of how often each user or host has performed each distinct activity, then scores a new event by how small a fraction of that entity's history the activity represents, so a never-before-seen activity scores close to maximum suspicion. It is normally combined with a second signal, such as time of day or peer-group comparison, because rarity alone flags any entirely legitimate first-time activity, a new project, a new tool, a role change, just as highly as a genuine threat, producing many false positives if used by itself.
Q7. Why does fetching the "10 percent" KDD Cup 99 subset directly with scikit-learn require resampling before it can be used to illustrate realistic intrusion detection?
The raw subset is 80.31 percent attack traffic, because the original KDD Cup 99 collection deliberately oversampled certain denial-of-service attacks when it was assembled. Genuine enterprise network traffic is nowhere near 80 percent malicious, so training or reporting results on the unmodified distribution produces numbers that look strong but do not reflect how a detector would actually perform on real traffic; resampling down to a realistic attack rate, such as the roughly 3 percent used in this unit, is necessary first.
Lab: Tune DBSCAN's eps Parameter
- Prepare a shared dataset. Using the same standardised network-flow features from this unit's k-means example, hold
min_samplesfixed at a reasonable value such as 8. - Sweep eps. Run DBSCAN three times, with
epsset to 0.3, 0.6, and 1.0, and record the number of clusters found and the number of points labelled-1(outliers) at each setting. - Compare against k-means. Run k-means with
n_clusters=3on the same data and compare its cluster assignments to DBSCAN's best-performing eps setting. Which algorithm more cleanly separates any injected or naturally occurring outliers? - Write the recommendation. In two or three sentences, state which eps value you would use in production and why, referencing the trade-off between too many small fragmented clusters (eps too small) and one merged cluster that swallows real structure (eps too large).