Real network and host telemetry routinely has dozens of features. Humans cannot plot forty dimensions, and many algorithms, including k-means, perform worse as dimensionality grows (the "curse of dimensionality": in high dimensions, most points end up roughly equidistant from each other, so distance stops being informative). Dimensionality reduction projects high-dimensional data down to two or three dimensions that preserve as much of the original structure as possible.
ML concept: PCA versus t-SNE
PCA (Principal Component Analysis) finds the directions of greatest variance in the data and projects onto them. It is fast, deterministic, and the projection has a precise mathematical meaning (each component is a weighted sum of the original features). t-SNE optimises a projection that preserves local neighbourhoods, often producing visually cleaner, more separated clusters, but it is slower, stochastic (a different run can look different), and its distances are not meaningful between widely separated clusters. Use PCA first, always; reach for t-SNE only when you specifically need a visualisation and do not need to reproduce exact distances.
pca_projection.py
from sklearn.decomposition import PCA
pca = PCA(n_components=2, random_state=5)
projected = pca.fit_transform(scaled_features)
print(f"Variance kept: {pca.explained_variance_ratio_.sum():.1%}")
On a real NSL-KDD-style intrusion dataset, this course's own run kept 36.3% of the variance in principal component 1 and 21.2% in principal component 2, 57.5% combined from just two numbers out of eighteen original features.
tsne_projection.py
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, perplexity=30, random_state=5, init="pca")
projected_tsne = tsne.fit_transform(scaled_features)
Caution: Do Not Cluster on a t-SNE Projection
It is tempting to run k-means directly on a t-SNE projection because the clusters look so clean. Resist this: t-SNE deliberately distorts distances to make local neighbourhoods visually clear, so a clustering algorithm run on its output can find clusters that are artefacts of the projection, not of the real data. Cluster on the original (or PCA-reduced) features; use t-SNE only to look at the result afterward.