Not every anomaly detector needs to be a trained model. User and Entity Behavior Analytics (UEBA) products commonly use a much simpler idea alongside, or instead of, the methods above: score an event by how rare it is for the specific user or host involved, not how far it sits from a fitted boundary.
- For each user or host, maintain a historical count of how often each distinct activity (a privilege used, a host contacted, a file accessed) has occurred.
- For a new event, compute its rarity: how small a fraction of this entity's history this specific activity represents.
- Map rarity to a score, typically with sharply increasing severity as rarity approaches zero (an activity never seen before for this entity scores far higher than one seen occasionally).
- Combine the rarity score with other signals (time of day, peer-group comparison) before alerting, since rarity alone produces many false positives for any user whose role is simply varied.
rarity_scorer.py
from collections import Counter
def rarity_score(entity_history: Counter, activity: str) -> float:
total = sum(entity_history.values())
if total == 0:
return 1.0
frequency = entity_history[activity] / total
return 1.0 - frequency # closer to 1.0 = rarer = more suspicious
# example: an admin who has used sudo 400 times, but never before
# touched this specific, sensitive configuration file
history = Counter({"sudo": 400, "ssh": 120, "edit_config_a": 3})
print(rarity_score(history, "edit_config_b")) # never seen: scores 1.0
print(rarity_score(history, "sudo")) # routine: scores low
Caution: Rarity Is Cheap, but Noisy
A rarity scorer needs no training data, no feature engineering, and no model maintenance, which is exactly why it is so widely deployed. Its weakness is equally simple: anything a user does for the first time scores as maximally suspicious, including entirely legitimate new activity (a new project, a new tool, a role change), so rarity scoring is almost always paired with a second signal, exactly as the algorithm box above recommends, rather than used to alert on its own.
Case study: Unsupervised Detection in Commercial Network Defence
Several commercial network detection and response products, publicly including Darktrace, describe their core approach as building an unsupervised statistical model of what is "normal" for a specific network, then flagging activity that deviates from that learned baseline, explicitly without relying on signatures of known attacks. The approach's strength is exactly the one this unit has built: it can flag a genuinely novel attack technique the day it first appears, because novelty itself is the signal, not a match against a known pattern. Its weakness is the same one every unsupervised method shares: a baseline learned during a quiet period can itself be wrong, and a slow, patient attacker who changes behaviour gradually can shift the learned "normal" without ever triggering a large deviation.
Try it yourself: Find the Right eps for DBSCAN
Run DBSCAN on the same traffic dataset for eps values of 0.3, 0.6, and 1.0, holding min_samples fixed. Count the number of clusters and the number of points labelled as outliers at each setting. You will see directly why DBSCAN's parameters need to be tuned per dataset, not copied from a tutorial.