Unit 5 Quick Reference, Self-Study and Lab

A good feature must be discriminative, available at decision time, and robust to trivial evasion. A feature that fails discriminative: the day of the week an email arrived, which rarely differs meaningfully between spam and legitimate mail. A feature that fails available at decision time: whether a file was eventually confirmed malicious by an antivirus vendor weeks later, which does not exist yet when the file first needs to be scored. A feature that fails robust to evasion: checking for the exact phrase "URGENT ACTION REQUIRED" in all capitals, which an attacker defeats by simply changing the capitalisation.

Shannon entropy measures randomness in a byte distribution. Ordinary executable code and readable strings sit well below the theoretical maximum of 8 bits per byte, because real code and text have structure and repetition. A packed or encrypted payload looks statistically close to random data, so its entropy sits close to the maximum, around 7.0 bits per byte or higher, making it a strong, cheap-to-compute single signal that a file is hiding its real content.

A BEC attack is often just a carefully worded email with no malicious attachment and no malware payload at all, impersonating an executive or vendor to request an urgent wire transfer. Since there is frequently no file to analyse and the text itself may read as ordinary business correspondence, structural signals, a mismatch between the sender's display name and actual address, an unusual request pattern, urgency language, become essential, because text content alone may not look suspicious.

Suppose a training dataset of phishing emails is scraped entirely from a corporate abuse-reporting mailbox, and every forwarded report happens to include a specific "Reported-By:" header the reporting tool automatically adds, which legitimate email never has. A model could learn to key heavily on the presence of that header rather than any genuine property of phishing content. It would score near-perfectly on the test set, since the header is present in every phishing example there too, but fail completely in production, where incoming phishing emails never carry that artefact.

WannaCry spread automatically and extremely fast using the EternalBlue exploit, infecting roughly 200,000 computers across 150 countries within days with no phishing click required, which was too fast for a purely file-based static signature response, pushing defenders toward behavioural and network-anomaly detection instead. Mirai's payload was trivial, just default-credential logins to IoT devices, so traditional file-based malware features barely apply to it; detecting it instead requires network-traffic features, specifically unusual outbound scanning on IoT-typical ports.

Structured features are interpretable and cheap to compute but require a human to think of every useful pattern in advance. Lexical TF-IDF features discover patterns automatically, including ones a human would never think to hand-code, but are harder to explain to a non-technical stakeholder and can overfit to coincidental vocabulary in a particular training set. You would prefer structured features when a model's decisions need to be explained to a compliance team or end user; you would prefer lexical features when you have a large labelled dataset and want to catch novel phishing patterns a hand-written rule set would miss.

It contains 5,574 real, hand-labelled English SMS messages with a realistic spam rate of about 13 percent, making it small enough to train on a laptop in seconds while still being realistic enough that a model's performance on it transfers meaningfully to real deployment. Its small size makes it ideal for quickly testing a pipeline end to end before scaling up to a much larger production dataset.