Unit Summary and Quick Reference
| Topic | Core idea |
|---|---|
| Why accuracy lies | On imbalanced data, a classifier that predicts the majority class scores high accuracy while catching nothing |
| Precision, recall, F1 | Built directly from the confusion matrix; precision and recall trade against each other by threshold |
| Adversarial attack taxonomy | Evasion, poisoning, extraction, and inference, each targeting a different stage of the ML lifecycle |
| Three real incidents | Goodfellow's 2014 FGSM panda/gibbon result, the 2019 Cylance bypass, and the 2016 Microsoft Tay poisoning |
| NSL-KDD and CICIDS2017 | The field's two standard benchmarks; NSL-KDD is older and cleaner, CICIDS2017 is newer and more realistic |
| Threat intel automation | The same TF-IDF pipeline from Unit 5, applied to triaging incoming intelligence reports at scale |
Self-Study Questions
Q1. Explain why a 99.8% accurate fraud classifier can still be a useless model, using precision and recall to make the argument precise.
A classifier that predicts "legitimate" for every single transaction in a dataset that is 99.8 percent legitimate scores 99.8 percent accuracy while having TP = 0: it never correctly flags a single fraudulent transaction. Its recall, TP / (TP + FN), is therefore 0, meaning it catches none of the actual fraud, which is the entire point of building the model. Accuracy alone hides this completely, while recall exposes it immediately.
Q2. Name and briefly describe the four categories of adversarial attack introduced in this unit.
Evasion modifies a malicious input at prediction time, just enough to cross the model's decision boundary, without changing what the input actually does. Poisoning corrupts the training data itself, so the model learns a flawed boundary before deployment. Model extraction repeatedly queries a deployed model to reconstruct a close copy of it without ever accessing its parameters. Model inversion or membership inference infers sensitive details about the training data itself from the model's outputs.
Q3. Using the three real incidents in this unit, match each one to the correct adversarial attack category from the taxonomy.
Goodfellow's 2014 panda-to-gibbon result and the 2019 Cylance bypass are both evasion attacks, since each one modified an input at prediction time to cross a trained model's decision boundary without changing the input's underlying function. The 2016 Microsoft Tay incident is a poisoning attack, since the chatbot's own training process was corrupted by coordinated malicious input it was learning from live.
Q4. Why does a strong result on NSL-KDD deserve more scrutiny today than the same result would have deserved in the early 2000s?
NSL-KDD's underlying traffic was captured during a 1998 DARPA evaluation and is now over two decades old; it does not reflect modern attack techniques or modern network protocols. A strong result on it today demonstrates that a model can learn that specific, dated dataset's patterns, not that it would perform well against current attack traffic, which is exactly why newer benchmarks like CICIDS2017 and UNSW-NB15 are now preferred for current research.
Q5. In this unit's real WAF evasion demo, the payload's score dropped from 0.9594 to 0.2836 by appending a single repeated character. Explain why this works, and which category in the adversarial taxonomy it belongs to.
The model had learned, from its training data, that a specific character n-gram was strongly associated with benign queries. Appending 150 copies of that character pulled the TF-IDF-weighted feature vector strongly toward the benign direction the model had learned, without altering the malicious XSS payload itself in any way that would stop it from executing. This is an evasion attack: the input was modified at prediction time, specifically to cross the decision boundary, while its underlying malicious function was left completely untouched.
Q6. Explain why the poisoning attack in this unit specifically depends on the model being retrained online with partial_fit, and name one other real system with this same exposure.
The attack works by repeatedly calling partial_fit with a small cluster of mislabelled chaff points, which only has an effect because the model keeps learning after its initial deployment rather than staying fixed. A model trained once and never retrained could not be bent this way by the same mechanism. Any system that retrains automatically on live, unvetted data carries the same exposure, for example a spam filter that learns from users' own "spam" or "not spam" clicks.
Q7. Describe, in your own words, how a precision-recall curve lets a team choose an operating threshold based on a business constraint (for example, "recall must be at least 90%") rather than an arbitrary default.
A precision-recall curve plots precision against recall at every possible decision threshold, so a team can start from a recall figure the business has decided it needs, such as catching at least 90 percent of real fraud, find the point or points on the curve that achieve at least that recall, and then read off the precision, and therefore the false-alarm rate, that recall actually costs at that threshold. This turns the threshold choice into an informed trade-off against a stated business requirement, rather than leaving the classifier at whatever default threshold, typically 0.5, happens to come out of the library.
Lab: Find Your Own Operating Point
- Train and score. Using any classifier from this course on a dataset with a rare positive class, such as the fraud or intrusion data from earlier units, compute
precision_recall_curveexactly as shown in this unit. - Set the business constraint. Decide on a minimum acceptable recall, such as 90%, matching the constraint this unit's own hands-on exercise used.
- Find the threshold. Search the
thresholdsarray returned byprecision_recall_curvefor the threshold that first achieves your minimum recall, and report the exact precision at that threshold. - Translate to a business statement. Convert the precision you found into a concrete operational statement, such as "at this threshold, roughly 1 in N flagged transactions is a false alarm," and state whether that false-alarm rate is realistic for a human team to review.
- Compare against average precision. Report the
average_precision_scorefor your model and compare it to this unit's own real result (Random Forest AP = 0.54 against a random classifier's AP = 0.05) to judge how your model's overall ranking quality compares.