Every classifier in this course assumes the data it sees at prediction time resembles the data it trained on. An adversarial attack breaks that assumption on purpose, by crafting input specifically to fool a known or guessed model.
| Attack type | What the attacker does |
|---|---|
| Evasion | Modifies a malicious input at prediction time, just enough to cross the model's decision boundary, without changing its underlying malicious function |
| Poisoning | Corrupts the training data itself, so the model learns a flawed boundary before it is ever deployed |
| Model extraction | Repeatedly queries a deployed model to reconstruct a close copy of it, stealing the model without ever accessing its parameters directly |
| Model inversion / membership inference | Infers sensitive details about the training data itself from the model's outputs |
These are the four adversarial attack categories most commonly discussed in the machine learning security literature, including the NIST taxonomy of adversarial machine learning.
Case study: Three Real Adversarial Incidents
Evasion, by image perturbation. In 2014, Goodfellow, Shlens and Szegedy showed that adding a carefully computed, visually near-invisible perturbation to an image of a panda caused a trained neural network to classify it as a gibbon with over 99 percent confidence, introducing the Fast Gradient Sign Method (FGSM) this topic builds toward.
Evasion, against a commercial product. Unit 2 already described how Skylight Cyber's 2019 research moved a malicious file's score on BlackBerry Cylance's AI engine from -852 to +999 simply by appending strings from an unrelated, benign video game executable, exploiting a learned bias in the model's features rather than any traditional software bug.
Poisoning, in deployment. In 2016, Microsoft's Tay chatbot, designed to learn conversational patterns from its interactions on Twitter, was deliberately fed coordinated offensive input by users within hours of launch and began repeating it, forcing Microsoft to take it offline in under a day, a stark demonstration of what happens when a learning system treats adversarial input as ordinary training signal.
ML concept: FGSM, the Idea Without the Full Mathematics
FGSM computes the gradient of the model's loss with respect to the input (not the model's weights, which is what training normally does), then nudges every input feature a small step $\epsilon$ in the direction that most increases the loss, that is, the direction that most confuses the model. The result is the smallest input change, in the direction the model is most sensitive to, that degrades its prediction.
- Never assume a model's training-time accuracy survives contact with an adaptive attacker; evaluate under perturbation, not only on a clean test set.
- Where feasible, train on adversarially perturbed examples alongside clean ones (adversarial training), the defended curve in the chart above.
- Rate-limit and monitor queries to any deployed model; both evasion probing and model extraction require many queries.
- Validate and sanity-check training data sources before retraining in production, the direct lesson of the Tay poisoning incident.
- Treat an unusually confident, unusual-looking prediction as a signal worth a second opinion, not an automatic verdict.