Stating What You Don't Claim
Every study needs to state what it does not claim. This is not an apology: a study with no stated boundary is one whose claims cannot be checked by anybody, including the person who ran it.
Scope is the boundary the researcher chose on purpose: which dataset, which population, which time window, which threat model, which metric.
Limitations are constraints the researcher did not choose but must disclose anyway: a small sample, an imbalanced dataset, a single-vendor test environment, or simply not enough time to run a larger study.
A limitations section written properly is a service to the next researcher: it says exactly which follow-up study would extend the finding, and it stops the current claim from being read more broadly than the evidence actually supports. A malware classifier that reaches 99% accuracy on one organisation's endpoint logs from a single quarter has a result worth reporting, provided the report says, in the same breath, that the figure hasn't yet been checked against a different organisation, a different malware family mix, or a different time period.
Further Reading
- M. Antonakakis et al., "Understanding the Mirai Botnet," 26th USENIX Security Symposium, 2017, pp. 1093-1110.
- C. Rossow et al., "Prudent Practices for Designing Malware Experiments: Status Quo and Outlook," IEEE Symposium on Security and Privacy, 2012, pp. 65-79. A paper this course returns to when it covers research design in Unit 2.
- Reporting on the WannaCry outbreak and the discovery of its kill switch, May 2017, widely covered by Microsoft's own security blog and by independent researchers including Marcus Hutchins.
Check Your Understanding
A student claims a malware classifier is "99% accurate" after testing it on 200 samples collected from one antivirus vendor's public feed. Write a scope statement and a limitations statement for this claim.