Null and Alternative Hypotheses
Unit 1 introduced the hypothesis as a tentative, testable statement. This topic makes that testing precise enough to act on.
The null hypothesis (H₀) is the default position: no effect, no difference, no association. "The mean detection time of the new IDS rule set is the same as the baseline." The alternative hypothesis (H₁) is what the researcher suspects might actually be true. "The mean detection time of the new rule set is different from the baseline."
A test never proves H₀ true. It either finds enough evidence to reject H₀, or it doesn't find enough evidence and fails to reject H₀, which is a weaker claim than "H₀ is confirmed."
Significance Level and p-value
The significance level (α) is the risk of rejecting H₀ when it's actually true, that the researcher accepts in advance. Conventionally set at 0.05, decided before the data are collected, not after.
The p-value is the probability of seeing data at least as extreme as what was actually observed, if H₀ were true. When the p-value falls below α, the result is called statistically significant.
A two-tailed test is used when the alternative hypothesis only claims a difference, in either direction. A one-tailed test is used when the alternative claims a direction, such as "faster than." Choosing one-tailed after already seeing the data, purely to make the p-value smaller, is a well-known way to mislead a reader and should never be done.
Two Ways a Test Can Go Wrong
A test can go wrong in exactly two ways, and this borrows directly from the confusion matrix used to score a classifier:
| H₀ is actually true | H₀ is actually false | |
|---|---|---|
| Reject H₀ | Type I error (false positive) | Correct decision |
| Fail to reject H₀ | Correct decision | Type II error (false negative) |
This parallel is worth taking seriously rather than treating as a teaching trick. Setting α lower reduces Type I error but raises Type II error, in exactly the same way that tightening an IDS's alert threshold cuts false alarms at the cost of missing more genuine attacks. A researcher choosing α = 0.01 instead of the usual 0.05 is making the same trade-off a SOC manager makes when tuning a detection rule.
A One-Sample Test, Applied
A SOC has historically taken an average of 42 seconds to detect a certain attack pattern. After deploying a new detection rule, the team times 25 detections and runs a one-sample test comparing the new average against the historical baseline, at the standard 5% significance level. Suppose the test fails to reach significance. The honest conclusion is not that the new rule made no difference. It's that 25 timed detections weren't enough to be confident, at the 5% level, that any drop was more than ordinary variation. The next step is a larger sample, not a more relaxed significance level chosen after the fact to force a "significant" result.
Effect Size
A p-value only answers "is this likely to be more than chance," and a large enough sample can make even a tiny, practically meaningless difference statistically significant. Effect size answers the question a p-value can't: how big is the difference? Cohen's d, a common measure for comparing two means, is conventionally read as small around 0.2, medium around 0.5, and large around 0.8, though Cohen himself intended these as a rough last resort, not a rigid rule, and what counts as a meaningful effect in security research depends on what the difference would actually cost or save in practice. Reporting effect size alongside the p-value, not instead of it, is what lets a reader judge whether a statistically significant improvement is also an operationally worthwhile one.
Check Your Understanding
Explain why a hypothesis test can reject H₀ but can never prove H₀ true, and what the difference is between "failing to reject H₀" and "confirming H₀." Then, using the Type I / Type II error table above, explain what it would mean for a research team to set an unusually strict significance level of α = 0.001 instead of the conventional 0.05: what is the team trading away to get that stricter threshold?