Why Accuracy Lies on Security Data

The real ULB fraud dataset's 0.17 percent positive class, and why a classifier that predicts legitimate for every single transaction scores 99.8 percent accuracy while catching zero fraud.

~5 min total·4 quadrants of structured content

By the end of this topic, you will

State the real ULB dataset's legitimate-to-fraud ratio and the accuracy a classifier that predicts nothing but the majority class would score
Explain why a high accuracy number can coexist with a completely useless classifier
Explain why this problem is specific to imbalanced data rather than classification in general

Quadrant 1 · e-Tutorial

Video lectures and walkthroughs

Video content coming soon.

Quadrant 2 · e-Content

Articles and case studies

Why Accuracy Lies on Security Data

A classifier that has learned absolutely nothing scores 99.8 percent accuracy on a realistic fraud dataset. That number sounds excellent and is worthless, because it catches zero fraud, the entire point of building the model in the first place.

Quadrant 3 · Web Resources