Unit Summary and Quick Reference
| Topic | Core idea |
|---|---|
| Decision trees | Split on the feature and threshold that most reduces impurity at each node; easy to overfit without depth limits |
| Random forests | Average many trees trained on random samples and feature subsets; cancels overfitting through weak correlation |
| Support vector machines | Find the widest-margin boundary; the kernel trick draws curved boundaries without computing high dimensions directly |
| Gradient boosting | Trains trees sequentially to correct prior errors; usually the strongest tabular classifier, at a tuning cost |
| Endpoint protection case study | Commercial AV vendors adopted statistical classifiers over pure signatures, then had to contend with the new failure mode of model-evasion attacks |
Self-Study Questions
Q1. Explain, in your own words, why an unconstrained decision tree tends to overfit, and name two hyperparameters that limit this.
An unconstrained tree keeps splitting until every leaf is perfectly pure, including leaves that contain a single training example, which means it has memorised the training set rather than learned a generalisable pattern. max_depth and min_samples_leaf both limit this by stopping the tree from growing arbitrarily deep or from creating leaves with too few examples to be a reliable rule.
Q2. Why does a random forest typically outperform any single tree it contains, even though each individual tree may be a weak model?
Each tree is trained on a different random bootstrap sample and a different random subset of features at each split, so the trees' individual errors are only weakly correlated with each other. Averaging many weakly correlated errors cancels out most of the noise, so the forest's combined prediction is more accurate and more stable than any one tree's prediction on its own.
Q3. What does the kernel trick let a support vector machine do that a plain linear boundary cannot?
The kernel trick implicitly projects the data into a higher-dimensional space where a straight boundary does separate the classes, without ever computing that higher-dimensional space directly. This lets an SVM draw smooth, curved boundaries around clusters of either class, which a plain linear boundary could never do on data that is not linearly separable.
Q4. Describe, step by step, how gradient boosting's sequential tree-building differs from a random forest's independent tree-building.
A random forest trains all of its trees independently, each on its own random sample and feature subset, and then averages their predictions. Gradient boosting instead trains its trees one after another: the first tree is trained on the original labels, and every tree after that is trained specifically to correct the residual error the trees before it are still making, so the trees build on each other rather than voting as independent equals.
Q5. Why did the real voting ensemble in this unit score lower than its own best individual member, and what does that tell you about when ensembling actually helps?
The ensemble combined a strong SVM (0.9306) with a weaker logistic regression (0.8583) and a weaker decision tree (0.8667), and averaging pulled the combined score (0.9028) down below the SVM alone. This shows that voting helps most when the individual models are roughly comparable in strength and make genuinely different mistakes; when one model is clearly the strongest, averaging it with weaker ones can hurt rather than help.
Q6. Explain why a single train/test split can be misleading on its own, and how k-fold cross-validation addresses that specific problem.
A single split's accuracy number depends partly on which particular rows happened to land in the test set, so the same model could look stronger or weaker purely by chance. k-fold cross-validation trains and evaluates the model k separate times, each time holding out a different fold as the test set, and reports the mean and standard deviation across all k runs, which is a far more honest picture of how the model actually performs than any one split alone.
Q7. Using the real ULB credit card dataset's numbers from this unit, explain precisely why a 99.83% accurate fraud classifier can still be worthless, in terms a non-technical manager would understand.
Out of 284,807 real transactions in the ULB dataset, only 492 are actually fraudulent, just 0.17 percent. A classifier that simply labels every single transaction "legitimate" never flags a single fraud, yet it would still be correct 99.83 percent of the time, because it is right on every one of the overwhelming majority of legitimate transactions. The headline accuracy number looks excellent while the classifier is doing the one job it exists for, catching fraud, precisely zero percent of the time.
Lab: Compare All Six Classifiers on One Split
- Build one shared split. Using any labelled malware or phishing dataset you have engineered features for, create a single
X_train, X_test, y_train, y_testsplit and reuse it for every model below, so the comparison is fair. - Train all six. Train a decision tree, a random forest, an SVM, scikit-learn's gradient boosting classifier, the real XGBoost library, and the voting ensemble from this unit, all on the same split.
- Score each one two ways. For each model, print its single test-split accuracy and its 5-fold cross-validated mean and standard deviation.
- Find the plateau. Run a small
GridSearchCVover your strongest model's two most important hyperparameters and identify whether a plateau of near-best combinations exists, the way this unit's own SVM grid search found one. - Write the recommendation. In two or three sentences, recommend which of the six models you would actually deploy, citing both its accuracy and a non-accuracy reason, such as interpretability or training cost, from the choosing-among-the-four table in this unit.