A random forest ensembles many trees. A voting ensemble goes further and combines entirely different model families, a logistic regression, a decision tree, and an SVM, letting them vote (or, with voting="soft", average their predicted probabilities). The idea is the same one behind the random forest, applied one level higher: models that make different kinds of mistakes cancel each other out more effectively than models that are all variations on the same algorithm.
voting_ensemble.py
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
voters = [
("logistic", LogisticRegression(max_iter=2000)),
("tree", DecisionTreeClassifier(max_depth=5, random_state=11)),
("svm", SVC(probability=True, random_state=11)),
]
ensemble = VotingClassifier(voters, voting="soft")
cv_scores = cross_val_score(ensemble, X_train, y_train, cv=5)
print(f"Ensemble CV accuracy: {cv_scores.mean():.4f} (+/- {cv_scores.std():.4f})")
ensemble.fit(X_train, y_train)
print(f"Ensemble test accuracy: {ensemble.score(X_test, y_test):.4f}")
Sample output
Ensemble CV accuracy: 0.8964 (+/- 0.0243)
Ensemble test accuracy: 0.9028
Individual test accuracies for comparison:
logistic regression: 0.8583
decision tree: 0.8667
SVM: 0.9306
Caution: A Voting Ensemble Is Not Automatically the Best Model
Notice, in the real run above, that the voting ensemble (0.9028) actually scored below the single best member, the SVM (0.9306). Voting helps most when the individual models are roughly comparable and make genuinely different mistakes; when one model is simply stronger than the others on this particular task, averaging it with weaker models can pull its score down. Always compare an ensemble against its own strongest individual member before assuming the ensemble wins.
Try it yourself: Compare All Six on Your Own Split
Using the same X_train, X_test, y_train, y_test throughout, train a decision tree, a random forest, an SVM, a gradient boosting classifier, an XGBoost classifier, and the voting ensemble above, and print each one's test accuracy and 5-fold CV mean. Which one wins changes with the dataset; running the comparison yourself, rather than trusting a rule of thumb, is the point of this exercise.
Choosing Among the Four, in Practice
| Classifier | When to reach for it |
|---|---|
| Decision tree | When you need a model a non-technical stakeholder can read directly, at some cost in raw accuracy |
| Random forest | A strong, low-maintenance default for tabular security features; resistant to overfitting without much tuning |
| Support vector machine | Smaller, cleaner datasets with a genuinely non-linear boundary; slower to train as the dataset grows |
| Gradient boosting (XGBoost) | When squeezing out the last few points of accuracy matters and some tuning time is available; usually the strongest tabular option |
No classifier wins every task; this unit trained all four on the same split specifically so you could compare, not guess.