A random forest builds its trees independently and averages them. Gradient boosting builds trees sequentially: each new tree is trained specifically to correct the errors the trees before it are still making, by fitting to the residual error rather than the original label. The result is usually the strongest tabular classifier available before reaching for a neural network, which is why gradient boosting libraries, most popularly XGBoost, dominate security and fraud-detection competitions and production systems alike.
gradient_boosting_sklearn.py
from sklearn.ensemble import GradientBoostingClassifier
gbm = GradientBoostingClassifier(
n_estimators=200, max_depth=3, learning_rate=0.1, random_state=11
)
gbm.fit(X_train, y_train)
print(f"Test accuracy: {gbm.score(X_test, y_test):.3f}")
Tool in focus: The Real XGBoost Library
The syllabus names XGBoost specifically, and it is worth meeting directly rather than only through scikit-learn's own gradient boosting. XGBClassifier implements the same sequential-correction idea with a more heavily optimised, regularised training procedure, which is why it has become the default choice in fraud and malware-classification competitions.
xgboost_malware.py
import xgboost as xgb
xgb_clf = xgb.XGBClassifier(
n_estimators=200, max_depth=3, learning_rate=0.1,
random_state=11, eval_metric="logloss"
)
xgb_clf.fit(X_train, y_train)
print(f"Test accuracy: {xgb_clf.score(X_test, y_test):.4f}")
Sample output
sklearn GradientBoostingClassifier accuracy: 0.9194
XGBoost (xgboost.XGBClassifier) accuracy: 0.9194
On this particular dataset and with matched hyperparameters, the two libraries land on the same accuracy, a genuine result from running both, not a rounding coincidence. XGBoost's real advantage shows up at larger scale: faster training on large tabular datasets, built-in handling of missing values, and regularisation terms that reduce overfitting on noisy security data more reliably than scikit-learn's implementation does out of the box.