XDF-CFI: An Explainable Cascade-Forest Framework for Anomaly Detection in Cloud and Network Forensic Investigation
This repository contains the full, from-scratch implementation of XDF-CFI: a regularised-regression feature-augmentation stage feeding a cascade-forest (gcForest-style) classifier, explained end-to-end with SHAP and LIME.
An earlier version of this work reported results on the BiSHM cloud-hypervisor-telemetry dataset (Purnaye & Kulkarni, 2022). While preparing this revision, that dataset became unavailable (its Kaggle listing now 404s; its IEEE DataPort release requires credentials we could not obtain — see data/README.md). Rather than leave the framework unvalidated, we re-ran the entire pipeline end-to-end on two large, independently downloadable, actively used public benchmarks — NSL-KDD and UNSW-NB15 — in both binary and multi-class form, against six baselines, with McNemar significance testing, bootstrap confidence intervals, 5-fold cross-validation, and an ablation study. Every number in the paper is produced by the code in this repository; none are carried over from the earlier BiSHM-based draft.
We also could not find a maintained, modern-Python build of the original gcForest library, so code/deep_forest.py is a from-scratch re-implementation of the cascade mechanism on top of scikit-learn.
code/
data_utils.py Dataset loading, cleaning, splitting, checksums
regression_stage.py Stage 1: Ridge/Lasso/ElasticNet/Logistic + feature augmentation
deep_forest.py Stage 2: from-scratch cascade-forest classifier
baselines.py The six comparison models
metrics_utils.py Accuracy/F1/MCC/ROC-AUC, McNemar's test, bootstrap CI
run_experiments.py Main experiment runner (one dataset x label-mode per run)
xai_and_figures.py Stage 3 (SHAP/LIME) + every figure in the paper
patch_lightgbm.py Class-weighting fix applied post-hoc for fairness (see below)
make_architecture_diagram.py Generates the Figure 1 pipeline diagram from scratch (matplotlib)
seed_sensitivity.py Reruns the cascade at extra seeds (7, 123) for the seed-sensitivity table
explanation_stability_check.py Non-adversarial SHAP/prediction stability probe (Threat Model, Table 1)
data/
README.md Dataset provenance, checksums, why BiSHM is absent
download_data.sh Re-fetches NSL-KDD / UNSW-NB15 from the mirrors used here
raw/ (not committed — see .gitignore; populate via download_data.sh)
results/
<tag>.json Full metrics/ablation/McNemar/timing output per configuration
log_<tag>.txt Training logs
(*_models.joblib are not committed — regenerate via run_experiments.py; see .gitignore)
figure_new/ Every figure in the paper, generated from real results
figure/ Architecture diagrams (methodology.jpg, gcforest.jpg)
requirements.txt
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
bash data/download_data.sh # fetches NSL-KDD + UNSW-NB15 (~64 MB)
cd code
python3 run_experiments.py --dataset nslkdd --label_mode binary
python3 run_experiments.py --dataset unsw --label_mode binary
python3 run_experiments.py --dataset nslkdd --label_mode multiclass
python3 run_experiments.py --dataset unsw --label_mode multiclass
python3 xai_and_figures.py --tag nslkdd_binary
python3 xai_and_figures.py --tag unsw_binary
python3 xai_and_figures.py --tag nslkdd_multiclass
python3 xai_and_figures.py --tag unsw_multiclassEach run_experiments.py call writes results/<tag>.json (all metrics reported in the paper) and results/<tag>_models.joblib (the fitted models, used by xai_and_figures.py). Hyperparameters (--n_estimators 150 --max_depth 18 --max_layers 6 --n_folds_cascade 3) are the defaults used for every number in the paper; see Section 5.2 of the manuscript for why these values were chosen over the original gcForest defaults.
Two supplementary robustness checks can be reproduced from the already-trained models: a multi-seed stability check on the cascade, and a non-adversarial explanation-stability probe.
python3 seed_sensitivity.py --dataset nslkdd --label_mode binary --seeds 7 123
python3 explanation_stability_check.py --tag nslkdd_binary --n_instances 40Code: MIT License (see LICENSE). The manuscript text and figures are © the authors; the paper itself is intended for submission to Computers & Security and standard academic reuse terms apply pending publication.