Skip to content
aalosbehPublic

About

Explainable cascade-forest (gcForest-style) anomaly detection: regularised-regression feature augmentation + SHAP/LIME explainability, benchmarked on NSL-KDD and UNSW-NB15 with McNemar testing, ablations, and seed-sensitivity checks.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

XDF-CFI: An Explainable Cascade-Forest Framework for Anomaly Detection in Cloud and Network Forensic Investigation

License: MIT

This repository contains the full, from-scratch implementation of XDF-CFI: a regularised-regression feature-augmentation stage feeding a cascade-forest (gcForest-style) classifier, explained end-to-end with SHAP and LIME.

Why this revision looks the way it does

An earlier version of this work reported results on the BiSHM cloud-hypervisor-telemetry dataset (Purnaye & Kulkarni, 2022). While preparing this revision, that dataset became unavailable (its Kaggle listing now 404s; its IEEE DataPort release requires credentials we could not obtain — see data/README.md). Rather than leave the framework unvalidated, we re-ran the entire pipeline end-to-end on two large, independently downloadable, actively used public benchmarks — NSL-KDD and UNSW-NB15 — in both binary and multi-class form, against six baselines, with McNemar significance testing, bootstrap confidence intervals, 5-fold cross-validation, and an ablation study. Every number in the paper is produced by the code in this repository; none are carried over from the earlier BiSHM-based draft.

We also could not find a maintained, modern-Python build of the original gcForest library, so code/deep_forest.py is a from-scratch re-implementation of the cascade mechanism on top of scikit-learn.

Repository layout

code/
  data_utils.py                Dataset loading, cleaning, splitting, checksums
  regression_stage.py           Stage 1: Ridge/Lasso/ElasticNet/Logistic + feature augmentation
  deep_forest.py                Stage 2: from-scratch cascade-forest classifier
  baselines.py                  The six comparison models
  metrics_utils.py               Accuracy/F1/MCC/ROC-AUC, McNemar's test, bootstrap CI
  run_experiments.py            Main experiment runner (one dataset x label-mode per run)
  xai_and_figures.py            Stage 3 (SHAP/LIME) + every figure in the paper
  patch_lightgbm.py             Class-weighting fix applied post-hoc for fairness (see below)
  make_architecture_diagram.py  Generates the Figure 1 pipeline diagram from scratch (matplotlib)
  seed_sensitivity.py           Reruns the cascade at extra seeds (7, 123) for the seed-sensitivity table
  explanation_stability_check.py  Non-adversarial SHAP/prediction stability probe (Threat Model, Table 1)
data/
  README.md                     Dataset provenance, checksums, why BiSHM is absent
  download_data.sh              Re-fetches NSL-KDD / UNSW-NB15 from the mirrors used here
  raw/                          (not committed — see .gitignore; populate via download_data.sh)
results/
  <tag>.json                    Full metrics/ablation/McNemar/timing output per configuration
  log_<tag>.txt                 Training logs
  (*_models.joblib are not committed — regenerate via run_experiments.py; see .gitignore)
figure_new/                    Every figure in the paper, generated from real results
figure/                        Architecture diagrams (methodology.jpg, gcforest.jpg)
requirements.txt

Reproducing the results

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

bash data/download_data.sh          # fetches NSL-KDD + UNSW-NB15 (~64 MB)

cd code
python3 run_experiments.py --dataset nslkdd --label_mode binary
python3 run_experiments.py --dataset unsw   --label_mode binary
python3 run_experiments.py --dataset nslkdd --label_mode multiclass
python3 run_experiments.py --dataset unsw   --label_mode multiclass

python3 xai_and_figures.py --tag nslkdd_binary
python3 xai_and_figures.py --tag unsw_binary
python3 xai_and_figures.py --tag nslkdd_multiclass
python3 xai_and_figures.py --tag unsw_multiclass

Each run_experiments.py call writes results/<tag>.json (all metrics reported in the paper) and results/<tag>_models.joblib (the fitted models, used by xai_and_figures.py). Hyperparameters (--n_estimators 150 --max_depth 18 --max_layers 6 --n_folds_cascade 3) are the defaults used for every number in the paper; see Section 5.2 of the manuscript for why these values were chosen over the original gcForest defaults.

Two supplementary robustness checks can be reproduced from the already-trained models: a multi-seed stability check on the cascade, and a non-adversarial explanation-stability probe.

python3 seed_sensitivity.py --dataset nslkdd --label_mode binary --seeds 7 123
python3 explanation_stability_check.py --tag nslkdd_binary --n_instances 40

License

Code: MIT License (see LICENSE). The manuscript text and figures are © the authors; the paper itself is intended for submission to Computers & Security and standard academic reuse terms apply pending publication.

About

Explainable cascade-forest (gcForest-style) anomaly detection: regularised-regression feature augmentation + SHAP/LIME explainability, benchmarked on NSL-KDD and UNSW-NB15 with McNemar testing, ablations, and seed-sensitivity checks.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages