Early Warning System for Student Dropout Prediction

An explainable and fairness-aware machine learning pipeline for identifying students who may Drop Out, remain Enrolled, or Graduate.

The main model is intentionally restricted to information available at enrollment time. First- and second-semester academic results are excluded from the default pipeline, making the project suitable for an early-warning use case rather than retrospective classification.

Why this project matters

Student-support teams need actionable signals before academic disengagement becomes difficult to reverse. This project combines:

• leakage-safe data splitting and preprocessing;
• multiple tabular machine-learning models;
• Dropout-focused evaluation and threshold tuning;
• a transparent 0–100 Student Risk Score;
• SHAP TreeExplainer global and per-class explanations;
• fairness auditing across Gender and Scholarship holder;
• a locally deployable FastAPI service.

Problem definition

Input

The UCI dataset contains 4,424 student records and enrollment-time demographic, application, academic-background, financial, admission, and macroeconomic variables.

Target

Three-class outcome prediction:

Encoded label	Outcome
0	Dropout
1	Enrolled
2	Graduate

Modeling constraint

The default configuration uses only enrollment-time features:

active_feature_groups:
  - enrollment

Semester feature groups remain documented in configs/config.yaml for experimentation, but they are not active in the early-warning model.

Methodology

The pipeline follows a strict train/validation/test protocol:

1. Load and clean the raw CSV.
2. Select only the configured enrollment-time features.
3. Create stratified train, validation, and test splits.
4. Fit the scaler and one-hot encoder on the training split only.
5. Run cross-validation on the training split; if resampling is enabled, it is performed inside each fold.
6. Train Logistic Regression, Random Forest, XGBoost, LightGBM, and CatBoost.
7. Select the best model using validation performance.
8. Evaluate the selected model once on the untouched test set.
9. Tune the Dropout decision threshold on validation only.
10. Generate SHAP explanations, fairness reports, plots, and deployment artifacts.

This prevents test-set information from influencing preprocessing, model selection, or threshold selection.

Results

The following benchmark was produced by the latest enrollment-only pipeline on the fixed stratified test split. Re-run the pipeline after changing dependencies or configuration before quoting these numbers in a report.

Model	Weighted F1	Macro F1	Balanced Accuracy	ROC-AUC
XGBoost	0.6010	0.5404	0.5379	0.7717
LightGBM	0.6055	0.5388	0.5363	0.7567
CatBoost	0.5986	0.5292	0.5274	0.7778
Random Forest	0.6026	0.5273	0.5298	0.7746
Logistic Regression	0.5945	0.5480	0.5569	0.7611

Final model selection

XGBoost was selected as the final model using validation macro F1, achieving 0.6070 on the validation split. The test set was used only for final evaluation and was not used for model selection or threshold tuning. Therefore, the final model is not necessarily the model with the highest test-set weighted F1.

XGBoost per-class evaluation

Class	Precision	Recall	F1	Support
Dropout	0.6406	0.5775	0.6074	284
Enrolled	0.3125	0.3145	0.3135	159
Graduate	0.6802	0.7217	0.7003	442

The Enrolled class is the most difficult class because it is both the smallest class and an intermediate outcome. This limitation is explicitly reported instead of being hidden by weighted metrics.

Dropout-focused evaluation

For an early-warning system, missing a student who is likely to Drop Out can be more costly than creating an additional review case. The pipeline therefore also evaluates:

Dropout vs. Not Dropout

The Dropout threshold is tuned on validation data using F2, which gives more importance to recall. The selected threshold is frozen before test evaluation and saved to:

On the current validation split, the selected threshold is 0.205, achieving 0.778 recall, 0.533 precision, 0.632 F1, and 0.717 PR-AUC for Dropout.

outputs/models/dropout_threshold.joblib

The complete report is saved to:

outputs/reports/dropout_focus_report.json

This report includes Dropout precision, recall, F1, PR-AUC, ROC-AUC, threshold, and binary confusion matrix.

Student Risk Score

The API converts the predicted Dropout probability into a transparent score:

Risk Score = 100 × P(Dropout)

Every prediction includes:

"dropout_risk": {
  "score": 72.4,
  "probability": 0.724,
  "level": "high",
  "threshold": 0.205
}

Risk levels are derived from the tuned threshold:

• low: probability below the intervention threshold;
• medium: probability above the intervention threshold but below the high-risk boundary;
• high: probability at or above the configured high-risk threshold.

The score is a prioritization signal, not a final decision about a student. It should support human review and student services.

SHAP explainability

The final XGBoost model uses shap.TreeExplainer, not a model-agnostic permutation fallback.

SHAP output includes:

• global feature importance across all classes;
• Dropout-specific feature importance;
• Enrolled-specific feature importance;
• Graduate-specific feature importance;
• one-hot columns aggregated back to their original feature names;
• machine-readable CSV rankings;
• a GitHub-friendly Markdown summary.

Typical high-ranking factors include Course, Tuition fees up to date, Scholarship holder, Age at enrollment, and Admission grade. These are associations learned by the model, not causal conclusions.

Generated files:

outputs/plots/shap_global_summary.png
outputs/plots/shap_Dropout.png
outputs/plots/shap_Enrolled.png
outputs/plots/shap_Graduate.png
outputs/plots/shap_feature_importance.csv
outputs/plots/shap_feature_importance.md

Why XGBoost is pinned

The project pins XGBoost to 2.1.4 to ensure compatibility with the current SHAP TreeExplainer workflow. If the XGBoost version is changed, the model and explainability artifacts should be regenerated and revalidated.

Fairness auditing

Fairness analysis is performed for Gender and Scholarship holder. Because Fairlearn's demographic parity and equalized odds metrics are binary metrics, the analysis evaluates:

Graduate vs. Not Graduate

The report includes demographic parity difference, equalized odds difference, disparate impact ratio, selection rates, and group-level F1. These are diagnostic indicators requiring domain interpretation; they do not by themselves prove that a model is fair or unfair.

Project structure

.
├── app/
│   └── main.py                         # FastAPI application
├── configs/
│   └── config.yaml                     # Features, models, metrics, paths
├── data/raw/
│   └── data.csv                        # UCI dataset
├── scripts/
│   └── run_pipeline.py                 # End-to-end training pipeline
├── src/
│   ├── data/loader.py                  # Loading and stratified splitting
│   ├── features/preprocessing.py       # Scaling and categorical encoding
│   ├── features/engineering.py         # Enrollment-time feature utilities
│   ├── models/trainers.py              # Model registry and CV training
│   ├── evaluation/metrics.py           # Metrics, reports, plots
│   ├── evaluation/risk.py               # Dropout metrics, threshold, risk score
│   ├── explainability/shap_analysis.py # TreeSHAP analysis
│   └── fairness/analysis.py            # Fairness metrics and plots
├── tests/
│   └── test_pipeline.py                # Smoke and artifact tests
├── outputs/                            # Generated artifacts
├── requirements.txt
├── pyproject.toml
└── LICENSE

Installation

Python 3.10 or newer is recommended.

git clone <your-repository-url>
cd early-warning

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Verify the important versions:

python -c "import xgboost, shap; print('xgboost:', xgboost.__version__, 'shap:', shap.__version__)"

XGBoost must report 2.1.4 for the guaranteed TreeExplainer path.

Run the training pipeline

Run from the project root:

source .venv/bin/activate
python -m scripts.run_pipeline

The pipeline creates or updates:

outputs/models/
outputs/reports/
outputs/plots/

Important artifacts:

File	Purpose
outputs/models/best_model.joblib	Selected deployment model
outputs/models/preprocessor.joblib	Fitted train-only transformer
outputs/models/label_encoder.joblib	Target label mapping
outputs/models/feature_cols.joblib	Expected input columns
outputs/models/dropout_threshold.joblib	Validation-tuned Dropout threshold
outputs/reports/test_metrics.json	Machine-readable test metrics
outputs/reports/dropout_focus_report.json	Dropout-focused evaluation
outputs/reports/FINAL_REPORT.md	Generated project report
outputs/reports/model_comparison.csv	Model comparison table

Run the API locally

After the pipeline has generated the model artifacts:

source .venv/bin/activate
uvicorn app.main:app --host 0.0.0.0 --port 8000

Open the interactive documentation at http://localhost:8000/docs.

Health check

curl http://localhost:8000/

Model information

curl http://localhost:8000/model-info

Single prediction

The API accepts enrollment-time feature values. Production clients should send all expected fields.

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "features": {
      "Marital status": 1,
      "Application mode": 1,
      "Application order": 1,
      "Course": 9254,
      "Daytime/evening attendance": 1,
      "Previous qualification": 1,
      "Previous qualification (grade)": 130.0,
      "Nacionality": 1,
      "Mother\u0027s qualification": 13,
      "Father\u0027s qualification": 14,
      "Mother\u0027s occupation": 6,
      "Father\u0027s occupation": 6,
      "Admission grade": 130.0,
      "Displaced": 1,
      "Educational special needs": 0,
      "Debtor": 0,
      "Tuition fees up to date": 1,
      "Gender": 1,
      "Scholarship holder": 0,
      "Age at enrollment": 20,
      "International": 0,
      "Unemployment rate": 10.8,
      "Inflation rate": 1.4,
      "GDP": 1.74
    }
  }'

The response contains the predicted class, class probabilities, confidence, and dropout_risk object.

Batch prediction

The following is a minimal smoke-test payload. The current API accepts partial dictionaries and fills omitted fields with zero; production clients should send all 24 enrollment-time features shown in the single-prediction example.

curl -X POST http://localhost:8000/predict-batch \
  -H "Content-Type: application/json" \
  -d '{"instances": [{"Course": 9254, "Age at enrollment": 20}]}'

Testing

Run the test suite after the pipeline has generated model artifacts:

python -m pytest tests/ -v

The live API smoke test is skipped when no server is running. Start Uvicorn in another terminal to exercise the HTTP endpoint test.

Limitations and responsible use

• The dataset is relatively small for a production decision system.
• Enrollment-only prediction is inherently harder than prediction after academic performance is observed.
• The Enrolled class is difficult to identify reliably.
• Fairness metrics show group disparities and should be investigated with domain experts.
• The risk score is not a diagnosis, rejection rule, or automated intervention decision.
• Model performance may change across institutions, cohorts, admission policies, and time.
• The model should support proactive human outreach, not replace it.

Data source

The project uses the UCI Machine Learning Repository — Predict Students' Dropout and Academic Success dataset. The dataset includes both enrollment-time and semester-related variables; this project deliberately excludes semester variables from the default early-warning model.

Reference: Cortez, P. (2018). Predict Students' Dropout and Academic Success. UCI Machine Learning Repository, Dataset 697.

License

Released under the MIT License. See LICENSE.

Citation

If you use this project in research or a portfolio, cite the original UCI dataset and describe the exact feature groups, split strategy, model version, and evaluation metrics used.