KKBox Customer Churn Prediction

A leakage audited churn prediction pipeline built on the KKBox WSDM Cup dataset. The primary focus is methodological rigor: every temporal feature is validated against a strict observation cutoff date before being included in the model.

---

Business Problem

Subscription platforms like KKBox incur significantly higher costs acquiring new users than retaining existing ones. This project builds a binary classifier to rank subscribers by churn risk (is_churn = 1), allowing retention teams to prioritize high risk users rather than distributing outreach resources uniformly across the user base.

---

Dataset

• train_v2.csv: Target labels (msno, is_churn) for 970,960 subscribers (~9.0% churn rate).
• members_v3.csv: Demographic features (city, bd, gender, registered_via, registration_init_time).
• transactions_v2.csv: Historical transactions (payment_method_id, payment_plan_days, plan_list_price, actual_amount_paid, is_auto_renew, transaction_date, membership_expire_date, is_cancel) spanning 2015-01-01 through 2017-03-31.

Note: Raw CSV files are excluded from this repository. Download them from the Kaggle KKBox Churn Prediction Challenge and place them in the Kaggle_Data/ directory.

---

Methodology

Observation Cutoff: Transaction data in transactions_v2 ends on 2017-03-31 for all subscribers. Because KKBox defines churn as failing to renew within 30 days of membership expiration, the 30-day decision window falls after 2017-03-31. To prevent data leakage, CUTOFF_DATE = 2017-03-31 is enforced as a hard boundary. All features are built strictly from transaction rows on or before this date.

Validation Strategy: The dataset represents a single labeled snapshot where all customers share the same observation window. Since multi cohort temporal validation is not possible with a single snapshot, we evaluate model generalization using an 80/20 stratified random split.

---

Leakage Prevention

To prevent temporal leakage, every feature was audited against a key constraint: was this information known on or before 2017-03-31?

During the audit, a key correctness bug was identified and resolved. Originally, a customer's latest expiration was computed as max(membership_expire_date) across their entire transaction history. For subscribers with old, long duration plans, this returned a stale expiration date rather than their active status. This was fixed by extracting the expiration date from each customer's chronologically last transaction.

Performance Note: Following the fix and feature rebuild, validation performance remained strong (ROC-AUC ~0.995, PR-AUC ~0.956). This high performance reflects strong pre cutoff behavioral signals: recent cancellation (latest_cancel), auto-renew status (latest_auto_renew), and expiration recency are naturally predictive of whether a user will lapse in the subsequent 30 day window.

---

Feature Engineering

All transaction metrics are aggregated to one row per msno using pre-cutoff data.

Category	Features
Demographics & Tenure	reg_tenure_days, reg_year, age_cleaned (clipped to 10-100), gender_cleaned, city, registered_via (categorical)
Full-History Transactions	transaction_count, total_paid, avg_paid, median_paid, avg_plan_days, avg_list_price, total_discount, avg_discount, discount_ratio, auto_renew_rate, cancellation_count, cancellation_rate, payment_method_count, plan_count, latest_auto_renew, latest_cancel
Recency	days_since_last_transaction, current_days_until_exp (computed from chronologically last transaction)
Behavior Changes	payment_method_changes, plan_changes
30/60/90-Day Windows	tx_count_30d/60d/90d, amount_paid_30d/90d, cancel_count_30d, cancel_rate_90d, auto_renew_rate_30d/90d
Trend	recent_vs_historical_payment, recent_vs_historical_tx_freq, recent_vs_historical_auto_renew

Subscribers without transactions (~3.9%) receive 0 for count/sum metrics (reflecting inactivity) and NaN for rate metrics (where 0/0 is undefined), which are handled downstream via median/mode imputation. city, registered_via, and gender_cleaned are encoded as nominal categorical variables (OneHotEncoder for Logistic Regression, native categories for LightGBM).

---

Models

1. Logistic Regression: Interpretable baseline utilizing a ColumnTransformer (median imputation and scaling for numeric features; most frequent imputation and one hot encoding for categorical features) fit strictly on the training fold.
2. LightGBM: Gradient boosted decision trees using native categorical handling, tuned via RandomizedSearchCV (20 iterations, 3 fold CV, scored on Average Precision) on a 120,000 row training sample and refit on the full training fold.

Imbalance Handling: The dataset contains 883,630 non churn vs. 87,330 churn cases (~10:1 ratio). We evaluated several strategies on the training fold:

Model	Strategy	ROC-AUC	PR-AUC
LightGBM	None	0.9944	0.9528
LightGBM	scale_pos_weight	0.9949	0.9522
LightGBM	SMOTE	0.9924	0.9389
Logistic Regression	class_weight=balanced	0.9764	0.8831
Logistic Regression	None	0.9717	0.8907

For LightGBM, unweighted training achieved the highest PR-AUC and was selected for final tuning.

---

Evaluation

Holdout Performance (20% Stratified Holdout)

Model	Imbalance Strategy	ROC-AUC	PR-AUC	Precision (0.50)	Recall (0.50)	F1 (0.50)
Logistic Regression	Balanced	0.9764	0.8831	0.6720	0.9229	0.7777
LightGBM (Tuned)	None	0.9954	0.9556	0.8093	0.9749	0.8844

Average Precision (PR-AUC) serves as the primary metric due to class imbalance.

Top-K Targeting & Lift (LightGBM, Tuned)

Segment	Customers Targeted	Churners Captured	Capture Rate	Precision	Lift
Top 5%	9,709	9,496	54.4%	97.8%	10.9x
Top 10%	19,419	16,550	94.8%	85.2%	9.5x
Top 20%	38,838	17,432	99.8%	44.9%	5.0x
Top 30%	58,257	17,460	99.97%	30.0%	3.3x

Targeting the top 10% highest risk subscribers captures 94.8% of all actual churners in the holdout set.

Probability Calibration

The tuned LightGBM model achieves a Brier score of 0.0207. Predicted probabilities are used primarily for relative risk ranking and percentile based tiering.

---

Model Interpretation

SHAP values (TreeExplainer on 3,000 validation samples) provide the primary feature importance ranking. The top predictive drivers are:

1. Transaction recency (days_since_last_transaction)
2. Expiration recency (current_days_until_exp)
3. Cumulative spend (total_list_price)
4. 90 day payment volume (amount_paid_90d)
5. Account tenure (reg_tenure_days)

---

Business Application

Subscribers are ranked into percentile based risk tiers (Top 10%, Next 20%, Bottom 70%) to match operational capacity. High predicted risk identifies users most likely to lapse, allowing campaigns to focus resources efficiently.

---

Project Structure

churn_prediction.ipynb
README.md

---

How to Run

1. Clone this repository.
2. Download train_v2.csv, members_v3.csv, and transactions_v2.csv from Kaggle into Kaggle_Data/.
3. Install dependencies:

   pip install numpy pandas matplotlib seaborn scikit-learn lightgbm shap imbalanced learn

4. Run churn_prediction.ipynb in Jupyter Notebook or Google Colab.

---

Limitations

• Single Snapshot Data: Validation relies on a single temporal cohort; it does not measure performance against future temporal drift.
• Exclusion of Daily Logs: user_logs_v2.csv was omitted to keep execution lightweight and Colab friendly.
• Non Causal: Scores measure churn propensity rather than causal uplift from retention interventions.