Olist Brazilian E-Commerce — Data Preprocessing Pipeline

A complete, reusable data preprocessing pipeline for the Brazilian E-Commerce Public Dataset by Olist, turning 9 raw e-commerce tables into a single modeling-ready dataset — with a full data-diagnosis report and a step-by-step decision log along the way.

Dataset: ~100,000 real orders (2016–2018) across 9 tables · Environment: Python 3.14, pandas 3.0, scikit-learn 1.9

---

What this project does

The workflow follows a "diagnose first, then process" methodology:

Structure Exploration → Quality Assessment → Distribution Check
    → Cleaning → Integration → Transformation → Reduction
    → Modeling-Ready Dataset

Stage	What it does
1. Structure Exploration	Load 9 tables; inspect shape, dtypes, memory, timestamp ranges, key business fields
2. Quality Assessment	Missingness · IQR outliers · duplicates · format consistency (incl. a crosstab proving "missing delivery date" is a business signal, not an error)
3. Distribution Check	Skew/kurtosis for numeric fields; rare-class detection for categoricals
4. Cleaning	Dedup · missing handling · Winsorize outliers · regex text cleaning (preserving Portuguese accents)
5. Integration	Aggregate detail tables to order level, then merge — with a direct-merge vs aggregate-then-merge comparison
6. Transformation	Yeo-Johnson · Z-score / Min-Max · ordinal label-encoding + nominal one-hot
7. Reduction	SelectKBest vs Lasso · PCA · stratified sampling · CSV vs Parquet
8. Export	Modeling-ready dataset + full decision log

Key results

• Distribution fix: total_price skew 4.820 → 0.002 after Yeo-Johnson (chosen over Box-Cox for zero/negative support).
• A real "missing ≠ error" insight: only 8 delivered orders lack a delivery date; the rest are genuinely undelivered (canceled / shipped / unavailable).
• Cartesian-product trap: merging items without aggregation inflates rows 99,441 → 113,425 (1.14×); aggregate-then-merge keeps 99,441.
• Feature agreement: SelectKBest ∩ Lasso = {freight, item count, product/category diversity, delivery days}.
• Compression: CSV 57.83 MB → Parquet 17.10 MB (3.38×), read ~19× faster.
• Final dataset: 99,441 × 66 (bool 37, float64 15, str 5, datetime64 5, int64 3, int32 1).

Getting started

Run on Kaggle (recommended)

1. Create → New Notebook, then File → Import Notebook and select olist-preprocessing-pipeline.ipynb.
2. Add Input → search brazilian ecommerce → add the dataset by Olist.
3. Run All. The notebook auto-detects the /kaggle/input/... path — no edits needed.

Run locally

1. Place the 9 CSVs (see below) in the same folder as the notebook.
2. Install deps: pip install pandas numpy scipy scikit-learn matplotlib pyarrow nbformat
3. Run the notebook in Jupyter.

The notebook writes figures/, olist_wide.csv/parquet and olist_clean_model_ready.parquet to the working directory.

Dataset files

olist_orders_dataset.csv · olist_order_items_dataset.csv · olist_order_payments_dataset.csv · olist_order_reviews_dataset.csv · olist_products_dataset.csv · olist_sellers_dataset.csv · olist_customers_dataset.csv · olist_geolocation_dataset.csv · product_category_name_translation.csv

Outputs

File	Description
olist_clean_model_ready.parquet	The modeling-ready dataset (99,441 × 66)
figures/2_missing_bar.png	Missing-value bar chart
figures/2_outliers_boxplot.png	IQR outlier boxplots
figures/6_price_transform.png	Raw vs Yeo-Johnson (histograms + Q-Q)
figures/7_pca_scree.png	PCA scree plot

---

Educational project for demonstrating data preprocessing methodology. Data: © Olist — Brazilian E-Commerce Public Dataset.