A distributed data processing engine built with Scala 2.12 and Apache Spark 3.5, designed for Datathon 2026.
This project calculates Customer Lifetime Value (CLV) and Daily Net Revenue by processing 15 e-commerce CSV datasets in a scalable distributed environment.
src/main/scala/com/datathon/distributed/
├── Schema.scala # 15 case classes mapped to CSV schemas
├── DistributedProcessor.scala # SparkSession + ETL pipeline
└── Visualization.scala # Vega-Lite visualizations
- Typed Dataset processing with Scala case classes
- Distributed ETL pipeline using Apache Spark
- CLV computation with window functions
- Daily revenue aggregation
- Delta Lake output support
- Vega-Lite HTML chart generation
- Optimized joins and partitioning strategies
- Adaptive Query Execution (AQE)
- Kryo serialization for performance
- JDK 8 or JDK 11
- sbt 1.9+
- Apache Spark 3.5.1
Dataset directory structure:
data/raw/
├── Transaction/
├── Master/
├── Operational/
└── Analytical/
sbt "run data/raw data/output"Build fat JAR:
sbt assemblySubmit Spark job:
spark-submit \
--class com.datathon.distributed.DistributedProcessor \
target/scala-2.12/distributed-processing-bigdata-assembly-1.0.0.jar \
data/raw data/output| Step | Description |
|---|---|
| Ingestion | Read 15 CSV files into typed Dataset[T] |
| Join | Sort-Merge Join for Orders ⋈ OrderItems; Broadcast Join for small dimension tables |
| Filter | Remove rows where cogs >= price |
| Net Revenue | (quantity × unit_price) − discount_amount − refund_amount |
| Window Aggregation | Running CLV using window functions partitioned by customer |
| Output | Write Delta Lake tables and HTML visualizations |
Enabled to automatically optimize:
- Shuffle partitions
- Join strategies
- Skewed data handling
Small dimension tables such as:
- Geography
- Promotions
are broadcast to avoid unnecessary shuffle operations.
Datasets are repartitioned by order_id before large joins to improve parallelism and reduce shuffle overhead.
The enriched dataset is cached because it is reused by:
- CLV computation
- Daily revenue aggregation
Uses Kryo serializer instead of Java serialization for:
- Faster shuffle performance
- Reduced memory usage
- More efficient caching
Filters invalid rows (cogs < price) as early as possible to reduce downstream processing volume.
Charts are generated using Vegas (Vega-Lite) and exported as HTML.
Supported visualizations:
- Daily revenue time-series
- Customer Lifetime Value histogram
import com.datathon.distributed._
val chart = Visualization.visualizeDailyRevenue(dailyRevenue)
kernel.publish.html(
Visualization.toHTML(chart)
)println("%html " + Visualization.toHTML(chart))displayHTML(
Visualization.toHTML(chart)
)| Library | Version | Purpose |
|---|---|---|
| Apache Spark | 3.5.1 | Distributed processing engine |
| Delta Lake | 3.2.0 | ACID tables and optimized storage |
| Vegas | 0.3.11 | Vega-Lite chart generation |
| Kryo | Built-in | High-performance serialization |
The system computes:
- Daily Net Revenue
- Customer Lifetime Value (CLV)
- Revenue trends over time
- Customer spending distribution
Generated outputs are stored in:
data/output/
├── delta/
├── charts/
└── reports/
- Distributed processing across Spark executors
- Optimized shuffle reduction
- Efficient memory serialization
- Reusable cached datasets
- Scalable architecture for large-scale e-commerce analytics
This project is intended for educational and Datathon competition purposes.