Skip to content

About

Modern lakehouse architecture for large-scale fashion e-commerce analytics and forecasting with Spark.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

Distributed Data Processing

A distributed data processing engine built with Scala 2.12 and Apache Spark 3.5, designed for Datathon 2026.
This project calculates Customer Lifetime Value (CLV) and Daily Net Revenue by processing 15 e-commerce CSV datasets in a scalable distributed environment.


Project Structure

src/main/scala/com/datathon/distributed/
├── Schema.scala                # 15 case classes mapped to CSV schemas
├── DistributedProcessor.scala  # SparkSession + ETL pipeline
└── Visualization.scala         # Vega-Lite visualizations

Features

  • Typed Dataset processing with Scala case classes
  • Distributed ETL pipeline using Apache Spark
  • CLV computation with window functions
  • Daily revenue aggregation
  • Delta Lake output support
  • Vega-Lite HTML chart generation
  • Optimized joins and partitioning strategies
  • Adaptive Query Execution (AQE)
  • Kryo serialization for performance

Requirements

  • JDK 8 or JDK 11
  • sbt 1.9+
  • Apache Spark 3.5.1

Dataset directory structure:

data/raw/
├── Transaction/
├── Master/
├── Operational/
└── Analytical/

Build & Run

Run with sbt

sbt "run data/raw data/output"

Production Deployment

Build fat JAR:

sbt assembly

Submit Spark job:

spark-submit \
  --class com.datathon.distributed.DistributedProcessor \
  target/scala-2.12/distributed-processing-bigdata-assembly-1.0.0.jar \
  data/raw data/output

Processing Pipeline

Step Description
Ingestion Read 15 CSV files into typed Dataset[T]
Join Sort-Merge Join for Orders ⋈ OrderItems; Broadcast Join for small dimension tables
Filter Remove rows where cogs >= price
Net Revenue (quantity × unit_price) − discount_amount − refund_amount
Window Aggregation Running CLV using window functions partitioned by customer
Output Write Delta Lake tables and HTML visualizations

Distributed Optimization Techniques

Adaptive Query Execution (AQE)

Enabled to automatically optimize:

  • Shuffle partitions
  • Join strategies
  • Skewed data handling

Broadcast Joins

Small dimension tables such as:

  • Geography
  • Promotions

are broadcast to avoid unnecessary shuffle operations.

Repartitioning

Datasets are repartitioned by order_id before large joins to improve parallelism and reduce shuffle overhead.

Dataset Caching

The enriched dataset is cached because it is reused by:

  • CLV computation
  • Daily revenue aggregation

Kryo Serialization

Uses Kryo serializer instead of Java serialization for:

  • Faster shuffle performance
  • Reduced memory usage
  • More efficient caching

Predicate Pushdown

Filters invalid rows (cogs < price) as early as possible to reduce downstream processing volume.


Visualization Support

Charts are generated using Vegas (Vega-Lite) and exported as HTML.

Supported visualizations:

  • Daily revenue time-series
  • Customer Lifetime Value histogram

Notebook Integration

Almond (Scala Jupyter Kernel)

import com.datathon.distributed._

val chart = Visualization.visualizeDailyRevenue(dailyRevenue)

kernel.publish.html(
  Visualization.toHTML(chart)
)

Apache Zeppelin

println("%html " + Visualization.toHTML(chart))

Databricks

displayHTML(
  Visualization.toHTML(chart)
)

Main Dependencies

Library Version Purpose
Apache Spark 3.5.1 Distributed processing engine
Delta Lake 3.2.0 ACID tables and optimized storage
Vegas 0.3.11 Vega-Lite chart generation
Kryo Built-in High-performance serialization

Example Metrics

The system computes:

  • Daily Net Revenue
  • Customer Lifetime Value (CLV)
  • Revenue trends over time
  • Customer spending distribution

Output

Generated outputs are stored in:

data/output/
├── delta/
├── charts/
└── reports/

Performance Highlights

  • Distributed processing across Spark executors
  • Optimized shuffle reduction
  • Efficient memory serialization
  • Reusable cached datasets
  • Scalable architecture for large-scale e-commerce analytics

License

This project is intended for educational and Datathon competition purposes.

About

Modern lakehouse architecture for large-scale fashion e-commerce analytics and forecasting with Spark.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages