LaneWindow

An end-to-end Azure roadways analytics pipeline built on Databricks, following the Medallion Architecture (Bronze → Silver → Gold) with automated CI/CD deployment across dev and UAT Databricks workspaces using Databricks Asset Bundles (DABs).

---

🚦 Overview

LaneWindow ingests, cleans, and models roadway/lane-level data to produce analytics-ready datasets. The project is structured for reproducible, environment-aware deployments — the same bundle can be deployed to dev or UAT with a single CLI flag, driven entirely through Azure DevOps CI/CD.

---

🏗️ Architecture

flowchart LR
    subgraph Source["Data Sources"]
        A[Raw Roadway Data]
    end

    subgraph ADLS["Azure Data Lake Storage Gen2"]
        B1[Bronze Layer\nRaw / Landing]
        B2[Silver Layer\nCleaned / Conformed]
        B3[Gold Layer\nAggregated / Curated]
    end

    subgraph Databricks["Azure Databricks"]
        C1[Auto Loader\nIngestion]
        C2[PySpark\nTransformations]
        C3[Delta Live Tables\nPipelines]
    end

    subgraph CICD["CI/CD - Azure DevOps"]
        D1[Databricks Asset Bundles]
        D2[Dev Workspace]
        D3[UAT Workspace]
    end

    A --> C1 --> B1
    B1 --> C2 --> B2
    B2 --> C3 --> B3
    D1 -->|deploy| D2
    D1 -->|promote| D3
    B3 --> E[Analytics / BI Consumption]

Loading

---

🧱 Medallion Architecture

Layer	Purpose	Format
Bronze	Raw ingested data, schema-on-read, immutable landing zone	Delta
Silver	Cleaned, deduplicated, conformed data	Delta
Gold	Aggregated, business-ready tables for reporting/analytics	Delta

---

⚙️ Tech Stack

• Compute / Processing: Azure Databricks, PySpark, Delta Live Tables
• Ingestion: Auto Loader
• Storage: Azure Data Lake Storage Gen2 (ADLS Gen2)
• Orchestration: Azure Data Factory (ADF)
• Deployment: Databricks Asset Bundles (DABs)
• CI/CD: Azure DevOps Pipelines (multi-target: dev → UAT)
• Governance: Unity Catalog

---

🚀 CI/CD Pipeline

This project uses Databricks Asset Bundles to package notebooks, jobs, and configuration into a single deployable unit. Azure DevOps pipelines drive deployment:

1. Dev deployment — triggered on push to main, deploys the bundle to the dev workspace for validation.
2. UAT promotion — triggered manually or on merge, promotes the validated bundle to the UAT workspace.

# Deploy to dev
databricks bundle deploy -t dev

# Deploy to UAT
databricks bundle deploy -t uat

---

📂 Repository Structure

LaneWindow/
├── .bundle/lanewinbundle/    # Bundle deployment artifacts
├── lanewinbundle/            # Source: notebooks, jobs, pipeline definitions
├── For Terminal.ipynb        # Terminal/exploration notebook
├── databricks.yml            # Bundle configuration (targets: dev, uat)
└── README.md

---

📈 Status

Actively developed — Bronze/Silver/Gold pipelines implemented, CI/CD across dev and UAT workspaces validated end-to-end.