auto-loader

Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.

End-to-end Azure Databricks retail data engineering project using Medallion Architecture (Bronze, Silver, Gold). Implements Auto Loader, Unity Catalog, Delta Lake, SCD Type 1 & 2 dimensions, and Fact Orders for analytics-ready star schema modeling.

Obsidian-ready Databricks 80/20 learning vault with hands-on SQL, PySpark, Delta Lake, Unity Catalog, Lakeflow, MLflow, tips and labs.

A production-shaped Databricks lakehouse pipeline that ingests, classifies, quality-gates, and aggregates core-banking transaction data. Built on the medallion architecture with Auto Loader, Delta Lake, Unity Catalog, enforced data quality, and infrastructure-as-code deployment via Databricks Asset Bundles.

End-to-end SaaS subscription data platform using AWS, Databricks (Delta Lake), and Snowflake featuring streaming ingestion, CDC/SCD modeling, medallion architecture, and Iceberg-based analytics.

Databricks Lakehouse | Medallion Architecture | SCD Type 2 | Auto Loader | Unity Catalog | PySpark | Workflows | Genie AI/BI

⚡ Real-time sales analytics pipeline using PySpark Structured Streaming on Databricks Free Edition — Auto Loader, windowed aggregations, watermarking, and Delta Lake sink. Beginner-friendly with full README.

Databricks lakehouse on Serverless: medallion pipeline (Auto Loader → Bronze/Silver/Gold), SCD2 dimensions, data-quality quarantine and reconciliation, Slack alerts, Liquid Clustering. 288 unit tests + end-to-end integration test. CI/CD with Asset Bundles, GitHub Actions and service principals. 3M rows in 11m38s.

A production-grade IPL Data Engineering pipeline using Medallion Architecture on Azure Databricks, featuring PySpark, Delta Lake Auto Loader, 'MERGE' logic, and Unity Catalog.

End-to-End GitHub Streaming Data Pipeline using Databricks, Auto Loader, Delta Lake, Unity Catalog, and Medallion Architecture.

Pipeline de dados no Databricks usando Arquitetura Medallion (Bronze, Silver, Gold). Demonstra uso de Databricks Asset Bundles (IaC) e Auto Loader para ingestão incremental resiliente (Schema Evolution). Foco em otimização de custos e Unity Catalog.

Enterprise-pattern Databricks lakehouse — Unity Catalog, Auto Loader, Lakeflow Declarative Pipelines, native SCD2, one-bundle deploy.

Streaming fraud detection on Databricks: Auto Loader ingest into a Unity Catalog medallion, verified end to end (v0.1.0); Silver, ML scoring and alerts next. PaySim data.

End-to-end Medallion architecture pipeline on Databricks (Auto Loader, Delta Lake, Unity Catalog, DLT) processing 7.7M US traffic accident records

Multi-tenant IoT telemetry Lakehouse on Databricks + Delta Lake. PySpark, Auto Loader, DLT, medallion architecture, Terraform IaC.

Medallion lakehouse on Databricks: PySpark, Delta Lake, Auto Loader incremental ingestion of 2.1M records, quarantine DQ gates, Kimball star schema, multi-task Workflow, Unity Catalog governance. Verified end-to-end run.

End-to-end cricket analytics platform built on Databricks

Medallion lakehouse on Databricks: Auto Loader bronze, Delta MERGE silver, gold aggregates, Unity Catalog, Terraform IaC

End-to-end Databricks LakeFlow (DLT) data engineering project using Medallion Architecture, Auto Loader, CDF, SCD, and Unity Catalog.