Databricks lakehouse on Serverless: medallion pipeline (Auto Loader → Bronze/Silver/Gold), SCD2 dimensions, data-quality quarantine and reconciliation, Slack alerts, Liquid Clustering. 288 unit tests + end-to-end integration test. CI/CD with Asset Bundles, GitHub Actions and service principals. 3M rows in 11m38s.
python apache-spark terraform ci-cd pytest pyspark data-engineering databricks data-pipeline data-quality etl-pipeline scd2 github-actions delta-lake lakehouse unity-catalog medallion-architecture databricks-asset-bundles auto-loader liquid-clustering
-
Updated
Oct 15, 2026 - Python