"When learning meets data, growth becomes measurable and inevitable."
Applied AI Engineer | Forward Deployed Engineer | Data Analytics | Data Engineering | AI Systems
As a Forward Deployed Engineer my role is to provide AI backed solutions to legacy Enterprise Systems across ANY TECH STACK as needed by clients.
It could be Google Cloud, Databricks, AWS, Snowflake, Agentic solutions and more.
Building end-to-end AI solutions across ETL, analytics, machine learning and AI engineering
Current Projects:
- 🛡️ Databricks Fraud Detection — streaming fraud pipeline on the Databricks lakehouse: Auto Loader replays 6M PaySim mobile-money transactions into a Unity Catalog medallion layout, with duplicate, late, schema-change and malformed-data scenarios each proven by SQL checks; a Lakeflow declarative pipeline of materialized views builds Silver with data-quality expectations, a quarantine shelf, duplicate removal and leak-free point-in-time velocity features; job and pipeline deployed as code via Databricks Asset Bundles; Bronze and Silver verified end to end on Databricks Free Edition (v0.2.0) with every count exact and incremental refresh measured, 79 tests in CI — next: ML training and scoring
- 🖱️ AWS Clickstream Analytics — serverless clickstream analytics lake on AWS turning 165,474 real web clicks into product and funnel insight: Lambda ingest → S3 Bronze/Silver/Gold → AWS Glue (PySpark) → Glue Data Catalog → Athena, with quality gates that quarantine bad rows instead of dropping them and a Gold star schema for BI; the whole stack is infrastructure as code (AWS SAM) with encryption, concurrency, timeout and query-scan cost guardrails enforced by tests; Spark output reconciles row-for-row with the source on every click, 123 tests in CI — next: first AWS deployment (Mumbai), then streaming and Redshift
- 🏗️ Sales Data Pipeline — production-grade Medallion (Bronze → Silver → Gold) ETL pipeline with automated data quality gates, feature engineering, and an embedded DuckDB analytical store; orchestrated via a self-hosted Apache Airflow DAG (9 independently retryable tasks) syncing into a partitioned/clustered BigQuery cloud warehouse, live-verified end-to-end against real GCP infrastructure; FastAPI query layer published to GHCR; watermark-based incremental load (insert-only CDC) now live, 160 tests enforced in CI — next: MLflow/W&B run tracking (v2.0)
I'm a data-driven professional passionate about applying AI, Data Engineering and Analytics to improve Business, Learning and Development (L&D) outcomes.
After a successful career in Aviation training and Airport operations, I've transitioned toward AI Engineering, data engineering and data analytics, where I can apply analytical methods to solve learning and business problems.
I build data-driven solutions covering:
- AI/ML Engineering — end-to-end training pipelines, inference APIs, and production cloud deployment
- Cloud Data Engineering — GCP Cloud Run, Artifact Registry, BigQuery; containerised CI/CD
- Lakehouse Data Engineering — Databricks Auto Loader, Spark Structured Streaming, Delta Lake, Unity Catalog, Asset Bundles
- Snowflake Data Cloud — Dynamic Tables, native Semantic Views, and Cortex Agents/Analyst for governed natural-language analytics
- Observability — structured JSON logging (Cloud Logging), Prometheus metrics endpoints
- ETL pipelines and data workflows
- Exploratory data analysis and visualization
- Predictive modelling using Python and R
- Analytics dashboards and reporting systems (R Shiny, Tableau, Looker Studio)
Programming & Analysis:
ML Engineering & APIs:
Visualization & Reporting:
Cloud Data Engineering:
Lakehouse & Streaming (Databricks):
AWS Data Engineering:
Data Warehousing & Cortex AI:
Issued by Snowflake
| Project | Description | Tools |
|---|---|---|
| 🛡️ Databricks Fraud Detection | Streaming fraud-detection pipeline on the Databricks lakehouse, built in verified phases. Auto Loader replays 6.36M PaySim mobile-money transactions as hourly JSON Lines files into a Unity Catalog medallion layout, with schema hints, addNewColumns evolution, rescued data and an exactly-once checkpoint. Duplicate, late, schema-change and malformed-data scenarios are each staged and proven by SQL checks. A Lakeflow declarative pipeline then builds Silver: data-quality expectations with a quarantine shelf, first-arrival dedup, and seven point-in-time velocity features on the receiving ("mule") account that use earlier hours only. The job and pipeline ship as code via Databricks Asset Bundles. Verified end to end on Databricks Free Edition (v0.2.0): 5,987,427 Bronze rows reconcile exactly to 5,987,412 clean + 5 rejected + 10 duplicates, features match an independent reference model, a late file corrects later features, and incremental refresh is measured from the event log. Next: ML training and scoring. |
Python, SQL, Databricks, Auto Loader, Lakeflow Declarative Pipelines, Spark Structured Streaming, Delta Lake, Unity Catalog, Asset Bundles, pytest, GitHub Actions |
| 🖱️ AWS Clickstream Analytics | Serverless clickstream analytics lake on AWS that turns raw web clicks into answers a retailer can act on: daily traffic, browse-depth funnel, and which products make visitors leave. Built on real data (165,474 clicks, 24,026 visits from an online clothing shop). A serverless Lambda ingest verifies every file (MD5 checksum, retry budget) and lands it by month in S3 Bronze; an AWS Glue PySpark job with job bookmarks builds Silver (hard quality rules quarantine bad rows with a reason, never silently drop them) and a Gold star schema with rerun-safe (idempotent) writes; 13 Glue Catalog tables with partition projection serve Athena. The entire stack is infrastructure as code (AWS SAM + CloudFormation) with enterprise guardrails enforced in CI: private encrypted buckets, reserved concurrency, job timeouts and an Athena 1 GB scan cutoff. Spark reconciles row-for-row with the source on all 165,474 clicks; architecture decisions recorded as ADRs, every tier cost-modelled before deployment. Status: built and tested; first AWS deployment next, then Kinesis/Firehose streaming and Redshift Serverless. | Python, PySpark, AWS Lambda, Amazon S3, AWS Glue, Glue Data Catalog, Amazon Athena, AWS SAM / CloudFormation, Parquet, DuckDB, pytest, GitHub Actions |
| 🏗️ Sales Data Pipeline (ETL) | Built a production-grade ETL pipeline using Medallion architecture (Bronze/Silver/Gold) to transform raw sales data into validated, analytics-ready datasets with automated data quality checks, feature engineering, and CI/CD workflows. Orchestrated via a self-hosted Apache Airflow DAG (9 independently retryable tasks) alongside the existing GitHub Actions cron, syncing into a partitioned/clustered BigQuery cloud warehouse — live-verified end-to-end, with watermark-based incremental load (insert-only CDC) now replacing full-dataset reprocessing on every run. | Python, Pandas, DuckDB, Apache Airflow, Google BigQuery, Docker, GitHub Actions |
| ⚡ GridPulse — GCP Data Platform | Bitemporal GCP data platform built to cover the Google Cloud Professional Data Engineer exam guide v4.2 end-to-end with runnable code, not notes — every settlement price keeps event_time, published_at, and settlement_run as separate dimensions so nothing is ever overwritten, and any past belief state is reconstructable via as-of queries. Runs entirely on free-tier services and local emulators (Pub/Sub, MinIO, Airflow, Spark) via Docker Compose, with real GCP reserved for short, budget-capped demo windows. Phase 1 in progress: the Elexon BMRS REST collector, Avro-enforced Pub/Sub, Apache Beam windowing pipeline, and DLQ replay are all built, tested, and verified end-to-end against live local services. |
Python, Terraform, Docker, Apache Beam, Pub/Sub, Airflow, BigQuery, GitHub Actions |
| 🚲 Bike Demand Prediction System | Built a 6-city live demand dashboard integrating OpenWeather forecasts, GBFS live station data, and a FastAPI ML backend. Features UC1 fleet rebalancing alerts and UC2 rider demand scores across Seoul, London, NYC, DC, Paris, and Chicago. | R, Shiny, httr, Leaflet, GBFS, FastAPI (backend), Docker, GitHub Actions |
| ⚙️ Bike Demand ML System | Production ML inference API live on GCP Cloud Run (v4.5.0). Trains 6-city Random Forest models (Seoul, London, NYC, DC, Paris, Chicago); models baked into Docker image at build time. CI auto-publishes to GHCR + Artifact Registry and redeploys on merge via gcloud run deploy. RMSE accuracy gates in CI, cost-audit alerting via Slack, structured JSON logging → Cloud Logging, Prometheus /metrics endpoint. |
Python, FastAPI, scikit-learn, Pydantic, Docker, GCP Cloud Run, Prometheus, GitHub Actions |
| 🏠 StayOps — Rental Ops Console | Multi-channel booking reconciliation engine and AI-assisted ops console for short/mid-term rental operators. Ingests bookings from CSV and Google Sheets (idempotent SHA-256 dedup), detects 4 conflict types automatically (duplicates, double-bookings, pricing anomalies, gap nights), and surfaces live KPI dashboards and SQL reports — built end-to-end with Claude Code on Next.js 16 + Supabase. Phase 2: Claude tool-calling agent layer. | TypeScript, Next.js 16, Drizzle ORM, Supabase, shadcn/ui, Anthropic SDK, Vercel |
| 🎓 Corporate Training Analytics Platform | Refactor->Re-write -> full-stack training records and analytics system to manage multi-course training programmes, featuring a unified data model, role-based admin dashboard, KPI tracking, event/result management, and reporting abstraction. | Java, SQL, Data Modeling, KPI Analytics, Role-Based Access |
| 🧾 Snowflake Cortex AI — AP Invoice Intelligence | Multi-source accounts-payable pipeline running entirely inside Snowflake. Dynamic Tables conform invoices from SAP, Oracle, Baan, and Workday into a native Semantic View per documented business rules; a Cortex Agent (Analyst) answers natural-language questions via governed text-to-SQL, checked against a 15-question golden evaluation set (13/15 passed). Live on Snowflake (v0.1.0). | Snowflake, SQL, Dynamic Tables, Cortex Analyst, Cortex Agents, Python |
| 🧾 AI Expense Text Classification | NLP pipeline classifying a real-world, anonymized FY23 expense ledger into Services/Equipment/Material — built against a documented PRD, with no pre-existing category labels to learn from. Compares TF-IDF + classical ML (94.4% accuracy, 0.93 macro F1 — selected) against a zero-shot transformer baseline, backed by a documented labeling rubric, 5-fold cross-validation, and an auto-generated executive summary. v1.0.0 shipped. | Python, scikit-learn, TF-IDF, Hugging Face Transformers, Jupyter, pytest |
The Sales Data Pipeline evolves from a production-grade Medallion ETL into a full customer analytics platform — unifying transactions, segmentation, and retention into a single source of truth.
flowchart TD
subgraph Sources["📥 Data Sources"]
S1["CRM"] & S2["POS / Transactions"] & S3["Web Analytics"]
end
subgraph ETL["🏗️ Medallion ETL ✅ Built — Python · Pandas · Pydantic"]
B["🥉 Bronze — Ingest + schema validation"]
C["🥈 Silver — Clean · Dedup · Feature engineering"]
D["🥇 Gold — Star schema · AOV · CLV pre-aggregated"]
end
subgraph Infra["⚙️ Data Infrastructure 🔄 In Progress — Airflow ✅ · BigQuery ✅ · Incremental load ✅"]
ORC["Apache Airflow ✅<br/>Self-hosted DAG · 9 tasks · Docker Compose"]
WH["BigQuery ✅<br/>Partitioned · Clustered · Live-verified"]
end
subgraph Seg["🧠 Customer Segmentation 🔜 Planned — scikit-learn · Databricks"]
E["RFM Analysis<br/>Recency · Frequency · Monetary"]
F["Cohort Analysis<br/>Signup cohorts · Engagement lifecycle"]
G["K-Means Clustering<br/>Unsupervised persona discovery"]
end
subgraph Ret["🔁 Retention Analytics 🔜 Planned — scikit-learn · Databricks"]
H["Churn Classification<br/>At-risk flagging · Re-engagement triggers"]
I["LTV Correlation<br/>High-value segment identification"]
end
subgraph Serving["⚡ Serving Layer ✅ Built — FastAPI · DuckDB · Docker"]
J["🦆 DuckDB — In-process analytics"]
K["FastAPI REST API"]
end
subgraph Dash["📊 Analytics Dashboard 🔜 Planned — Tableau · Streamlit"]
L["KPI tracking · Segment views<br/>Retention curves · LTV by cohort"]
end
subgraph Agent["🤖 Conversational Analytics Agent 🔜 Planned — LangGraph · Claude Sonnet 5"]
M["Text-to-SQL Tool-Calling Agent<br/>Natural-language queries over the warehouse"]
end
Sources --> B
B --> C
C --> D
D --> ORC
ORC --> WH
D --> J
WH --> E
WH --> F
E --> G
G --> H
F --> H
H --> I
J --> K
I --> K
K --> L
L --> M
- Currently enrolled in IIT Delhi — Advanced Certification: Forward Deployed Engineer
- Google Data Analytics Professional Certificate
- IBM Data Analytics Professional Certificate with Excel & R
My mission is to bridge Data Engineering and Learning — using data to make learning, Business Analysis and training more effective.
📍 Mumbai, India
📧 [email protected]
🔗 LinkedIn
💼 GitHub Projects
"When learning meets data, growth becomes measurable and inevitable."