A point-in-time data pipeline that nowcasts euro-area inflation from open, higher-frequency ECB signals — and evaluates itself the way a central bank would: with a walk-forward backtest that never peeks at the future.
The interesting part of this project is not the model (a small ridge regression). It is the data engineering underneath it: incremental collection, an immutable partitioned landing zone, data-quality gates, a mixed-frequency point-in-time feature panel, and a DuckDB warehouse that Power BI or Tableau can plug straight into. Nowcasting is a domain where a single data-engineering mistake — using a number before it was published — silently inflates every result, so it is a good forcing function for doing the plumbing right.
Extract ECB Data Portal (HICP, energy, FX, policy rate) ← no API key
│ incremental, watermarked
Landing data/landing/ingest_date=YYYY-MM-DD/*.parquet ← immutable snapshots
│
Validate pandera schema + cross-row gates (fails loud)
│
Transform typed staging → monthly, availability-aware table
│
Features point-in-time panel ← the no-look-ahead core
│
Warehouse (DuckDB) dim_series · dim_month · fact_observation · mart_monthly
│ mart_panel · mart_backtest → Power BI / Tableau connect here
Nowcast ridge regression + walk-forward backtest vs a random walk
│
Look-ahead the same backtest with the gate off — what cheating is worth
Euro-area HICP is published with a lag: the flash estimate lands at the end of the reference month, the final print about two and a half weeks later. Anyone who has to act inside that window — reprice a catalogue, settle an index-linked contract, open a wage negotiation, size a hedge — is either waiting or guessing.
A nowcast is worth building when the cost of waiting exceeds the cost of being roughly wrong. That framing sets the bar, and it is a low-glamour one: the honest competitor is not "no information", it is "assume it is the same as last month", which on a series this persistent is already a strong answer.
So the deliverable is not the model. It is a defensible statement of how much better than that guess you can get — 16.3%, or 0.311 pp of RMSE against 0.372 — plus enough machinery to prove the number was not bought with a look-ahead bug. At that error a nowcast is usable for a decision tolerant of a third of a percentage point and useless for one that is not, and saying which is which is the analyst's job here.
Walk-forward backtest, 192 months (2010–2025), euro-area HICP headline inflation, trained only on the past at each step:
| Nowcast | Baseline (random walk) | |
|---|---|---|
| RMSE | 0.311 pp | 0.372 pp |
| MAE | 0.224 pp | 0.252 pp |
| Skill vs baseline | +16.3 % | — |
| Direction hit-rate | 67.5 % | — |
+16 % over a random walk is a genuine, modest result — and that is the point.
Inflation is highly persistent, so "assume it's the same as last month" is a
hard baseline to beat; a nowcast that claimed 90 % skill would be a look-ahead
bug, not a breakthrough. The numbers above come straight from
reports/backtest_metrics.json, regenerated on
every run.
The model's pull is economically sensible — last month's inflation dominates, then the energy-price component, then the policy rate and exchange rates:
This is the property everything else serves. For each target month M the nowcast is made at a cutoff of "end of M", and a signal value may enter the feature row only if its publication date is on or before that cutoff. Each series in the catalogue declares a publication lag, and late-arriving series therefore show up one or more months stale — the ragged edge that makes nowcasting a real problem rather than an ordinary regression.
Concretely, in the built panel the monthly FX signals come in ~0.7 months stale (you don't have a full month's average until the month is over), while the energy component — co-released with the flash estimate — is contemporaneous. A test asserts the guarantee directly: for target month M, the signal used is M-1, never M. Another asserts the backtest never trains on the future.
This uses publication-lag gating, not archived real-time vintages (the first-print-vs-revised history). Lag gating already reproduces the ragged edge; true vintages — e.g. via FRED/ALFRED — are a documented next step, not a silent omission.
Asserting point-in-time correctness is cheap, and a reader has no way to tell
whether the gate is load-bearing or ceremony. So the pipeline runs the backtest a
second time with the gate switched off — same catalogue, same panel code, same
ridge, same walk-forward — and reports the difference
(reports/lookahead_comparison.json, regenerated
every run):
| RMSE | Skill vs random walk | Direction hit-rate | |
|---|---|---|---|
| Point-in-time (shipped) | 0.311 | +16.3 % | 0.675 |
| Look-ahead (gate off) | 0.316 | +15.1 % | 0.707 |
Nothing. Cheating does not help — and the 1.2-point gap is noise across 192 months, so the honest reading is "no material difference", not "look-ahead hurts".
That is a real answer rather than a failed measurement, and the per-feature breakdown says why:
| Feature | Months the gate changes | Correlation, gated vs ungated |
|---|---|---|
hicp_energy_yoy |
0 % | 1.00 — published with the flash estimate, lag 0 |
refi_rate |
0 % | 1.00 — announced the day it changes, lag 0 |
eur_usd_mom |
71 % | 0.51 |
eur_gbp_mom |
71 % | 0.36 |
The gate binds on exactly two features, and it binds hard — a correlation of 0.36 means last month's FX move is nearly a different variable from this month's. But FX is the weakest input to euro-area headline inflation at a monthly horizon (see the coefficient chart above), and the two features that actually carry the nowcast are contemporaneous by construction. So there is nothing for look-ahead to steal.
Two things follow, and they are why this ablation lives in the pipeline rather than in a footnote:
- The headline +16.3 % is not propped up by a leak. That is now demonstrated rather than promised.
- The size of a look-ahead leak depends entirely on which features are gated. Add one genuinely high-frequency signal — Google Trends, electricity demand, any of the extensions listed below — and this table changes completely. Re-running the ablation on every pipeline run is what turns that from a thing somebody has to remember into a number that appears in the report.
refi_rate is worth one more look: its mean staleness is 9.5 months in both
arms. The policy rate simply does not change often, so the latest observation is
usually old. Staleness and gating are not the same thing, and the breakdown keeps
them apart.
Each series remembers the last period it fetched
(data/extract_state.json); a re-run asks the
ECB only for points from the watermark on. Every run writes a fresh, immutable
parquet snapshot under data/landing/ingest_date=…/, so the raw record of "what
we pulled and when" is replayable — and over repeated runs those snapshots
accumulate into genuine data vintages.
Between landing and modelling, a pandera
schema checks column types and non-null contracts, and hand-written gates catch
what a schema cannot: duplicate (series, period) rows, an entirely missing
series, an implausible target value, staleness. A violation raises and stops
the run — a bad extract dies here instead of quietly poisoning the nowcast.
The modelled data lands in a single-file DuckDB warehouse
(warehouse.py): dim_series, dim_month,
a fact_observation table, and marts for the monthly panel, the feature panel,
and the backtest. DuckDB needs no server and both Power BI and Tableau connect
to it (or to the parquet marts) directly — so "matplotlib now, BI tool later"
needs no rework.
A small dependency-ordered runner executes
extract → validate → transform → warehouse → nowcast, times each stage, and
writes reports/run_log.json with per-stage
status, row counts and the model's top coefficients — a minimal lineage record.
The stage boundaries are drawn so that swapping this for Airflow or Prefect later
means wrapping each run_* in a task, nothing more.
git clone https://github.com/diskaver-it/macro-nowcast
cd macro-nowcast
pip install -e ".[dev]"
python -m macro_nowcast.pipeline # collect from the ECB, build, backtest
python -m macro_nowcast.pipeline --offline # re-run modelling on already-landed data
pytest -q # 32 tests, no network (mocked HTTP + synthetic data)Explore the warehouse with any DuckDB client, or from Python:
from macro_nowcast import warehouse
warehouse.query("select month, actual, nowcast, baseline from mart_backtest order by month desc limit 12")The ECB Data Portal needs no API key. Collected data and the warehouse are gitignored — the pipeline regenerates them.
src/macro_nowcast/
├── config.py the series catalogue (id, flow, key, freq, publication lag)
├── sources/ pluggable source adapters (ecb.py; fred.py is the keyed next step)
├── extract.py incremental, watermarked pull → immutable parquet landing
├── validate.py pandera schema + cross-row data-quality gates
├── transform.py landing → typed staging → monthly, availability-aware table
├── features.py the point-in-time feature panel (no look-ahead)
├── nowcast.py ridge model + walk-forward backtest
├── evaluate.py skill vs the random-walk baseline
├── lookahead.py the same backtest with the gate off - what cheating is worth
├── warehouse.py DuckDB star schema + BI-ready marts
├── pipeline.py the DAG runner + run log
└── plots.py nowcast-vs-actual, error, coefficient figures
tests/ 32 tests — mocked HTTP, synthetic data, no network
reports/ backtest_metrics.json, lookahead_comparison.json,
run_log.json, figures/
- More signals. Add a row to the catalogue — Google Trends, electricity demand (EIA/ENTSO-E), commodity prices, PMIs. The pipeline picks it up with no other change.
- More countries. The ECB serves national HICP; the US via a FRED adapter (a keyed source, already stubbed in the registry).
- True vintages. Swap publication-lag gating for FRED/ALFRED real-time archives for fully first-print-accurate backtests.
- Real orchestration. Wrap the stages in Airflow/Prefect; run daily.
- dbt. The staging→core→marts layering maps directly onto dbt models on DuckDB.
- The signal set is illustrative, not the state of the art. Real nowcasting models use dozens of series and mixed-frequency methods (MIDAS, dynamic factor models). The point here is the honest pipeline, not beating the ECB.
- Publication-lag gating, not archived vintages — see the note above. The ablation prices the gating; it cannot price the revision history, because the ECB portal serves only the latest print.
- The ablation is one comparison, not a significance test. 192 months of walk-forward errors, one RMSE each — a gap of a point or two is inside the noise, and the table is read as "no material difference" rather than as a ranking. A Diebold-Mariano test on the paired errors would be the next step if the gap ever looked meaningful.
- Euro area, headline HICP only. Core inflation, components, and other regions are catalogue additions, not redesigns.
- A small linear model. Deliberately — it is interpretable and hard to overfit on monthly data, and it makes the backtest the star rather than the estimator.
This is an educational data-engineering project that estimates a public macroeconomic statistic. It is not financial or investment advice.
Code: MIT. Data: European Central Bank Data Portal (freely available, © ECB).