diff --git a/CHANGELOG.md b/CHANGELOG.md index a45a194..1ca2e27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [1.13.0] - 2026-10-06 + +### Added +- **Metrics next to the model.** A second, optional file beside the model, + `.metrics.yml`, says what the model's numbers mean, defined by its own spec, [metrics + spec 0.1.0](model2data/spec/metrics/README.md), with a JSON Schema at + `https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json`. A metric is a + column aggregated (`measure: orders.total_amount`, by `agg` or the column's own `measure`), a + row count (`count: orders`), a ratio of two metrics' totals (`ratio: {numerator, denominator}`), + or arithmetic over metrics (`expression: revenue - refunds`). Simple metrics and row counts take + a `where` filter (a value, a list, or `eq`, `ne`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, + `between`, `is_null`, combined with `any` and `all`, with nulls as in SQL) and a `time` column; + a filter may name a column of any table the metric's table reaches through its foreign keys, + many-to-one, along exactly one path. Every `measure` column of the model also yields a metric of + its own (`orders_total_amount`) and every fact a row count (`orders_count`), named as model2data + studio's semantic export names them; `infer: false` turns that off, and a metric of the same + name in the file replaces the inferred one. `format: percent` expects a fraction (0.6 shows as + 60%) and is passed on as a percent format, never multiplied. `dimensions:` offers a column as a dimension, with a + label, or never. **The generator never reads the file**: a model generates the same data with + or without one. +- **`model2data validate` checks metrics files** (`*.metrics.yml`): the schema, then against the + model, `--model`, or else the model beside it with the same stem. It names every unknown table + or column, an aggregation that does not fit its column, an unknown metric in a ratio or + expression, a cycle between metrics, a filter column no path reaches or two paths reach (naming + both), a value that is not a member of its enum or not of its column's type, and a comparison + on an enum, boolean or text column; it warns about a metric with no date to report it by and a + filter that can never match the generated data (a `gt` above a column's `max`). +- **`generate --metrics FILE`** adds, and changes no other file: + `metric_values.json`, each metric's **known value** over the generated data, overall and per + calendar month (counts exact, everything else rounded to 6 decimal places, half to even; the + same seed gives the same bytes); a singular dbt test per metric, + `data-tests/metrics/metric_.sql`, that computes it over the project's staging models and + fails unless it gets that value, so `dbt build` proves the metric logic on the warehouse, with + defects and `--days` too; and `osi/.yml`, the model and its metrics as an Apache Ossie + (Open Semantic Interchange) 0.1.1 semantic model, as tagged `osi-0.1.1-rc1` in apache/ossie (no + final 0.1.1 tag exists yet; 0.2.0 is an unreleased draft). A + metrics file beside the model is not picked up on its own: an unchanged command keeps writing + the same files. +- **`model2data metrics list`** shows every metric, inferred ones included, and **`model2data + metrics export --to ossie`** writes the Ossie file without generating. What Ossie 0.1.1 has no + field for (a metric's label, format and time, an enum's members, a table's role and grain) goes + in `custom_extensions` and is listed at the top of the file. +- **`model2data.metrics`**, the Python API: `load` and `validate` a metrics file, `resolve` it with + its model into one semantic representation (entities, relationships, dimensions, measures and + metrics with their join paths), `known_values` over a run's frames, `to_ossie`, and + `write_metric_tests`. +- **`examples/ecommerce.metrics.yml`**: revenue, orders, average order value and units sold for + the e-commerce example, for `--metrics`. + +### Changed +- The spec's example model, `model2data/spec/examples/coffee_webshop.model2data.yml`, marks + `orders.total_amount` and `order_items.quantity` as measures, for the metrics example beside + it. No model under `examples/` changed, and no output changed for any model. +- **The README and the PyPI page were rewritten** to show what model2data does today: what you + get, one capability per line with a link; a before and after of a real run with metrics; who + it is for; and how it works, with a new demo GIF. The reference that was in the README moved to + `docs/`, one page per topic (the model file, generating, the dbt project, days, history, time + shapes, distributions, defects, metrics, CI, agents), worded as it was. + ## [1.12.0] - 2026-10-06 ### Changed diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 57955a0..420dd65 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -163,7 +163,8 @@ Releases are published to PyPI automatically when a GitHub Release is published 4. The floating major tag (`uses: JB-Analytica/model2data@v1`) must follow the release. `.github/workflows/major-tag.yml` does it on `release: published` for `vX.Y.Z` tags. By hand: `git tag -f v1 v1.10.0 && git push -f origin v1`. Also bump the default `version` input - in `action.yml`, and the versions in the README's CI snippets, to the new release. + in `action.yml`, and the versions in the CI snippets of + `docs/ci.md`, to the new release. 5. Confirm the new version is live: `pip index versions model2data`, or check `https://pypi.org/project/model2data//` (the PyPI JSON API can lag the actual upload by a minute or two — the version page itself is the more immediate check). diff --git a/LLMS.md b/LLMS.md index 3337214..ad7cce3 100644 --- a/LLMS.md +++ b/LLMS.md @@ -236,7 +236,8 @@ generated project), a `TableGroup` its `groups`. for parsing and generate independent per-column FK values, but — same caveat as above — the combination isn't guaranteed to match a real parent row unless separately enforced. - Only DuckDB (default, zero-config) and Postgres (`--adapter postgres`, needs - `pip install "model2data[postgres]"` and connection env vars — see README.md) are supported + `pip install "model2data[postgres]"` and connection env vars — see + [docs/dbt-project.md](docs/dbt-project.md)) are supported targets today. - To show that a project's tests fire (or to teach dbt), generate with `--defects training`, or list defects per table in a `.model2data.yml` (`defects: [{type: nulls, column: x, count: @@ -245,6 +246,18 @@ generated project), a `TableGroup` its `groups`. - For a source that keeps its own history (an SCD2 table to snapshot or join point-in-time), set `history: true` in the table's `incremental` and generate with `--days N`: `_history` gets every version with `valid_from`, `valid_to` and `is_current`. +- When the user names the numbers that matter (revenue, orders, conversion), write them in a + metrics file beside the model, `.metrics.yml` (metrics spec 0.1.0, + `model2data/spec/metrics/README.md`): `measure: table.column` with `agg`, `count: table`, + `ratio: {numerator, denominator}` or `expression: a - b` over metric names, and `where` filters + (`table.column: value`, a list, or `{ne|in|not_in|gt|gte|lt|lte|between|is_null: ...}`, combined + with `any`/`all`). `format: percent` means the value is a fraction (0.6 shows as 60%): write + shares as `a / b`, never `* 100`. A filter column must be on the metric's table or reachable through foreign + keys child-to-parent along one path. Run `model2data validate .metrics.yml`, then generate + with `--metrics .metrics.yml`: `metric_values.json` holds each metric's known value, and + `dbt build` runs a test per metric that must pass. A failing metric test means the metric's SQL + and the data disagree; never edit `metric_values.json` to make it pass. The file must be passed; + `generate` does not pick it up from beside the model. - model2data requires dbt-core >= 1.11 (tracking dbt's own supported-version policy). Everything here, `--unit-tests` included, works with a plain `pip install model2data`. @@ -280,7 +293,15 @@ model2data --file SCHEMA.dbml [OPTIONS] tests are seen to fail; writes defects_report.json and EXPECTED_FAILURES.md into the project. none: ignore every defect, the model's own too +--metrics PATH The model's metrics file: adds metric_values.json (known values), + osi/NAME.yml (Apache Ossie 0.1.1) and data-tests/metrics/ (a dbt test + per metric). Changes no other file + +model2data validate FILE... [--model MODEL] a metrics file (*.metrics.yml) is checked against + --model, or the model beside it with its stem +model2data metrics list -f MODEL [--metrics FILE] every metric, inferred too +model2data metrics export -f MODEL [--metrics FILE] --to ossie [-o FILE] without generating ``` -For anything not covered here, see [README.md](README.md) — this file exists to make a schema +For anything not covered here, see [README.md](README.md) and [docs/](docs/README.md) — this file exists to make a schema → demo turnaround fast, not to duplicate the full documentation. diff --git a/README.md b/README.md index 1b48521..8e72b27 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # model2data [![PyPI](https://img.shields.io/pypi/v/model2data)](https://pypi.org/project/model2data/) +[![PyPI downloads](https://img.shields.io/pypi/dm/model2data)](https://pypi.org/project/model2data/) +[![Python versions](https://img.shields.io/pypi/pyversions/model2data)](https://pypi.org/project/model2data/) [![CI](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml/badge.svg)](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml) [![codecov](https://codecov.io/gh/JB-Analytica/model2data/branch/main/graph/badge.svg)](https://codecov.io/gh/JB-Analytica/model2data) [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) @@ -11,26 +13,16 @@ Built and maintained by [JB Analytica](https://www.jbanalytica.com/) — data pl Give `model2data` a data model — a `.model2data.yml` file, or a [DBML](https://dbml.dbdiagram.io/docs/) schema hand-written or exported from an existing database — and it generates realistic, -relationship-preserving synthetic data -*and* a complete, runnable dbt project around it: seeds, staging models, tests, and a -DuckDB or Postgres profile. No sample data to hunt down, no dbt boilerplate to hand-write, no -production data to risk exposing. +relationship-preserving synthetic data *and* a complete, runnable dbt project around it: seeds, +staging models, tests, and a DuckDB or Postgres profile. Add a metrics file and it also knows +what your revenue comes to, and writes a dbt test that proves it. No sample data to hunt down, +no dbt boilerplate to hand-write, no production data to risk exposing. -Driving model2data from a coding agent? Have it start here; the instructions ship with the -installed version and pick a topic from the current directory: - -```bash -uvx model2data@latest guide -``` - -A working analytics stack — real (synthetic) data, tested dbt models, queryable in -DuckDB — from a schema file, in seconds: - -![model2data generating a project and running it with dbt](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/demo.gif) +![model2data generating a project with metrics and running it with dbt](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/demo.gif) ```bash pip install model2data -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 # examples/ is in this repository cd dbt_ecommerce && dbt build ``` @@ -41,220 +33,270 @@ cd dbt_ecommerce && dbt build --- -## Why this exists +## What you get + +**Values that look real.** Column names are matched against about 40 common patterns — `email`, +`first_name`, `city`, `phone`, `company`, ... — so a column called `email` gets real-looking +emails, not `Lorem ipsum`. Type a column with any Faker provider (`billing_country state`, +`sku ean13`) to pick its generator outright, and `--locale nl_BE` makes every person and address +come from one country. → [The model file](docs/model-file.md) + +**Foreign keys that resolve.** Tables are generated in dependency order and every foreign key +points at a parent row that exists — onto a primary key or a unique one. Composite primary and +unique keys hold, and a column that repeats a parent's value through a foreign key +(`orders.customer_email`) holds the email of the order's own customer. +→ [The model file](docs/model-file.md) + +**A dbt project that runs.** Seeds, staging models that `ref()` them, `not_null`, `unique`, +`relationships` and `accepted_values` tests, data tests written from the model's own hints, and +a DuckDB (zero-config) or Postgres profile. One `dbt build` loads, builds and tests it, and +`--unit-tests` adds dbt unit test fixtures. → [The generated dbt project](docs/dbt-project.md) + +**The same files, byte for byte.** The same model, `--seed`, `--as-of` and options give the same +files on any machine, any day, with the same model2data version — safe to commit as fixtures, +safe to diff in CI. `--table-seed orders=7` re-rolls one table and leaves every other one +byte-identical. → [The determinism promise](docs/generating.md#the-determinism-promise) + +**Days after the first.** A table marked `incremental` moves on day by day: `--days 7` or +`--next` adds new rows and updates existing ones, an enum with `transitions` moves an order from +`pending` to `shipped`, and each day is written as a batch or a changelog to load incrementally. +→ [Days after the first](docs/days.md) + +**History tables.** `history: true` writes `
_history`, every version of every row with +`valid_from`, `valid_to` and `is_current`: an SCD type 2 source to build snapshots against. +→ [A source that keeps its history](docs/history.md) + +**When things happen.** `--business-hours`, `--growth` and `--seasonality` shape every timestamp +(weekdays and working hours, a trend, a Q4 peak), or one column at a time. `updated_at` never +lands before `created_at`, `after` orders any two columns, and `when` gives `shipped_at` a value +on shipped and delivered orders only. → [When things happen](docs/time-shapes.md) + +**How the data is spread.** `--skew` lets a few customers place most of the orders; `weights`, +`true_rate`, `null_rate` and `distinct` shape a column; `distribution` draws a number from a +normal, lognormal or exponential curve. → [How the data is spread](docs/distributions.md) + +**Defects on purpose.** `--defects training` breaks each standard dbt test once, `messy` puts a +small share of every defect on every table, and a table can list its own: duplicate keys, +orphans, nulls, invalid values, messy text, late-arriving rows. Every run with defects writes +`EXPECTED_FAILURES.md`, so a test that does not fire shows. → [Defects on purpose](docs/defects.md) + +**Metrics with known values** (new in 1.13.0). A metrics file beside the model defines revenue, +orders, average order value. The run writes each metric's value over the generated data to +`metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the +warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`. +→ [Metrics with known values](docs/metrics.md) + +**Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub +annotations; `uses: JB-Analytica/model2data@v1` fails a pull request that breaks one. A +pre-commit hook and a GitLab CI snippet do the same. → [Validate models in CI](docs/ci.md) + +**Use it from an agent.** `uvx model2data@latest guide` prints instructions written for a coding +agent, shipped with the installed version; [LLMS.md](LLMS.md) takes an LLM from a plain-English +description to a running project. → [Agents and LLMs](docs/agents.md) + +**Or in the browser.** [model2data studio](#model2data-studio--the-same-engine-in-the-browser) +is the same engine with a DBML editor and a live diagram. + +--- + +## Before and after + +In: a model (here the `orders` table of +[`examples/ecommerce.model2data.yml`](examples/ecommerce.model2data.yml)) and, optionally, the +metrics that matter ([`examples/ecommerce.metrics.yml`](examples/ecommerce.metrics.yml)): + +```yaml + orders: + columns: + id: {type: bigint, pk: true, not_null: true} + customer_id: {type: bigint, not_null: true, references: customers.id} + order_date: {type: timestamp, not_null: true} + total_amount: + type: numeric + not_null: true + generate: {min: 10, max: 5000} + status: order_status +``` + +```yaml +metrics: + revenue: + label: Revenue + measure: orders.total_amount + agg: sum + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + format: currency + orders: + count: orders + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + average_order_value: + ratio: {numerator: revenue, denominator: orders} +``` + +```bash +model2data --file examples/ecommerce.model2data.yml --metrics examples/ecommerce.metrics.yml \ + --rows 200 --seed 42 --as-of 2026-01-31 +``` + +Out: a dbt project. + +``` +dbt_ecommerce/ +├── seeds/raw/ customers.csv, orders.csv, order_items.csv, products.csv, +│ product_reviews.csv, __seed_config.yml +├── models/staging/ stg_
.sql + stg_
.yml (with tests), for each table +├── data-tests/metrics/ metric_revenue.sql, metric_orders.sql, +│ metric_average_order_value.sql, metric_units_sold.sql +├── macros/ generate_schema_name.sql, model2data_hint_tests.sql +├── osi/ecommerce.yml the model and its metrics as Apache Ossie 0.1.1 +├── metric_values.json each metric's known value +├── ecommerce.model2data.yml the model, and +├── ecommerce.metrics.yml its metrics, copied as given +├── dbt_project.yml +└── profiles.yml DuckDB +``` + +`dbt build` loads the seeds, builds the staging models and runs every test, the four metric +tests included: + +``` +Finished running 5 seeds, 49 data tests, 5 view models in 0 hours 0 minutes and 0.71 seconds (0.71s). + +Completed successfully + +Done. PASS=59 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=59 +``` + +And `metric_values.json` says what a dashboard on this data must show: + +```json +{ + "model2data-metrics": "0.1.0", + "model": "ecommerce", + "run": { + "seed": 42, + "as_of": "2026-01-31" + }, + "rounding": "counts, and sums, minimums and maximums of integer columns, are exact; every other value is rounded to 6 decimal places, half to even", + "metrics": { + "revenue": { + "kind": "simple", + "label": "Revenue", + "value": 220291.8, + "time": "orders.order_date", + "by_month": { + "2025-01": 3967.88, + "2025-02": 10676.66, + ... + "2026-01": 31879.13 + } + }, + "orders": { "kind": "count", "label": "Orders", "value": 92, ... }, + "average_order_value": { "kind": "ratio", "label": "Average order value", "value": 2394.476087, ... }, + "units_sold": { "kind": "simple", "label": "Units sold", "value": 734, ... } + } +} +``` + +Run it again tomorrow, or on another machine, with the same model2data version, and every one +of those files is the same. + +--- + +## Who it's for Analytics engineers hit the same wall constantly: you need realistic data to build or test a pipeline, but production data is off-limits (privacy, access, scale), and hand-rolling mock CSVs -is tedious and doesn't scale past two tables. `model2data` closes that gap — from a schema -definition to a seeded, tested dbt project you can actually run, with no database or production -access required. - -- **Privacy-safe.** Nothing but a schema definition goes in; nothing but synthetic data comes out. -- **Realistic, not random.** Column names are matched against ~35 common patterns — `email`, - `first_name`, `city`, `phone`, `company`, ... — so a column called `email` gets real-looking - emails, not `Lorem ipsum` text. Type a column with any Faker provider (`billing_country state`, - `sku ean13`) to pick its generator outright when the name is wrong for the data. -- **Relationship-preserving.** Foreign keys resolve to real parent rows; tables are generated in - dependency order. -- **Deterministic, byte for byte.** The same model, `--seed` and `--as-of` (and the same options) - give the same files, byte for byte, on any machine and any day with the same model2data - version — safe to commit fixtures, safe to diff across CI runs. See - [The determinism promise](#the-determinism-promise). -- **Re-rollable one table at a time.** `--table-seed orders=7` regenerates a single table and - leaves every other table byte-identical, so you can keep the four tables that look right. -- **A real dbt project, not just CSVs.** Seeds, staging models that `ref()` them, schema tests, - and a ready-to-use profile — the thing you'd otherwise spend an afternoon scaffolding by hand. - A single `dbt build` loads, transforms, and tests the whole thing. - -## Who is model2data for? - -- **Analytics engineers** — generate realistic datasets and a working dbt project without - waiting on production access. -- **Data engineers** — produce deterministic test data from an existing schema for pipeline and - migration testing. -- **Software & data teams** — prototype integrations and analytics workflows without exposing - production data. -- **Consultants & architects** — spin up realistic environments for demos, workshops, and - architecture validation in minutes, not hours. +is tedious and doesn't scale past two tables. model2data closes that gap. Nothing but a schema +definition goes in; nothing but synthetic data comes out. + +- **A consultant who needs a client demo before source access.** Write the client's model from a + kick-off call, generate a year of their data, and show a working dbt project and dashboard in + the first week, without touching production. +- **An analytics engineer who wants fixtures in CI.** Commit the model, generate with `--seed` + and `--as-of`, and every CI run builds against the same bytes. `--days` gives an incremental + model the next day's batch to process. +- **A trainer who needs data that fails on purpose.** `--defects training` breaks each standard + dbt test once and says which, so a class sees `unique`, `not_null`, `relationships` and + `accepted_values` fail for the right reason. +- **A team defining metrics once.** Write revenue in one metrics file, check every number the + semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL, + and hand the definitions on as Apache Ossie. ## How it works ```mermaid flowchart LR - subgraph input [" "] - A["📄 model\n.model2data.yml or DBML"] - end + A["model
.model2data.yml or DBML"] + M["metrics file, optional
.metrics.yml"] subgraph m2d ["model2data"] direction LR - B["Parse\ntables, columns,\nrelationships"] --> C["Generate\nFaker + name-aware\ninference, FK-aware"] - C --> D["Scaffold\nseeds · staging models\ntests · profile"] + B["Read and validate
tables, keys, hints,
metrics"] --> C["Generate
name-aware values,
FK-aware, seeded"] + C --> D["Scaffold
dbt project"] + C --> K["Compute
each metric's
known value"] end subgraph output ["Generated dbt project"] direction TB E["seeds/*.csv"] - F["models/staging/*.sql + *.yml"] - G["profiles.yml\n(DuckDB or Postgres)"] + F["models/staging/
*.sql + *.yml tests"] + G["data-tests/metrics/"] + J["metric_values.json"] + O["osi/ Apache Ossie"] + P["profiles.yml
DuckDB or Postgres"] end A --> B - D --> E - D --> F - D --> G - E & F & G --> H["dbt build"] - H --> I[("Analytics-ready\ndataset")] + M --> B + D --> E & F & P + K --> G & J & O + E & F & G & P --> H["dbt build"] + H --> I[("Tested dataset,
metrics proven")] classDef m2dStyle fill:#0A3866,stroke:#2196F0,color:#F6F8FB classDef outStyle fill:#182333,stroke:#A8C9EE,color:#F6F8FB classDef endStyle fill:#FA9306,stroke:#FA9306,color:#182333 - class B,C,D m2dStyle - class E,F,G outStyle + class B,C,D,K m2dStyle + class E,F,G,J,O,P outStyle class H,I endStyle ``` 1. **Read.** Reads the model — its tables, columns, keys, references and generation hints — from a `.model2data.yml` document, checking it against [the spec](model2data/spec/README.md), or - from DBML, which it converts to the same model first. + from DBML, which it converts to the same model first. A metrics file is checked against + [its spec](model2data/spec/metrics/README.md) and against the model. 2. **Generate.** Produces synthetic values per column — typed generation for known SQL types (int, date, timestamp, ...), then a Faker provider named as the type (`sku ean13`), then name-aware inference for everything else (`email`, `phone`, `city`, ...), foreign keys - resolved against already-generated parent rows. + resolved against already-generated parent rows. The metrics file is never read here: the data + is the same with or without it. 3. **Scaffold.** Writes a complete dbt project around that data: CSV seeds, staging models that `ref()` those seeds, `not_null`/`unique`/`relationships` tests, `accepted_values` tests for - enum-typed columns, singular SQL tests for composite primary/unique keys, table and column - `description:` fields from the model, and a profile for DuckDB (zero-config, file-based) or - Postgres. + enum-typed columns, singular SQL tests for composite primary/unique keys, tests from the + model's hints, table and column `description:` fields from the model, and a profile for + DuckDB (zero-config, file-based) or Postgres. +4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes + its known value, a dbt test that checks it, and the Apache Ossie file. --- -## The model file - -A model is one YAML document, `.model2data.yml` — or the same document as JSON. Its -format is [spec 0.4.0](model2data/spec/README.md), with a JSON Schema -([`model.schema.json`](model2data/spec/model.schema.json)) your editor can autocomplete and check -against: - -```yaml -# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/0.2.0/model.schema.json -model2data: 0.2.0 -name: coffee_webshop - -enums: - order_status: [pending, paid, shipped, delivered, cancelled] - -tables: - customers: - description: One row per registered customer account - columns: - id: {type: bigint, pk: true} - email: {type: email, unique: true, not_null: true} - phone: - type: phone_number - generate: {null_rate: 0.4} - - orders: - columns: - id: {type: bigint, pk: true} - customer_id: - type: bigint - not_null: true - references: customers.id - generate: {skew: 0.8} - total_amount: - type: numeric - not_null: true - generate: - min: 10 - max: 5000 - distribution: {kind: lognormal, median: 120, spread: 0.7} - status: - type: order_status - generate: - weights: {delivered: 20, cancelled: 1} - -run: - seed: 1 - as_of: 2026-01-01 -``` - -A column's `type` is an SQL type, an enum of the model, or a generator (`email`, `first_name`, -`ean13`, ...). `generate` holds its hints, each a typed value the schema documents. `run` saves -the generation settings with the model; every CLI option overrides the one it names. The -[reference example](model2data/spec/examples/coffee_webshop.model2data.yml) is a complete model, -and every schema in [`examples/`](examples/) comes as one. - -Check a model without generating anything — every issue is printed with its path in the document: - -```bash -model2data validate examples/ecommerce.model2data.yml -``` - -### DBML is supported input - -`--file` also takes a `.dbml` file, which model2data converts to the same model before generating -(parsed with [pydbml](https://github.com/Vanderhoof/PyDBML)). Hints written as a JSON note on a -column (`[note: '{"min": 1, "max": 5}']`) become its `generate`; any other note is its -description. `convert` writes the model a DBML file converts to, for you to keep: +## Installation ```bash -model2data convert examples/ecommerce.dbml -o ecommerce.model2data.yml +pip install model2data # DuckDB included +pip install "model2data[postgres]" # to target Postgres with --adapter postgres +uvx model2data --help # or run it without installing ``` -The spec's [From 0.1](model2data/spec/README.md#from-01) section lists how each DBML feature -converts. A few newer or rarer DBML features — `check` constraints, `Records`, `TablePartial`, -unquoted non-ASCII names — can't be read yet, and are refused with the line and what to change. +Python 3.10 to 3.14. The examples used on this page are in [`examples/`](examples/). --- -## Validate models in CI - -Teams that keep `*.model2data.yml` models in git can fail a pull request that breaks one. -`model2data validate` takes several files and `--glob` patterns, prefixes each issue with its -file, exits 1 if any model has an error, and with `--format github` prints annotations that show -inline on the pull request's Files tab. `--require-files` fails when nothing matched. - -**GitHub Actions** - -```yaml -name: model2data -on: - pull_request: - paths: ["**/*.model2data.yml"] -jobs: - validate: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v5 - - uses: JB-Analytica/model2data@v1 -``` - -Inputs: `files` (glob, default `**/*.model2data.yml`), `version` (default: the release the tag -points at, e.g. `1.12.0`; a specifier such as `>=1.10,<2` also works), `python-version` (3.12). - -**pre-commit** - -```yaml -repos: - - repo: https://github.com/JB-Analytica/model2data - rev: v1.12.0 - hooks: - - id: model2data-validate -``` - -**GitLab CI** - -```yaml -model2data: - image: python:3.12-slim - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - changes: ["**/*.model2data.yml"] - script: - - pip install model2data==1.12.0 - - model2data validate --glob "**/*.model2data.yml" -``` - ## model2data studio — the same engine, in the browser [**model2data studio**](https://studio.jbanalytica.com) puts everything on this page behind a @@ -279,396 +321,39 @@ model gets designed and shown. --- -## Installation - -```bash -pip install model2data -``` - ---- - -## Quick start - -We bundle several example schemas in `examples/` — this walkthrough uses the e-commerce one -(`examples/ecommerce.model2data.yml`: customers, products, orders, order items, and reviews). - -Generate a project with synthetic data: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 -``` - -This creates a `dbt_ecommerce/` folder with your data and dbt setup. - -Real schemas are rarely uniform. `--rows-for` sizes individual tables, so a handful of customers -can sit behind a large orders table the way they would in the warehouse you're modelling: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 \ - --rows-for customers=50 --rows-for order_items=5000 -``` - -`--seed` reproduces a run's numbers, but dates and timestamps are generated relative to the -current date, so the same seed drifts once the day turns over. `--as-of` pins the date they're -anchored on, and the whole dataset reproduces on any later day — which is what makes a generated -fixture safe to commit: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --as-of 2026-01-31 -``` - -### The determinism promise - -The same model, seed and `--as-of` — with the same options (`--rows`, `--rows-for`, -`--table-seed`, `--locale`, `--defects`, `--days`, …) — produce the same files, byte for byte: - -- on any machine and operating system, and on every supported Python version; -- with any Faker or pandas version model2data accepts; -- with the same model2data version, whenever you run it. - -A new release may change what a seed produces, to fix a bug or make the data more realistic. -When it does, the changelog says which models are affected and why. Pin the version -(`model2data==1.11.0`) when fixtures must stay the same across upgrades. - -`tests/test_determinism.py` holds the file hashes of a spread of models, presets, multi-day runs -and locales, and runs on every pull request — against the newest Faker and pandas too — so a -change to a single output byte fails the build unless it is deliberate and in the changelog. - -If one table comes out wrong and the rest looks right, `--table-seed` re-rolls just that table. -Every other table's seed CSV stays byte-identical, and children of the re-rolled table still -reference rows that exist, so there's nothing to re-check but the table you asked to change: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --table-seed orders=7 -``` - -`--locale` picks the country every generated person and address comes from (`en_US` by default); -it's a per-run setting, so a table can't end up holding one Belgian and one American address. A -`country` column that sits beside a `city`/`street`/`state`/`postcode` column always agrees with -that place; a `country` column with none of those beside it isn't describing anyone's address, so -it reads as an international mix instead, with the locale's own country the most common: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --locale nl_BE -``` - -### Generate the next days - -A table with `incremental` in the model moves on day by day: each day adds `new_per_day` rows -(their dates and timestamps falling on that day) and updates `update_rate` of the rows it already -holds, and an enum column with `transitions` moves from a state to one of the states allowed to -follow it. See `examples/ecommerce_daily.model2data.yml` and "Days after the first" in the -[spec](model2data/spec/README.md). - -```bash -model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --days 7 -model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --next -``` - -`--days N` generates day 0, the run you would get without it, and N days after it (`--next` is -`--days 1`). The dbt seeds hold the state after the last day. `--days-format` picks what else is -written beside the project, outside `seeds/` so dbt does not load it: `batches` (the default), -`days/
/day_000.csv` whole and then one file per day with the rows inserted and updated that -day; `changelog`, `changelog/
.csv` with a `_day` and an `_op` (`insert` or `update`) column -on every row; or `final`, nothing more. A day depends only on the seed, `--as-of`, the day and the -model, so day *n* never changes when you generate more days or add an unrelated table. From Python: - -```python -from model2data.generate.days import generate_days -from model2data.model import load - -days = generate_days(load("examples/ecommerce_daily.model2data.yml"), 7, seed=42) -days[3].tables["orders"].inserted # rows day 3 added -days[3].tables["orders"].updated # rows day 3 changed, with their new values -days[3].tables["orders"].state # the table after day 3 -``` - -### Shape when things happen - -By default, every date and timestamp is drawn uniformly across its window. `--business-hours`, -`--growth`, and `--seasonality` shape that instead — weekdays and working hours, a trend across -the window, and an annual cycle peaking in Q4: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 \ - --business-hours --growth 0.5 --seasonality 0.3 -``` - -Within a row, created/updated/deleted-style columns are ordered automatically — `updated_at` never -lands before its own `created_at` — under any profile, uniform included. A column whose name -doesn't say what it depends on can say so explicitly with an `after` hint: - -```yaml -shipped_at: - type: timestamp - generate: {after: ordered_at} -``` - -A column that only has a value in some states says which with `when` (spec 0.4.0: the model says -`model2data: 0.4.0`): it holds a value on -exactly the rows whose named column holds one of the listed values, and is null on every other -row — no `pending` order with a `shipped_at`, no `cancelled` subscription without a -`cancelled_at`: - -```yaml -shipped_at: - type: timestamp - generate: {after: ordered_at, when: {status: [shipped, delivered]}} -``` - -The named column is another column of the table (an enum, boolean, number or text column; several -named must all match), and the column carrying `when` is nullable. On the matching rows it is -never null, unless it has a `null_rate`, which then counts only those rows. On days after the -first (`--days`), a row that moves into a listed status gets its value that day and a row that -moves out loses it. Every other column comes out exactly as it would without the hint. - -The flags above (or `run.shape` in the model) shape every date and timestamp column the same -way, run-wide. `business_hours`, `growth`, and `seasonality` column hints override that for one -column at a time — the whole point being a run can be uniform everywhere except the one column -that needs shaping, or shaped everywhere except the one column that shouldn't be: - -```yaml -orders: - columns: - id: {type: int, pk: true} - created_at: - type: timestamp - generate: {business_hours: true, growth: 0.4} - refunded_at: - type: timestamp - generate: {growth: 0} -``` - -Here `created_at` gets business hours and growth even on an otherwise-uniform run, while -`refunded_at` stays flat even under `--growth 0.5` — each hint only replaces the fields it names, -so a partial hint like `{growth: 0}` leaves that column's `business_hours`/`seasonality` at -whatever the run-level flags set. - -### Shape how the data is spread - -By default every parent row is equally likely to be picked for a child row, and every column -gets the same generic null rate, value spread, and true/false split. `--skew` changes the first -part: `0.0` is that uniform default, `1.0` means a handful of parents hold most of the children — -"a fifth of the customers place most of the orders": - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --skew 0.8 -``` - -Column hints shape the rest, per column: - -```yaml -orders: - columns: - id: {type: int, pk: true} - customer_id: - type: int - references: customers.id - generate: {skew: 0.9} - status: - type: order_status - generate: - weights: {delivered: 20, cancelled: 2} - is_paid: - type: boolean - generate: {true_rate: 0.9} - discount_code: - type: varchar - generate: {null_rate: 0.8} - shipping_city: - type: varchar - generate: {distinct: 12} -``` - -`skew` on a foreign key overrides `--skew` for just that column. `weights` biases an enum column -toward the values named (unnamed values still appear, at weight 1). `true_rate` is the fraction of -non-null rows a boolean column comes back `true`. `null_rate` replaces the column's default null -fraction outright. `distinct` draws the column's values from a fixed-size pool instead of a fresh -value per row — a `shipping_city` most warehouses only ever see a handful of. - -### Shape a number's distribution - -`min`/`max` alone only ever drew uniformly between them. A `distribution` hint on an integer or -decimal column picks a different shape instead: - -```yaml -total_amount: - type: numeric - generate: - min: 5 - distribution: {kind: lognormal, median: 80, spread: 0.6} -``` - -`normal` takes `mean` (the centre) and `stddev` (the spread); `lognormal` takes `median` (the -typical value) and `spread` (how heavy the tail is — 0.3 is mild, 1.0 is heavy); `exponential` -takes `mean` (the average). Any left unset default to the midpoint of the column's effective -`min`/`max` (or `stddev` = range / 6, `spread` = 0.5). `min`/`max` still clip the result — a -`normal` centred near an edge redraws a bounded number of times before clamping, so it never -loops forever and never crosses the bound. Leaving `distribution` out, or setting it to -`uniform`, is exactly today's behaviour. - -Run dbt to load, transform, and test the data: - -```bash -cd dbt_ecommerce -dbt build -``` - -Staging models `ref()` their seeds, so a single `dbt build` loads the seeds, builds the models, -and runs every generated test in one dependency-ordered pass — no separate `dbt seed`/`dbt run` -needed, even on a brand-new database. (The individual `dbt deps`, `dbt seed`, and `dbt run` -commands still work if you'd rather drive the steps yourself; the generated project declares no -packages, so `dbt deps` is a no-op.) - -Your analytics-ready dataset is now in DuckDB! - -To target Postgres instead, install the extra and pass `--adapter postgres`: - -```bash -pip install "model2data[postgres]" -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --adapter postgres -``` - -Connection details are read from environment variables (`MODEL2DATA_PG_HOST`, `MODEL2DATA_PG_PORT`, `MODEL2DATA_PG_USER`, `MODEL2DATA_PG_PASSWORD`, `MODEL2DATA_PG_DATABASE`), defaulting to `localhost:5432` with a `postgres`/`postgres` user for local development. - -After generation, the CLI prints a short summary — tables and rows generated, relationships found in the model, and any columns that fell back to generic placeholder text because neither their type nor name could be matched. - -Pass `--unit-tests` to also generate deterministic dbt unit test fixtures (`models/staging/ut_stg_
.yml`) from the actually-generated seed rows: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --unit-tests -``` - -This targets dbt-core's native unit testing feature, which works out of the box with the base -install — see [dbt-core versions](#dbt-core-versions) below. - -Your model's hints also write **data tests** for the dbt project, so the model that generates the -fixtures guards the real pipeline too. Every hint that states a constraint becomes a generic test -on the staging model, shipped as a self-contained macro in `macros/model2data_hint_tests.sql` (no -dbt package, so `dbt build` still works offline): - -| In the model | dbt test | Tolerance | -|---|---|---| -| `generate: {min, max}` on a numeric column | `model2data_between` | none | -| `generate: {after: other}` | `model2data_not_before` (where both are not null) | none | -| `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 | -| `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none | -| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none | -| `grain` on a table | `model2data_unique_combination` | none | -| an enum-typed column | `accepted_values` (always written) | none | - -`true_rate`, `weights`, `skew`, `distribution` and the temporal shape hints (`business_hours`, -`growth`, `seasonality`) describe a statistical shape rather than a constraint a row can break, so -they write no test. `--hint-tests {error,warn,off}` sets these tests' severity (default `warn`: -they describe intent and should not break a first `dbt build` on real data); the structural tests -(`not_null`, `unique`, `relationships`) are unaffected. A model with no such hints generates the -same YAML as before. To call the mapping from Python, `model2data.dbt.hint_tests.hint_tests_for(model)` -returns the tests as data. - -### Break the data on purpose - -A dbt test you have never seen fail is a test you hope works. `--defects` puts deliberate, -counted defects in the generated data, so `dbt build` fails exactly the tests it should — for -teaching dbt, or for proving a project's tests fire: - -```bash -model2data --file examples/ecommerce.model2data.yml --seed 42 --defects training -cd dbt_ecommerce && dbt build # fails unique, not_null, relationships, accepted_values once each -``` - -`training` breaks each kind of standard dbt test once; `messy` puts a small share of every defect -on every table; `clean` (the default) is the data as it always was, byte for byte; `none` also -ignores the defects a model's tables list, for the clean data of a training model. A table can -also list its own defects in the model (spec 0.3.0): - -```yaml -orders: - defects: - - {type: duplicate_keys, count: 3} # fails unique - - {type: orphan_foreign_keys, column: customer_id, share: 0.02} # fails relationships - - {type: nulls, column: order_date, count: 5} # fails not_null -``` - -The types are `duplicate_keys`, `orphan_foreign_keys`, `nulls`, `invalid_values` (fails -`accepted_values`), `messy_text` (whitespace and casing to clean in staging), and, with `--days`, -`late_arriving` (rows an incremental model filtering on `updated_at > max(updated_at)` skips) and -`late_updates` (versions a `strategy: timestamp` snapshot misses) and `overlapping_history` (see -below). Every run with defects writes -`defects_report.json` (each defect, the rows it broke, the tests it breaks) and -`EXPECTED_FAILURES.md` into the project, so a test that does not fire is visible. Defects are applied -after the clean data, from their own random stream: the same seed gives the same bytes, and a -defect on one table moves nothing in another. From Python, `model2data.defects.planned_defects` -says what a run will break, `model2data.output.finish_run` applies it (and builds the histories) -after `generate_days`, and `write_defects_report` / `write_expected_failures` write the two files. - -### A source that keeps its history - -`history: true` in a table's `incremental` (spec 0.3.0) also writes `
_history`: every -version of every row over the generated days, with `valid_from`, `valid_to` (null for the current -version) and `is_current` — an SCD type 2 source to build snapshots and point-in-time joins -against. The project tests it with two package-free tests, one current row per key and no -overlapping versions; the `overlapping_history` defect breaks the second. +## Validate models in CI ```yaml -orders: - incremental: {new_per_day: 20, update_rate: 0.1, updated_at: updated_at, history: true} -``` - ---- - -## Generated dbt project structure - -The generated dbt project includes: - -``` -dbt_{project_name}/ -├── seeds/ -│ └── raw/ -│ ├── __seed_config.yml # seed descriptions + column-type overrides -│ ├── table1.csv -│ └── table2.csv -├── models/ -│ └── staging/ -│ ├── stg_table1.sql -│ ├── stg_table1.yml -│ ├── ut_stg_table1.yml # only with --unit-tests -│ └── ... -├── data-tests/ -│ └── unique_combination_stg_table1_col_a_col_b.sql # only for composite pk/unique keys -├── macros/ -│ ├── generate_schema_name.sql -│ └── model2data_hint_tests.sql # only when the model's hints write tests -├── dbt_project.yml -├── profiles.yml # DuckDB or Postgres config, depending on --adapter -└── {project_name}_profile.duckdb # DuckDB adapter only +name: model2data +on: + pull_request: + paths: ["**/*.model2data.yml"] +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: JB-Analytica/model2data@v1 ``` -- **Seeds**: CSV files with generated synthetic data, plus `__seed_config.yml` — each seed's - `description:` (the table's `description` in the model) and the column-type overrides that keep - all-digit text columns (barcodes, zero-padded postcodes, ...) from being loaded as integers. -- **Staging Models**: Basic dbt models that `ref()` their seed. Using `ref()` rather than - declaring the seeds as dbt `sources` is what gives each model a real DAG edge to the seed - behind it, so one `dbt build` orders seeds before models on a fresh database. -- **Tests**: A YAML per staging model with column tests (`not_null`, `unique`, `relationships`, - and `accepted_values` for enum-typed columns), plus the hint tests above. A column's `description` in the model becomes - its `description:` field. -- **Composite key tests**: Composite primary/unique keys (`keys`, or several `pk: true` columns) get - a singular SQL test under `data-tests/`, dbt's configured `test-paths`. -- **Profiles**: Pre-configured for DuckDB (file-based) or Postgres (via env vars), with schema handling. -- **Unit tests** (opt-in via `--unit-tests`): `models/staging/ut_stg_
.yml` fixtures built - from real generated rows, co-located with each staging model so dbt (which only parses unit - tests from `model-paths`) picks them up. +Issues show inline on the pull request's Files tab. The inputs, a pre-commit hook and GitLab CI +are in [Validate models in CI](docs/ci.md). --- -## Using model2data with an LLM +## Documentation -If you want to go from a plain-English description of a data model straight to a running, -demo-ready dbt project, [LLMS.md](LLMS.md) is written for an LLM/agent to read: it covers the -DBML feature set model2data understands (enums, notes, defaults, composite keys, both -relationship syntaxes, self-references), which it converts to a spec 0.2.0 model and the exact command sequence to run. Point an -LLM-backed coding assistant at it and describe your data model — it can author the DBML and run -model2data for you. +- [The model file](docs/model-file.md) — `.model2data.yml`, DBML as input, `validate`, `convert` +- [Generating a project](docs/generating.md) — row counts, `--as-of`, the determinism promise, `--table-seed`, `--locale` +- [The generated dbt project](docs/dbt-project.md) — `dbt build`, Postgres, unit tests, hint tests, structure +- [Days after the first](docs/days.md) and [history tables](docs/history.md) +- [When things happen](docs/time-shapes.md) and [how the data is spread](docs/distributions.md) +- [Defects on purpose](docs/defects.md) +- [Metrics with known values](docs/metrics.md) +- [Validate models in CI](docs/ci.md) +- [Agents and LLMs](docs/agents.md), and [LLMS.md](LLMS.md) +- The [model spec](model2data/spec/README.md) and the [metrics spec](model2data/spec/metrics/README.md) +- [Examples](examples/README.md) and the [changelog](CHANGELOG.md) --- @@ -689,6 +374,7 @@ If you're pinned to an older dbt-core, use model2data 0.5.x, which supported dow - **DuckDB Default**: Chosen for its zero-config, file-based nature, making it easy to get started without database setup. Postgres is supported via `--adapter postgres`; other adapters can be configured manually. - **dbt Integration**: Leverages dbt's transformation capabilities for a familiar workflow in analytics engineering. - **Synthetic Data**: Uses deterministic generation for reproducibility; not intended for production use or as a replacement for real data. +- **Metrics describe, never steer**: The generator never reads the metrics file, so adding or changing a metric changes no generated data, only the known values and tests computed from it. - **Non-goals**: This is not a data migration tool, ETL pipeline, or real-time data generator. It focuses on static, synthetic datasets for testing and prototyping. --- @@ -705,18 +391,17 @@ If you're pinned to an older dbt-core, use model2data 0.5.x, which supported dow ## Project status -As of `1.0.0`, model2data is stable and feature-complete for its intended use case: turning a -data model into realistic synthetic data and a runnable dbt project, reliably. It is actively -maintained — CI runs a real `dbt build` against both the oldest supported dbt-core and the newest -release on every push, so compatibility is proven rather than assumed. New capabilities are not -the focus; correctness, dbt-core compatibility and community contributions are. +model2data is stable and actively developed. Releases follow semantic versioning: new +capabilities arrive in minor releases (since 1.0: days after the first, time shapes, +distributions, defects, history tables, CI validation, the agent guide and, in 1.13.0, metrics +with known values and Apache Ossie export). CI runs a real `dbt build` against both the oldest supported dbt-core and the newest +release on every push, and checks the output of a spread of models byte for byte. The +[changelog](CHANGELOG.md) says what each release changed, including any change to what a seed +produces. -Ideas that came up during development but were deliberately left out of scope, in case anyone -wants to pick them up as a contribution: +Ideas still open, in case anyone wants to pick them up as a contribution: - Additional database adapters (e.g. Snowflake, BigQuery). -- A rule-based semantic layer scaffold (`semantic_models.yml`/basic metrics) derived from the - parsed schema shape. - Example mart-layer models on top of staging (the generated `dbt_project.yml` carries a ready-to-uncomment `marts` schema/materialization config for this). diff --git a/README_PYPI.md b/README_PYPI.md index d8d788b..9304bf2 100644 --- a/README_PYPI.md +++ b/README_PYPI.md @@ -1,22 +1,28 @@ # model2data [![PyPI](https://img.shields.io/pypi/v/model2data)](https://pypi.org/project/model2data/) +[![PyPI downloads](https://img.shields.io/pypi/dm/model2data)](https://pypi.org/project/model2data/) +[![Python versions](https://img.shields.io/pypi/pyversions/model2data)](https://pypi.org/project/model2data/) [![CI](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml/badge.svg)](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml) [![codecov](https://codecov.io/gh/JB-Analytica/model2data/branch/main/graph/badge.svg)](https://codecov.io/gh/JB-Analytica/model2data) [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/JB-Analytica/model2data/blob/main/LICENSE) +Built and maintained by [JB Analytica](https://www.jbanalytica.com/) — data platform architecture and analytics engineering. + **Turn a data model into a running analytics stack in one command.** Give `model2data` a data model — a `.model2data.yml` file, or a [DBML](https://dbml.dbdiagram.io/docs/) schema hand-written or exported from an existing database — and it generates realistic, -relationship-preserving synthetic data -*and* a complete, runnable dbt project around it: seeds, staging models, tests, and a -DuckDB or Postgres profile. No sample data to hunt down, no dbt boilerplate to hand-write, no -production data to risk exposing. +relationship-preserving synthetic data *and* a complete, runnable dbt project around it: seeds, +staging models, tests, and a DuckDB or Postgres profile. Add a metrics file and it also knows +what your revenue comes to, and writes a dbt test that proves it. No sample data to hunt down, +no dbt boilerplate to hand-write, no production data to risk exposing. + +![model2data generating a project with metrics and running it with dbt](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/demo.gif) ```bash pip install model2data -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 # examples/ is in the GitHub repository cd dbt_ecommerce && dbt build ``` @@ -25,373 +31,239 @@ cd dbt_ecommerce && dbt build > column will generate before you generate it, then export CSVs or a runnable dbt project. > Nothing to install, free to start. -That's a working analytics stack — real (synthetic) data, tested dbt models, queryable in -DuckDB — from a schema file, in seconds: - -![model2data generating a project and running it with dbt](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/demo.gif) - --- -## Why this exists - -Analytics engineers hit the same wall constantly: you need realistic data to build or test a -pipeline, but production data is off-limits (privacy, access, scale), and hand-rolling mock CSVs -is tedious and doesn't scale past two tables. `model2data` closes that gap — from a schema -definition to a seeded, tested dbt project you can actually run, with no database or production -access required. - -- **Privacy-safe.** Nothing but a schema definition goes in; nothing but synthetic data comes out. -- **Realistic, not random.** Column names are matched against ~35 common patterns — `email`, - `first_name`, `city`, `phone`, `company`, ... — so a column called `email` gets real-looking - emails, not `Lorem ipsum` text. -- **Relationship-preserving.** Foreign keys resolve to real parent rows; tables are generated in - dependency order. -- **Deterministic.** Pass `--seed` and the same schema always produces the same data — safe to - commit fixtures, safe to diff across CI runs. Add `--as-of` to pin the date the data is anchored - on, and the run reproduces on any later day rather than only on the day it first ran. -- **Re-rollable one table at a time.** `--table-seed orders=7` regenerates a single table and - leaves every other table byte-identical, so you can keep the four tables that look right. -- **A real dbt project, not just CSVs.** Seeds, staging models that `ref()` them, schema tests, - and a ready-to-use profile — the thing you'd otherwise spend an afternoon scaffolding by hand. - A single `dbt build` loads, transforms, and tests the whole thing. - -## Who is model2data for? - -- **Analytics engineers** — generate realistic datasets and a working dbt project without - waiting on production access. -- **Data engineers** — produce deterministic test data from an existing schema for pipeline and - migration testing. -- **Software & data teams** — prototype integrations and analytics workflows without exposing - production data. -- **Consultants & architects** — spin up realistic environments for demos, workshops, and - architecture validation in minutes, not hours. - -## How it works - -1. **Read.** Reads the model — its tables, columns, keys, references and generation hints — from - a `.model2data.yml` document, checking it against [the spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), or - from DBML, which it converts to the same model first. -2. **Generate.** Produces synthetic values per column — typed generation for known SQL types - (int, date, timestamp, ...), name-aware inference for everything else (`email`, `phone`, - `city`, ...), foreign keys resolved against already-generated parent rows. -3. **Scaffold.** Writes a complete dbt project around that data: CSV seeds, staging models that - `ref()` those seeds, `not_null`/`unique`/`relationships` tests, `accepted_values` tests for - enum-typed columns, singular SQL tests for composite primary/unique keys, table and column - `description:` fields from the model, and a profile for DuckDB (zero-config, file-based) or - Postgres. +## What you get + +**Values that look real.** Column names are matched against about 40 common patterns — `email`, +`first_name`, `city`, `phone`, `company`, ... — so a column called `email` gets real-looking +emails, not `Lorem ipsum`. Type a column with any Faker provider (`billing_country state`, +`sku ean13`) to pick its generator outright, and `--locale nl_BE` makes every person and address +come from one country. → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md) + +**Foreign keys that resolve.** Tables are generated in dependency order and every foreign key +points at a parent row that exists — onto a primary key or a unique one. Composite primary and +unique keys hold, and a column that repeats a parent's value through a foreign key +(`orders.customer_email`) holds the email of the order's own customer. +→ [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md) + +**A dbt project that runs.** Seeds, staging models that `ref()` them, `not_null`, `unique`, +`relationships` and `accepted_values` tests, data tests written from the model's own hints, and +a DuckDB (zero-config) or Postgres profile. One `dbt build` loads, builds and tests it, and +`--unit-tests` adds dbt unit test fixtures. → [The generated dbt project](https://github.com/JB-Analytica/model2data/blob/main/docs/dbt-project.md) + +**The same files, byte for byte.** The same model, `--seed`, `--as-of` and options give the same +files on any machine, any day, with the same model2data version — safe to commit as fixtures, +safe to diff in CI. `--table-seed orders=7` re-rolls one table and leaves every other one +byte-identical. → [The determinism promise](https://github.com/JB-Analytica/model2data/blob/main/docs/generating.md#the-determinism-promise) + +**Days after the first.** A table marked `incremental` moves on day by day: `--days 7` or +`--next` adds new rows and updates existing ones, an enum with `transitions` moves an order from +`pending` to `shipped`, and each day is written as a batch or a changelog to load incrementally. +→ [Days after the first](https://github.com/JB-Analytica/model2data/blob/main/docs/days.md) + +**History tables.** `history: true` writes `
_history`, every version of every row with +`valid_from`, `valid_to` and `is_current`: an SCD type 2 source to build snapshots against. +→ [A source that keeps its history](https://github.com/JB-Analytica/model2data/blob/main/docs/history.md) + +**When things happen.** `--business-hours`, `--growth` and `--seasonality` shape every timestamp +(weekdays and working hours, a trend, a Q4 peak), or one column at a time. `updated_at` never +lands before `created_at`, `after` orders any two columns, and `when` gives `shipped_at` a value +on shipped and delivered orders only. → [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md) + +**How the data is spread.** `--skew` lets a few customers place most of the orders; `weights`, +`true_rate`, `null_rate` and `distinct` shape a column; `distribution` draws a number from a +normal, lognormal or exponential curve. → [How the data is spread](https://github.com/JB-Analytica/model2data/blob/main/docs/distributions.md) + +**Defects on purpose.** `--defects training` breaks each standard dbt test once, `messy` puts a +small share of every defect on every table, and a table can list its own: duplicate keys, +orphans, nulls, invalid values, messy text, late-arriving rows. Every run with defects writes +`EXPECTED_FAILURES.md`, so a test that does not fire shows. → [Defects on purpose](https://github.com/JB-Analytica/model2data/blob/main/docs/defects.md) + +**Metrics with known values** (new in 1.13.0). A metrics file beside the model defines revenue, +orders, average order value. The run writes each metric's value over the generated data to +`metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the +warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`. +→ [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md) + +**Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub +annotations; `uses: JB-Analytica/model2data@v1` fails a pull request that breaks one. A +pre-commit hook and a GitLab CI snippet do the same. → [Validate models in CI](https://github.com/JB-Analytica/model2data/blob/main/docs/ci.md) + +**Use it from an agent.** `uvx model2data@latest guide` prints instructions written for a coding +agent, shipped with the installed version; [LLMS.md](https://github.com/JB-Analytica/model2data/blob/main/LLMS.md) takes an LLM from a plain-English +description to a running project. → [Agents and LLMs](https://github.com/JB-Analytica/model2data/blob/main/docs/agents.md) + +**Or in the browser.** [model2data studio](https://studio.jbanalytica.com) +is the same engine with a DBML editor and a live diagram. --- -## The model file +## Before and after -A model is one YAML document, `.model2data.yml` — or the same document as JSON. Its format -is [spec 0.4.0](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), with a JSON Schema your editor can autocomplete and -check against: +In: a model (here the `orders` table of +[`examples/ecommerce.model2data.yml`](https://github.com/JB-Analytica/model2data/blob/main/examples/ecommerce.model2data.yml)) and, optionally, the +metrics that matter ([`examples/ecommerce.metrics.yml`](https://github.com/JB-Analytica/model2data/blob/main/examples/ecommerce.metrics.yml)): ```yaml -# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/0.2.0/model.schema.json -model2data: 0.2.0 -name: coffee_webshop - -enums: - order_status: [pending, paid, shipped, delivered, cancelled] - -tables: - customers: - columns: - id: {type: bigint, pk: true} - email: {type: email, unique: true, not_null: true} - orders: columns: - id: {type: bigint, pk: true} - customer_id: - type: bigint - not_null: true - references: customers.id - generate: {skew: 0.8} + id: {type: bigint, pk: true, not_null: true} + customer_id: {type: bigint, not_null: true, references: customers.id} + order_date: {type: timestamp, not_null: true} total_amount: type: numeric - generate: - min: 10 - max: 5000 - distribution: {kind: lognormal, median: 120, spread: 0.7} - status: - type: order_status - generate: - weights: {delivered: 20, cancelled: 1} - -run: - seed: 1 - as_of: 2026-01-01 + not_null: true + generate: {min: 10, max: 5000} + status: order_status ``` -`generate` holds a column's hints; `run` saves the generation settings with the model, and every -CLI option overrides the one it names. `model2data validate FILE` checks a model and prints every -issue with its path in the document. - -**Validate models in CI.** `model2data validate a.model2data.yml b.model2data.yml` (or -`--glob "**/*.model2data.yml"`, `--format github` for inline pull request annotations) exits 1 -when any model has an error. A GitHub Action (`uses: JB-Analytica/model2data@v1`), a pre-commit -hook (`model2data-validate`) and a GitLab CI snippet are in the -[README](https://github.com/JB-Analytica/model2data#validate-models-in-ci). - -**DBML is supported input.** `--file` also takes a `.dbml` file, converted to the same model -before generating; a JSON note on a column (`[note: '{"min": 1}']`) becomes its `generate`, any -other note its description. `model2data convert schema.dbml -o schema.model2data.yml` writes the -model a DBML file converts to. - ---- - -## model2data studio — the same engine, in the browser - -[**model2data studio**](https://studio.jbanalytica.com) puts everything on this page behind a -web UI. It is built by JB Analytica on top of this library, it's the fastest way to try -model2data, and it's the better fit while a schema is still being designed: - -- **Type DBML, see the diagram.** Syntax highlighting, autocomplete and live error checking; the - entity diagram redraws as you type. Click a column to trace what actually joins to it. -- **See what you'll get before you generate.** Every column shows an example of the value it will - produce, and columns nothing recognises are marked — so placeholder data is visible rather than - silent. -- **Generate and export.** Per-table row counts, then CSVs or a complete dbt project: the same - seeds, staging models, tests and DuckDB profile this CLI produces, reproducing the exact rows - you previewed. -- **Share the model.** A share link that also embeds as a chrome-free diagram in a Notion, - Confluence or wiki page. - -Free to start, nothing to install: [studio.jbanalytica.com](https://studio.jbanalytica.com). -The CLI stays the right tool for scripting, CI and fixtures you commit; the studio is where a -model gets designed and shown. - ---- - -## Installation - -```bash -pip install model2data +```yaml +metrics: + revenue: + label: Revenue + measure: orders.total_amount + agg: sum + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + format: currency + orders: + count: orders + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + average_order_value: + ratio: {numerator: revenue, denominator: orders} ``` ---- - -## Quick start - -We bundle several example schemas in `examples/` — this walkthrough uses the e-commerce one -(`examples/ecommerce.model2data.yml`: customers, products, orders, order items, and reviews). - -Generate a project with synthetic data: - ```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 +model2data --file examples/ecommerce.model2data.yml --metrics examples/ecommerce.metrics.yml \ + --rows 200 --seed 42 --as-of 2026-01-31 ``` -This creates a `dbt_ecommerce/` folder with your data and dbt setup. +Out: a dbt project. -`--seed` reproduces a run's numbers, but dates and timestamps are generated relative to the -current date, so the same seed drifts once the day turns over. `--as-of` pins the date they're -anchored on, and the whole dataset reproduces on any later day — which is what makes a generated -fixture safe to commit. If one table comes out wrong and the rest looks right, `--table-seed` -re-rolls just that table, leaving every other table's seed CSV byte-identical. `--locale` picks -the country every generated person and address comes from: - -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 \ - --as-of 2026-01-31 --table-seed orders=7 --locale nl_BE ``` - -Run dbt to load, transform, and test the data: - -```bash -cd dbt_ecommerce -dbt build +dbt_ecommerce/ +├── seeds/raw/ customers.csv, orders.csv, order_items.csv, products.csv, +│ product_reviews.csv, __seed_config.yml +├── models/staging/ stg_
.sql + stg_
.yml (with tests), for each table +├── data-tests/metrics/ metric_revenue.sql, metric_orders.sql, +│ metric_average_order_value.sql, metric_units_sold.sql +├── macros/ generate_schema_name.sql, model2data_hint_tests.sql +├── osi/ecommerce.yml the model and its metrics as Apache Ossie 0.1.1 +├── metric_values.json each metric's known value +├── ecommerce.model2data.yml the model, and +├── ecommerce.metrics.yml its metrics, copied as given +├── dbt_project.yml +└── profiles.yml DuckDB ``` -Staging models `ref()` their seeds, so a single `dbt build` loads the seeds, builds the models, -and runs every generated test in one dependency-ordered pass — no separate `dbt seed`/`dbt run` -needed, even on a brand-new database. (The individual `dbt deps`, `dbt seed`, and `dbt run` -commands still work if you'd rather drive the steps yourself; the generated project declares no -packages, so `dbt deps` is a no-op.) - -Your analytics-ready dataset is now in DuckDB! - -To target Postgres instead, install the extra and pass `--adapter postgres`: +`dbt build` loads the seeds, builds the staging models and runs every test, the four metric +tests included: -```bash -pip install "model2data[postgres]" -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --adapter postgres ``` +Finished running 5 seeds, 49 data tests, 5 view models in 0 hours 0 minutes and 0.71 seconds (0.71s). -Connection details are read from environment variables (`MODEL2DATA_PG_HOST`, `MODEL2DATA_PG_PORT`, `MODEL2DATA_PG_USER`, `MODEL2DATA_PG_PASSWORD`, `MODEL2DATA_PG_DATABASE`), defaulting to `localhost:5432` with a `postgres`/`postgres` user for local development. - -After generation, the CLI prints a short summary — tables and rows generated, relationships found in the model, and any columns that fell back to generic placeholder text because neither their type nor name could be matched. +Completed successfully -Pass `--unit-tests` to also generate deterministic dbt unit test fixtures (`models/staging/ut_stg_
.yml`) from the actually-generated seed rows: +Done. PASS=59 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=59 +``` -```bash -model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --unit-tests +And `metric_values.json` says what a dashboard on this data must show: + +```json +{ + "model2data-metrics": "0.1.0", + "model": "ecommerce", + "run": { + "seed": 42, + "as_of": "2026-01-31" + }, + "rounding": "counts, and sums, minimums and maximums of integer columns, are exact; every other value is rounded to 6 decimal places, half to even", + "metrics": { + "revenue": { + "kind": "simple", + "label": "Revenue", + "value": 220291.8, + "time": "orders.order_date", + "by_month": { + "2025-01": 3967.88, + "2025-02": 10676.66, + ... + "2026-01": 31879.13 + } + }, + "orders": { "kind": "count", "label": "Orders", "value": 92, ... }, + "average_order_value": { "kind": "ratio", "label": "Average order value", "value": 2394.476087, ... }, + "units_sold": { "kind": "simple", "label": "Units sold", "value": 734, ... } + } +} ``` -This targets dbt-core's native unit testing feature, which works out of the box with the base -install — see [dbt-core versions](#dbt-core-versions) below. - -Your model's hints also write **data tests** for the dbt project, so the model that generates the -fixtures guards the real pipeline too. Every hint that states a constraint becomes a generic test -on the staging model, shipped as a self-contained macro in `macros/model2data_hint_tests.sql` (no -dbt package, so `dbt build` still works offline): - -| In the model | dbt test | Tolerance | -|---|---|---| -| `generate: {min, max}` on a numeric column | `model2data_between` | none | -| `generate: {after: other}` | `model2data_not_before` (where both are not null) | none | -| `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 | -| `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none | -| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none | -| `grain` on a table | `model2data_unique_combination` | none | -| an enum-typed column | `accepted_values` (always written) | none | - -`true_rate`, `weights`, `skew`, `distribution` and the temporal shape hints (`business_hours`, -`growth`, `seasonality`) describe a statistical shape rather than a constraint a row can break, so -they write no test. `--hint-tests {error,warn,off}` sets these tests' severity (default `warn`: -they describe intent and should not break a first `dbt build` on real data); the structural tests -(`not_null`, `unique`, `relationships`) are unaffected. A model with no such hints generates the -same YAML as before. To call the mapping from Python, `model2data.dbt.hint_tests.hint_tests_for(model)` -returns the tests as data. +Run it again tomorrow, or on another machine, with the same model2data version, and every one +of those files is the same. --- -### Generate the next days - -A table with `incremental` in the model moves on day by day: each day adds `new_per_day` rows -(their dates and timestamps falling on that day) and updates `update_rate` of the rows it already -holds, and an enum column with `transitions` moves from a state to one of the states allowed to -follow it. See `examples/ecommerce_daily.model2data.yml` and "Days after the first" in the -[spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md). - -```bash -model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --days 7 -model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --next -``` - -`--days N` generates day 0, the run you would get without it, and N days after it (`--next` is -`--days 1`). The dbt seeds hold the state after the last day. `--days-format` picks what else is -written beside the project, outside `seeds/` so dbt does not load it: `batches` (the default), -`days/
/day_000.csv` whole and then one file per day with the rows inserted and updated that -day; `changelog`, `changelog/
.csv` with a `_day` and an `_op` (`insert` or `update`) column -on every row; or `final`, nothing more. A day depends only on the seed, `--as-of`, the day and the -model, so day *n* never changes when you generate more days or add an unrelated table. From Python: - -```python -from model2data.generate.days import generate_days -from model2data.model import load - -days = generate_days(load("examples/ecommerce_daily.model2data.yml"), 7, seed=42) -days[3].tables["orders"].inserted # rows day 3 added -days[3].tables["orders"].updated # rows day 3 changed, with their new values -days[3].tables["orders"].state # the table after day 3 -``` +## Who it's for -### Break the data on purpose +Analytics engineers hit the same wall constantly: you need realistic data to build or test a +pipeline, but production data is off-limits (privacy, access, scale), and hand-rolling mock CSVs +is tedious and doesn't scale past two tables. model2data closes that gap. Nothing but a schema +definition goes in; nothing but synthetic data comes out. + +- **A consultant who needs a client demo before source access.** Write the client's model from a + kick-off call, generate a year of their data, and show a working dbt project and dashboard in + the first week, without touching production. +- **An analytics engineer who wants fixtures in CI.** Commit the model, generate with `--seed` + and `--as-of`, and every CI run builds against the same bytes. `--days` gives an incremental + model the next day's batch to process. +- **A trainer who needs data that fails on purpose.** `--defects training` breaks each standard + dbt test once and says which, so a class sees `unique`, `not_null`, `relationships` and + `accepted_values` fail for the right reason. +- **A team defining metrics once.** Write revenue in one metrics file, check every number the + semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL, + and hand the definitions on as Apache Ossie. -A dbt test you have never seen fail is a test you hope works. `--defects` puts deliberate, -counted defects in the generated data, so `dbt build` fails exactly the tests it should — for -teaching dbt, or for proving a project's tests fire: +## How it works -```bash -model2data --file examples/ecommerce.model2data.yml --seed 42 --defects training -cd dbt_ecommerce && dbt build # fails unique, not_null, relationships, accepted_values once each ``` - -`training` breaks each kind of standard dbt test once; `messy` puts a small share of every defect -on every table; `clean` (the default) is the data as it always was, byte for byte; `none` also -ignores the defects a model's tables list, for the clean data of a training model. A table can -also list its own defects in the model (spec 0.3.0): - -```yaml -orders: - defects: - - {type: duplicate_keys, count: 3} # fails unique - - {type: orphan_foreign_keys, column: customer_id, share: 0.02} # fails relationships - - {type: nulls, column: order_date, count: 5} # fails not_null +model (.model2data.yml or DBML) + metrics file (optional) + -> read and validate -> generate -> scaffold the dbt project + compute each metric's known value + -> seeds, staging models and tests, metric tests, metric_values.json, osi/, profiles.yml + -> dbt build ``` -The types are `duplicate_keys`, `orphan_foreign_keys`, `nulls`, `invalid_values` (fails -`accepted_values`), `messy_text` (whitespace and casing to clean in staging), and, with `--days`, -`late_arriving` (rows an incremental model filtering on `updated_at > max(updated_at)` skips) and -`late_updates` (versions a `strategy: timestamp` snapshot misses) and `overlapping_history` (see -below). Every run with defects writes -`defects_report.json` (each defect, the rows it broke, the tests it breaks) and -`EXPECTED_FAILURES.md` into the project, so a test that does not fire is visible. Defects are applied -after the clean data, from their own random stream: the same seed gives the same bytes, and a -defect on one table moves nothing in another. From Python, `model2data.defects.planned_defects` -says what a run will break, `model2data.output.finish_run` applies it (and builds the histories) -after `generate_days`, and `write_defects_report` / `write_expected_failures` write the two files. - -### A source that keeps its history - -`history: true` in a table's `incremental` (spec 0.3.0) also writes `
_history`: every -version of every row over the generated days, with `valid_from`, `valid_to` (null for the current -version) and `is_current` — an SCD type 2 source to build snapshots and point-in-time joins -against. The project tests it with two package-free tests, one current row per key and no -overlapping versions; the `overlapping_history` defect breaks the second. - -```yaml -orders: - incremental: {new_per_day: 20, update_rate: 0.1, updated_at: updated_at, history: true} -``` +1. **Read.** Reads the model — its tables, columns, keys, references and generation hints — from + a `.model2data.yml` document, checking it against [the spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), or + from DBML, which it converts to the same model first. A metrics file is checked against + [its spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/metrics/README.md) and against the model. +2. **Generate.** Produces synthetic values per column — typed generation for known SQL types + (int, date, timestamp, ...), then a Faker provider named as the type (`sku ean13`), then + name-aware inference for everything else (`email`, `phone`, `city`, ...), foreign keys + resolved against already-generated parent rows. The metrics file is never read here: the data + is the same with or without it. +3. **Scaffold.** Writes a complete dbt project around that data: CSV seeds, staging models that + `ref()` those seeds, `not_null`/`unique`/`relationships` tests, `accepted_values` tests for + enum-typed columns, singular SQL tests for composite primary/unique keys, tests from the + model's hints, table and column `description:` fields from the model, and a profile for + DuckDB (zero-config, file-based) or Postgres. +4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes + its known value, a dbt test that checks it, and the Apache Ossie file. -## Generated dbt project structure +--- -The generated dbt project includes: +## Installation -``` -dbt_{project_name}/ -├── seeds/ -│ └── raw/ -│ ├── __seed_config.yml # seed descriptions + column-type overrides -│ ├── table1.csv -│ └── table2.csv -├── models/ -│ └── staging/ -│ ├── stg_table1.sql -│ ├── stg_table1.yml -│ ├── ut_stg_table1.yml # only with --unit-tests -│ └── ... -├── data-tests/ -│ └── unique_combination_stg_table1_col_a_col_b.sql # only for composite pk/unique keys -├── macros/ -│ ├── generate_schema_name.sql -│ └── model2data_hint_tests.sql # only when the model's hints write tests -├── dbt_project.yml -├── profiles.yml # DuckDB or Postgres config, depending on --adapter -└── {project_name}_profile.duckdb # DuckDB adapter only +```bash +pip install model2data # DuckDB included +pip install "model2data[postgres]" # to target Postgres with --adapter postgres +uvx model2data --help # or run it without installing ``` -- **Seeds**: CSV files with generated synthetic data, plus `__seed_config.yml` — each seed's - `description:` (the table's `description` in the model) and the column-type overrides that keep - all-digit text columns (barcodes, zero-padded postcodes, ...) from being loaded as integers. -- **Staging Models**: Basic dbt models that `ref()` their seed. Using `ref()` rather than - declaring the seeds as dbt `sources` is what gives each model a real DAG edge to the seed - behind it, so one `dbt build` orders seeds before models on a fresh database. -- **Tests**: A YAML per staging model with column tests (`not_null`, `unique`, `relationships`, - and `accepted_values` for enum-typed columns), plus the hint tests above. A column's `description` in the model becomes - its `description:` field. -- **Composite key tests**: Composite primary/unique keys (`keys`, or several `pk: true` columns) get - a singular SQL test under `data-tests/`, dbt's configured `test-paths`. -- **Profiles**: Pre-configured for DuckDB (file-based) or Postgres (via env vars), with schema handling. -- **Unit tests** (opt-in via `--unit-tests`): `models/staging/ut_stg_
.yml` fixtures built - from real generated rows, co-located with each staging model so dbt (which only parses unit - tests from `model-paths`) picks them up. - ---- - -## Using model2data with an LLM - -If you want to go from a plain-English description of a data model straight to a running, -demo-ready dbt project, [LLMS.md](https://github.com/JB-Analytica/model2data/blob/main/LLMS.md) is written for an LLM/agent to read: it covers the -DBML feature set model2data understands (enums, notes, defaults, composite keys, both -relationship syntaxes, self-references), which it converts to a spec 0.2.0 model and the exact command sequence to run. Point an -LLM-backed coding assistant at it and describe your data model — it can author the DBML and run -model2data for you. +Python 3.10 to 3.14. The examples used on this page are in [`examples/`](https://github.com/JB-Analytica/model2data/tree/main/examples/). --- @@ -407,74 +279,23 @@ If you're pinned to an older dbt-core, use model2data 0.5.x, which supported dow --- -## Design decisions / non-goals - -- **DuckDB Default**: Chosen for its zero-config, file-based nature, making it easy to get started without database setup. Postgres is supported via `--adapter postgres`; other adapters can be configured manually. -- **dbt Integration**: Leverages dbt's transformation capabilities for a familiar workflow in analytics engineering. -- **Synthetic Data**: Uses deterministic generation for reproducibility; not intended for production use or as a replacement for real data. -- **Non-goals**: This is not a data migration tool, ETL pipeline, or real-time data generator. It focuses on static, synthetic datasets for testing and prototyping. - ---- - -## Limitations - -- Synthetic data generation is heuristic-based (typed generation, name-aware inference, enum/default awareness) and may not perfectly mimic real-world distributions or edge cases. -- DuckDB and Postgres are supported today; other databases require manual profile adjustments. -- No support for incremental models or advanced dbt features in generated projects. -- Composite foreign keys (across a bridge/join table) are generated as independent single-column FKs — each column's values are individually valid, but the *combination* isn't guaranteed to match a real parent composite key unless that key is separately enforced as a key of the child (`keys`). -- A model that doesn't conform to the spec — or DBML that can't be read, or converts to such a model — is refused with every issue and where it is, rather than generated from in part. A reference onto a column that is not a key is allowed with a warning: its values are drawn from the ones the parent column holds. - ---- - -## Project status - -As of `1.0.0`, model2data is considered **feature-complete for its intended use case**: turning a -data model into realistic synthetic data and a runnable dbt project, reliably. There's no active -roadmap of new capabilities planned — the focus from here is maintenance: bug fixes, keeping pace -with new dbt-core releases, and reviewing community contributions. - -Ideas that came up during development but were deliberately left out of scope, in case anyone -wants to pick them up as a contribution: - -- Additional database adapters (e.g. Snowflake, BigQuery). -- A rule-based semantic layer scaffold (`semantic_models.yml`/basic metrics) derived from the - parsed schema shape. -- Example mart-layer models on top of staging (the generated `dbt_project.yml` carries a - ready-to-uncomment `marts` schema/materialization config for this). - -See [CONTRIBUTING.md](https://github.com/JB-Analytica/model2data/blob/main/CONTRIBUTING.md) if you'd like to work on any of these. - ---- - -## Contributing - -We welcome contributions! - -- Open issues for bugs or feature requests. -- Submit PRs to add new example models, custom data generators, or improvements. -- Ensure all new features include tests if possible. - -See [CONTRIBUTING.md](https://github.com/JB-Analytica/model2data/blob/main/CONTRIBUTING.md) for detailed guidelines, and [DEVELOPMENT.md](https://github.com/JB-Analytica/model2data/blob/main/DEVELOPMENT.md) for the local dev setup and release process. - -## Code of Conduct - -Please read our [Code of Conduct](https://github.com/JB-Analytica/model2data/blob/main/CODE_OF_CONDUCT.md) to understand our community standards. - ---- - ## License -MIT License. See LICENSE for details. +MIT License. See [LICENSE](https://github.com/JB-Analytica/model2data/blob/main/LICENSE). Built and maintained by [JB Analytica](https://www.jbanalytica.com). +Try [model2data studio](https://studio.jbanalytica.com) — model2data in the browser, nothing to install. --- -

- - JB Analytica - -
- Built and maintained by JB Analytica — - Data & Analytics Engineering · Data Platform Architecture · Modern BI. -
- Try model2data studio — model2data in the browser, nothing to install. -

+## Documentation + +- [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md) — `.model2data.yml`, DBML as input, `validate`, `convert` +- [Generating a project](https://github.com/JB-Analytica/model2data/blob/main/docs/generating.md) — row counts, `--as-of`, the determinism promise, `--table-seed`, `--locale` +- [The generated dbt project](https://github.com/JB-Analytica/model2data/blob/main/docs/dbt-project.md) — `dbt build`, Postgres, unit tests, hint tests, structure +- [Days after the first](https://github.com/JB-Analytica/model2data/blob/main/docs/days.md) and [history tables](https://github.com/JB-Analytica/model2data/blob/main/docs/history.md) +- [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md) and [how the data is spread](https://github.com/JB-Analytica/model2data/blob/main/docs/distributions.md) +- [Defects on purpose](https://github.com/JB-Analytica/model2data/blob/main/docs/defects.md) +- [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md) +- [Validate models in CI](https://github.com/JB-Analytica/model2data/blob/main/docs/ci.md) +- [Agents and LLMs](https://github.com/JB-Analytica/model2data/blob/main/docs/agents.md), and [LLMS.md](https://github.com/JB-Analytica/model2data/blob/main/LLMS.md) +- The [model spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md) and the [metrics spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/metrics/README.md) +- [Examples](https://github.com/JB-Analytica/model2data/blob/main/examples/README.md) and the [changelog](https://github.com/JB-Analytica/model2data/blob/main/CHANGELOG.md) diff --git a/action.yml b/action.yml index 086716a..51ad88f 100644 --- a/action.yml +++ b/action.yml @@ -12,10 +12,10 @@ inputs: default: "**/*.model2data.yml" version: description: >- - The model2data engine version to install: a bare version such as `1.12.0`, or a full + The model2data engine version to install: a bare version such as `1.13.0`, or a full requirement specifier such as `>=1.9,<2`. required: false - default: "1.12.0" + default: "1.13.0" python-version: description: The Python version to run model2data with. required: false diff --git a/assets/demo.gif b/assets/demo.gif index b649be8..b9031c0 100644 Binary files a/assets/demo.gif and b/assets/demo.gif differ diff --git a/assets/demo.tape b/assets/demo.tape index 540602c..90e0899 100644 --- a/assets/demo.tape +++ b/assets/demo.tape @@ -1,25 +1,21 @@ # Regenerate the README demo GIF: -# brew install vhs duckdb +# brew install vhs (jq ships with macOS 15; elsewhere install it too) # uv sync --extra dev # cd /tmp && mkdir m2d-gif-demo && cd m2d-gif-demo && cp -r /examples . # vhs /assets/demo.tape # (recorded from a scratch directory so the printed paths in the output don't -# leak a real machine's home directory) +# leak a real machine's home directory; replace /path/to/model2data below with +# the checkout whose .venv should run) # -# Then shrink it before committing -- vhs writes a ~2.5MB GIF, and a reduced -# palette at a lower frame rate cuts that by more than half with no visible -# loss on terminal text: -# ffmpeg -i demo.gif -vf "fps=12,palettegen=max_colors=32:stats_mode=diff" -y pal.png -# ffmpeg -i demo.gif -i pal.png \ -# -lavfi "fps=12[x];[x][1:v]paletteuse=dither=none:diff_mode=rectangle" \ -# -y /assets/demo.gif +# Commit the GIF as vhs writes it (about 250 KB at this size). A reduced +# palette makes it smaller but turns dbt's green PASS and jq's colours grey. Output demo.gif Set Shell "zsh" Set FontSize 16 -Set Width 1200 -Set Height 700 +Set Width 1520 +Set Height 540 Set Padding 24 Set Framerate 20 Set Theme {"name": "jba", "black": "#182333", "red": "#e5989b", "green": "#7cc576", "yellow": "#FA9306", "blue": "#2196F0", "purple": "#A8C9EE", "cyan": "#2196F0", "white": "#F6F8FB", "brightBlack": "#3b4252", "brightRed": "#e5989b", "brightGreen": "#7cc576", "brightYellow": "#FA9306", "brightBlue": "#2196F0", "brightPurple": "#A8C9EE", "brightCyan": "#2196F0", "brightWhite": "#FFFFFF", "background": "#0e1117", "foreground": "#F6F8FB", "cursor": "#FA9306"} @@ -31,37 +27,28 @@ Enter Sleep 500ms Show -# 1. Schema in -- synthetic data and a whole dbt project out. -Type "model2data --file examples/ecommerce.dbml --rows 200 --seed 42" +# 1. A model and its metrics in -- synthetic data, a dbt project and each +# metric's known value out. +Type "model2data -f examples/ecommerce.model2data.yml --metrics examples/ecommerce.metrics.yml --rows 200 --seed 42 --as-of 2026-01-31" Sleep 400ms Enter -Sleep 4000ms +Sleep 3500ms Type "clear" Enter Sleep 300ms -# 2. One command: loads the seeds, builds the models, runs every test. -Type "cd dbt_ecommerce && dbt build" +# 2. One command loads the seeds, builds the models and runs every test. The +# full log is 120 lines; the grep keeps the metric tests (each one computes +# its metric on the warehouse and must match the known value) and dbt's +# summary line, so both stay on screen long enough to read. +Type `cd dbt_ecommerce && dbt build --use-colors | grep -E "PASS metric_|Done\."` Sleep 300ms Enter -# dbt itself takes ~4s of this, so the sleep has to be generous enough that -# the finished PASS=.../ERROR=0 summary actually sits on screen long enough -# to read -- it is the whole point of this segment. -Sleep 9000ms +Sleep 7500ms -Type "clear" -Enter -Sleep 300ms - -# 3. The payoff. Real names and emails, not lorem ipsum -- and a join that -# actually resolves, because the foreign keys point at real parent rows. -Type `duckdb ecommerce_profile.duckdb -c "from staging.stg_customers select first_name, last_name, email, country limit 5"` -Sleep 300ms -Enter -Sleep 3000ms - -Type `duckdb ecommerce_profile.duckdb -c "from staging.stg_customers c join staging.stg_orders o on o.customer_id = c.id select c.country, count(*) orders, round(sum(o.total_amount)) revenue group by 1 order by revenue desc limit 5"` +# 3. The known values themselves: what a dashboard on this data must show. +Type `jq '.metrics | map_values(.value)' metric_values.json` Sleep 300ms Enter -Sleep 4000ms +Sleep 4500ms diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e72f6a5 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# model2data documentation + +The [README](../README.md) shows what model2data does; these pages say how each part works. + +- [The model file](model-file.md) — `.model2data.yml`, DBML as input, `validate` and `convert`. +- [Generating a project](generating.md) — row counts, `--as-of`, the determinism promise, + `--table-seed`, `--locale`. +- [The generated dbt project](dbt-project.md) — `dbt build`, Postgres, unit tests, the data + tests written from the model's hints, the folder structure. +- [Days after the first](days.md) — `incremental`, `--days`, `--next`, `transitions`, + `--days-format`. +- [A source that keeps its history](history.md) — `
_history`, an SCD type 2 source. +- [When things happen](time-shapes.md) — business hours, growth, seasonality, `after`, `when`. +- [How the data is spread](distributions.md) — skew, weights, `true_rate`, `null_rate`, + `distinct`, number distributions. +- [Defects on purpose](defects.md) — `--defects messy|training`, defects per table, the + expected-failures report. +- [Metrics with known values](metrics.md) — the metrics file, `metric_values.json`, a dbt test + per metric, the Apache Ossie export. +- [Validate models in CI](ci.md) — the GitHub Action, pre-commit, GitLab CI. +- [Using model2data from a coding agent or an LLM](agents.md) — `model2data guide`, LLMS.md. + +The formats themselves are specified in the package: the [model spec](../model2data/spec/README.md) +and the [metrics spec](../model2data/spec/metrics/README.md), each with a JSON Schema. diff --git a/docs/agents.md b/docs/agents.md new file mode 100644 index 0000000..1de46e5 --- /dev/null +++ b/docs/agents.md @@ -0,0 +1,26 @@ +# Using model2data from a coding agent or an LLM + +## `model2data guide` + +Driving model2data from a coding agent? Have it start here; the instructions ship with the +installed version and pick a topic from the current directory: + +```bash +uvx model2data@latest guide +``` + +`model2data guide [setup|triage|tune]` prints one page of Markdown written for a coding agent: +`setup` (install, a starter model, the first run), `triage` (what `validate`, `generate` and +`dbt build` print and what to do about each, in order) and `tune` (every option, model key and +exit code). Without a topic it picks `triage` when the current directory holds a +`*.model2data.yml` or `*.dbml` file, else `setup`, and says so on its first line. It reads only +file names in the current directory and writes nothing. + +## LLMS.md + +If you want to go from a plain-English description of a data model straight to a running, +demo-ready dbt project, [LLMS.md](../LLMS.md) is written for an LLM/agent to read: it covers the +DBML feature set model2data understands (enums, notes, defaults, composite keys, both +relationship syntaxes, self-references), which it converts to a spec 0.2.0 model and the exact command sequence to run. Point an +LLM-backed coding assistant at it and describe your data model — it can author the DBML and run +model2data for you. diff --git a/docs/ci.md b/docs/ci.md new file mode 100644 index 0000000..f64bf80 --- /dev/null +++ b/docs/ci.md @@ -0,0 +1,47 @@ +# Validate models in CI + +Teams that keep `*.model2data.yml` models in git can fail a pull request that breaks one. +`model2data validate` takes several files and `--glob` patterns, prefixes each issue with its +file, exits 1 if any model has an error, and with `--format github` prints annotations that show +inline on the pull request's Files tab. `--require-files` fails when nothing matched. + +**GitHub Actions** + +```yaml +name: model2data +on: + pull_request: + paths: ["**/*.model2data.yml"] +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: JB-Analytica/model2data@v1 +``` + +Inputs: `files` (glob, default `**/*.model2data.yml`), `version` (default: the release the tag +points at, e.g. `1.13.0`; a specifier such as `>=1.10,<2` also works), `python-version` (3.12). + +**pre-commit** + +```yaml +repos: + - repo: https://github.com/JB-Analytica/model2data + rev: v1.13.0 + hooks: + - id: model2data-validate +``` + +**GitLab CI** + +```yaml +model2data: + image: python:3.12-slim + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + changes: ["**/*.model2data.yml"] + script: + - pip install model2data==1.13.0 + - model2data validate --glob "**/*.model2data.yml" +``` diff --git a/docs/days.md b/docs/days.md new file mode 100644 index 0000000..2a715f6 --- /dev/null +++ b/docs/days.md @@ -0,0 +1,33 @@ +# Days after the first + +A table with `incremental` in the model moves on day by day: each day adds `new_per_day` rows +(their dates and timestamps falling on that day) and updates `update_rate` of the rows it already +holds, and an enum column with `transitions` moves from a state to one of the states allowed to +follow it. See `examples/ecommerce_daily.model2data.yml` and "Days after the first" in the +[spec](../model2data/spec/README.md). + +```bash +model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --days 7 +model2data --file examples/ecommerce_daily.model2data.yml --seed 42 --as-of 2026-01-31 --next +``` + +`--days N` generates day 0, the run you would get without it, and N days after it (`--next` is +`--days 1`). The dbt seeds hold the state after the last day. `--days-format` picks what else is +written beside the project, outside `seeds/` so dbt does not load it: `batches` (the default), +`days/
/day_000.csv` whole and then one file per day with the rows inserted and updated that +day; `changelog`, `changelog/
.csv` with a `_day` and an `_op` (`insert` or `update`) column +on every row; or `final`, nothing more. A day depends only on the seed, `--as-of`, the day and the +model, so day *n* never changes when you generate more days or add an unrelated table. From Python: + +```python +from model2data.generate.days import generate_days +from model2data.model import load + +days = generate_days(load("examples/ecommerce_daily.model2data.yml"), 7, seed=42) +days[3].tables["orders"].inserted # rows day 3 added +days[3].tables["orders"].updated # rows day 3 changed, with their new values +days[3].tables["orders"].state # the table after day 3 +``` + +`history: true` in a table's `incremental` also writes every version of every row as a +`
_history` table: see [A source that keeps its history](history.md). diff --git a/docs/dbt-project.md b/docs/dbt-project.md new file mode 100644 index 0000000..42862b1 --- /dev/null +++ b/docs/dbt-project.md @@ -0,0 +1,113 @@ +# The generated dbt project + +## Running it + +Run dbt to load, transform, and test the data: + +```bash +cd dbt_ecommerce +dbt build +``` + +Staging models `ref()` their seeds, so a single `dbt build` loads the seeds, builds the models, +and runs every generated test in one dependency-ordered pass — no separate `dbt seed`/`dbt run` +needed, even on a brand-new database. (The individual `dbt deps`, `dbt seed`, and `dbt run` +commands still work if you'd rather drive the steps yourself; the generated project declares no +packages, so `dbt deps` is a no-op.) + +Your analytics-ready dataset is now in DuckDB! + +To target Postgres instead, install the extra and pass `--adapter postgres`: + +```bash +pip install "model2data[postgres]" +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --adapter postgres +``` + +Connection details are read from environment variables (`MODEL2DATA_PG_HOST`, `MODEL2DATA_PG_PORT`, `MODEL2DATA_PG_USER`, `MODEL2DATA_PG_PASSWORD`, `MODEL2DATA_PG_DATABASE`), defaulting to `localhost:5432` with a `postgres`/`postgres` user for local development. + +After generation, the CLI prints a short summary — tables and rows generated, relationships found in the model, and any columns that fell back to generic placeholder text because neither their type nor name could be matched. + +Pass `--unit-tests` to also generate deterministic dbt unit test fixtures (`models/staging/ut_stg_
.yml`) from the actually-generated seed rows: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --unit-tests +``` + +This targets dbt-core's native unit testing feature, which works out of the box with the base +install — see [dbt-core versions](../README.md#dbt-core-versions). + +## Data tests from the model's hints + +Your model's hints also write **data tests** for the dbt project, so the model that generates the +fixtures guards the real pipeline too. Every hint that states a constraint becomes a generic test +on the staging model, shipped as a self-contained macro in `macros/model2data_hint_tests.sql` (no +dbt package, so `dbt build` still works offline): + +| In the model | dbt test | Tolerance | +|---|---|---| +| `generate: {min, max}` on a numeric column | `model2data_between` | none | +| `generate: {after: other}` | `model2data_not_before` (where both are not null) | none | +| `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 | +| `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none | +| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none | +| `grain` on a table | `model2data_unique_combination` | none | +| an enum-typed column | `accepted_values` (always written) | none | + +`true_rate`, `weights`, `skew`, `distribution` and the temporal shape hints (`business_hours`, +`growth`, `seasonality`) describe a statistical shape rather than a constraint a row can break, so +they write no test. `--hint-tests {error,warn,off}` sets these tests' severity (default `warn`: +they describe intent and should not break a first `dbt build` on real data); the structural tests +(`not_null`, `unique`, `relationships`) are unaffected. A model with no such hints generates the +same YAML as before. To call the mapping from Python, `model2data.dbt.hint_tests.hint_tests_for(model)` +returns the tests as data. + +## Structure + +The generated dbt project includes: + +``` +dbt_{project_name}/ +├── seeds/ +│ └── raw/ +│ ├── __seed_config.yml # seed descriptions + column-type overrides +│ ├── table1.csv +│ └── table2.csv +├── models/ +│ └── staging/ +│ ├── stg_table1.sql +│ ├── stg_table1.yml +│ ├── ut_stg_table1.yml # only with --unit-tests +│ └── ... +├── data-tests/ +│ ├── unique_combination_stg_table1_col_a_col_b.sql # only for composite pk/unique keys +│ └── metrics/metric_revenue.sql # only with --metrics: one per metric +├── osi/{project_name}.yml # only with --metrics: Apache Ossie 0.1.1 +├── metric_values.json # only with --metrics: each metric's known value +├── defects_report.json # only with defects: each defect and the tests it breaks +├── EXPECTED_FAILURES.md # only with defects +├── {model file} # the model it was generated from, copied as given +├── {stem}.metrics.yml # only with --metrics, copied as given +├── macros/ +│ ├── generate_schema_name.sql +│ └── model2data_hint_tests.sql # only when the model's hints write tests +├── dbt_project.yml +├── profiles.yml # DuckDB or Postgres config, depending on --adapter +└── {project_name}_profile.duckdb # DuckDB adapter only +``` + +- **Seeds**: CSV files with generated synthetic data, plus `__seed_config.yml` — each seed's + `description:` (the table's `description` in the model) and the column-type overrides that keep + all-digit text columns (barcodes, zero-padded postcodes, ...) from being loaded as integers. +- **Staging Models**: Basic dbt models that `ref()` their seed. Using `ref()` rather than + declaring the seeds as dbt `sources` is what gives each model a real DAG edge to the seed + behind it, so one `dbt build` orders seeds before models on a fresh database. +- **Tests**: A YAML per staging model with column tests (`not_null`, `unique`, `relationships`, + and `accepted_values` for enum-typed columns), plus the hint tests above. A column's `description` in the model becomes + its `description:` field. +- **Composite key tests**: Composite primary/unique keys (`keys`, or several `pk: true` columns) get + a singular SQL test under `data-tests/`, dbt's configured `test-paths`. +- **Profiles**: Pre-configured for DuckDB (file-based) or Postgres (via env vars), with schema handling. +- **Unit tests** (opt-in via `--unit-tests`): `models/staging/ut_stg_
.yml` fixtures built + from real generated rows, co-located with each staging model so dbt (which only parses unit + tests from `model-paths`) picks them up. diff --git a/docs/defects.md b/docs/defects.md new file mode 100644 index 0000000..c71a91b --- /dev/null +++ b/docs/defects.md @@ -0,0 +1,35 @@ +# Defects on purpose + +A dbt test you have never seen fail is a test you hope works. `--defects` puts deliberate, +counted defects in the generated data, so `dbt build` fails exactly the tests it should — for +teaching dbt, or for proving a project's tests fire: + +```bash +model2data --file examples/ecommerce.model2data.yml --seed 42 --defects training +cd dbt_ecommerce && dbt build # fails unique, not_null, relationships, accepted_values once each +``` + +`training` breaks each kind of standard dbt test once; `messy` puts a small share of every defect +on every table; `clean` (the default) is the data as it always was, byte for byte; `none` also +ignores the defects a model's tables list, for the clean data of a training model. A table can +also list its own defects in the model (spec 0.3.0): + +```yaml +orders: + defects: + - {type: duplicate_keys, count: 3} # fails unique + - {type: orphan_foreign_keys, column: customer_id, share: 0.02} # fails relationships + - {type: nulls, column: order_date, count: 5} # fails not_null +``` + +The types are `duplicate_keys`, `orphan_foreign_keys`, `nulls`, `invalid_values` (fails +`accepted_values`), `messy_text` (whitespace and casing to clean in staging), and, with `--days`, +`late_arriving` (rows an incremental model filtering on `updated_at > max(updated_at)` skips) and +`late_updates` (versions a `strategy: timestamp` snapshot misses) and `overlapping_history` (see +[A source that keeps its history](history.md)). Every run with defects writes +`defects_report.json` (each defect, the rows it broke, the tests it breaks) and +`EXPECTED_FAILURES.md` into the project, so a test that does not fire is visible. Defects are applied +after the clean data, from their own random stream: the same seed gives the same bytes, and a +defect on one table moves nothing in another. From Python, `model2data.defects.planned_defects` +says what a run will break, `model2data.output.finish_run` applies it (and builds the histories) +after `generate_days`, and `write_defects_report` / `write_expected_failures` write the two files. diff --git a/docs/distributions.md b/docs/distributions.md new file mode 100644 index 0000000..73f3d3b --- /dev/null +++ b/docs/distributions.md @@ -0,0 +1,62 @@ +# How the data is spread + +By default every parent row is equally likely to be picked for a child row, and every column +gets the same generic null rate, value spread, and true/false split. `--skew` changes the first +part: `0.0` is that uniform default, `1.0` means a handful of parents hold most of the children — +"a fifth of the customers place most of the orders": + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --skew 0.8 +``` + +Column hints shape the rest, per column: + +```yaml +orders: + columns: + id: {type: int, pk: true} + customer_id: + type: int + references: customers.id + generate: {skew: 0.9} + status: + type: order_status + generate: + weights: {delivered: 20, cancelled: 2} + is_paid: + type: boolean + generate: {true_rate: 0.9} + discount_code: + type: varchar + generate: {null_rate: 0.8} + shipping_city: + type: varchar + generate: {distinct: 12} +``` + +`skew` on a foreign key overrides `--skew` for just that column. `weights` biases an enum column +toward the values named (unnamed values still appear, at weight 1). `true_rate` is the fraction of +non-null rows a boolean column comes back `true`. `null_rate` replaces the column's default null +fraction outright. `distinct` draws the column's values from a fixed-size pool instead of a fresh +value per row — a `shipping_city` most warehouses only ever see a handful of. + +## A number's distribution + +`min`/`max` alone only ever drew uniformly between them. A `distribution` hint on an integer or +decimal column picks a different shape instead: + +```yaml +total_amount: + type: numeric + generate: + min: 5 + distribution: {kind: lognormal, median: 80, spread: 0.6} +``` + +`normal` takes `mean` (the centre) and `stddev` (the spread); `lognormal` takes `median` (the +typical value) and `spread` (how heavy the tail is — 0.3 is mild, 1.0 is heavy); `exponential` +takes `mean` (the average). Any left unset default to the midpoint of the column's effective +`min`/`max` (or `stddev` = range / 6, `spread` = 0.5). `min`/`max` still clip the result — a +`normal` centred near an edge redraws a bounded number of times before clamping, so it never +loops forever and never crosses the bound. Leaving `distribution` out, or setting it to +`uniform`, is exactly today's behaviour. diff --git a/docs/generating.md b/docs/generating.md new file mode 100644 index 0000000..1a7c468 --- /dev/null +++ b/docs/generating.md @@ -0,0 +1,68 @@ +# Generating a project + +We bundle several example schemas in `examples/` — this walkthrough uses the e-commerce one +(`examples/ecommerce.model2data.yml`: customers, products, orders, order items, and reviews). + +Generate a project with synthetic data: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 +``` + +This creates a `dbt_ecommerce/` folder with your data and dbt setup. + +Real schemas are rarely uniform. `--rows-for` sizes individual tables, so a handful of customers +can sit behind a large orders table the way they would in the warehouse you're modelling: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 \ + --rows-for customers=50 --rows-for order_items=5000 +``` + +`--seed` reproduces a run's numbers, but dates and timestamps are generated relative to the +current date, so the same seed drifts once the day turns over. `--as-of` pins the date they're +anchored on, and the whole dataset reproduces on any later day — which is what makes a generated +fixture safe to commit: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --as-of 2026-01-31 +``` + +## The determinism promise + +The same model, seed and `--as-of` — with the same options (`--rows`, `--rows-for`, +`--table-seed`, `--locale`, `--defects`, `--days`, …) — produce the same files, byte for byte: + +- on any machine and operating system, and on every supported Python version; +- with any Faker or pandas version model2data accepts; +- with the same model2data version, whenever you run it. + +A new release may change what a seed produces, to fix a bug or make the data more realistic. +When it does, the changelog says which models are affected and why. Pin the version +(`model2data==1.11.0`) when fixtures must stay the same across upgrades. + +`tests/test_determinism.py` holds the file hashes of a spread of models, presets, multi-day runs +and locales, and runs on every pull request — against the newest Faker and pandas too — so a +change to a single output byte fails the build unless it is deliberate and in the changelog. + +If one table comes out wrong and the rest looks right, `--table-seed` re-rolls just that table. +Every other table's seed CSV stays byte-identical, and children of the re-rolled table still +reference rows that exist, so there's nothing to re-check but the table you asked to change: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --table-seed orders=7 +``` + +`--locale` picks the country every generated person and address comes from (`en_US` by default); +it's a per-run setting, so a table can't end up holding one Belgian and one American address. A +`country` column that sits beside a `city`/`street`/`state`/`postcode` column always agrees with +that place; a `country` column with none of those beside it isn't describing anyone's address, so +it reads as an international mix instead, with the locale's own country the most common: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 --locale nl_BE +``` + +Next: [run the generated dbt project](dbt-project.md), or shape the data with +[days after the first](days.md), [when things happen](time-shapes.md), +[how the data is spread](distributions.md) and [defects on purpose](defects.md). diff --git a/docs/history.md b/docs/history.md new file mode 100644 index 0000000..e5d004e --- /dev/null +++ b/docs/history.md @@ -0,0 +1,12 @@ +# A source that keeps its history + +`history: true` in a table's `incremental` (spec 0.3.0) also writes `
_history`: every +version of every row over the generated days, with `valid_from`, `valid_to` (null for the current +version) and `is_current` — an SCD type 2 source to build snapshots and point-in-time joins +against. The project tests it with two package-free tests, one current row per key and no +overlapping versions; the `overlapping_history` defect breaks the second. + +```yaml +orders: + incremental: {new_per_day: 20, update_rate: 0.1, updated_at: updated_at, history: true} +``` diff --git a/docs/metrics.md b/docs/metrics.md new file mode 100644 index 0000000..cfa2a7b --- /dev/null +++ b/docs/metrics.md @@ -0,0 +1,78 @@ +# Metrics with known values + +What the numbers mean -- revenue, orders, average order value -- goes in a second, optional file +beside the model, `.metrics.yml`, defined by the [metrics spec 0.1.0](../model2data/spec/metrics/README.md) +([schema](../model2data/spec/metrics/metrics.schema.json)): + +```yaml +# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json +model2data-metrics: 0.1.0 +model: coffee_webshop + +metrics: + revenue: + label: Revenue + measure: orders.total_amount # summed: the column's `measure` is true + where: + orders.status: [paid, shipped, delivered] + time: orders.order_date + format: currency + orders: + measure: orders.id + agg: count_distinct + average_order_value: + ratio: {numerator: revenue, denominator: orders} + coffee_units_sold: + measure: order_items.quantity # dated by orders.order_date, through order_id + where: + orders.status: {ne: cancelled} + products.category: [beans, ground, capsules] + any: + - orders.channel: web + - order_items.quantity: {gte: 3} +``` + +(the whole file, which conforms against the example model, is +[`model2data/spec/examples/coffee_webshop.metrics.yml`](../model2data/spec/examples/coffee_webshop.metrics.yml)). +A metric is a column aggregated (`measure`, with `agg` or the column's own `measure`), a row count +(`count`), a `ratio` of two metrics' totals, or arithmetic over metrics (`expression`). A filter +can reach any table the metric's table reaches through its foreign keys, many-to-one, along exactly +one path; nulls behave as in SQL. Every `measure` column of the model also yields a metric on its +own (`orders_total_amount`), and a fact a row count (`orders_count`); `infer: false` turns that +off. `format` changes no value: `percent` expects a fraction (0.6 shows as 60%), so a share is +`web_revenue / revenue`, never `* 100`. The generator never reads the file: the data is the same +with or without it. + +```bash +model2data validate coffee_webshop.metrics.yml # against coffee_webshop.model2data.yml beside it +model2data --file coffee_webshop.model2data.yml --metrics coffee_webshop.metrics.yml --rows 200 --seed 42 --as-of 2026-01-31 +cd dbt_coffee_webshop && dbt build # one test per metric, each checking its known value +``` + +With `--metrics`, the run also writes, and changes nothing else: + +- **`metric_values.json`** -- each metric's **known value** over the generated data, overall and + per calendar month of its time column: with the command above, revenue is 28344.22, orders 200, + average order value 141.7211. A run knows its data, so it knows its numbers; a semantic layer or + dashboard built on the data can be checked against them. Counts, and sums, minimums and maximums of + integer columns, are exact; everything else is rounded to 6 decimal places, half to even. Same seed, same bytes. +- **`data-tests/metrics/metric_.sql`** -- a singular dbt test per metric that computes it + over the project's staging models and fails unless it gets the known value (within + `0.000001 + 0.000000001 × |value|`), so `dbt build` proves the metric logic on the warehouse. +- **`osi/.yml`** -- the model and its metrics as an [Apache Ossie](https://github.com/apache/ossie) + (Open Semantic Interchange) 0.1.1 semantic model: tables as datasets over the staging models + (`staging.stg_
`), foreign keys as relationships, date columns as time dimensions, metrics + as ANSI SQL expressions. What Ossie 0.1.1 has no field for (a metric's label, format and time, + enum members) is carried in `custom_extensions` and listed at the top of the file. + +`model2data metrics list -f MODEL [--metrics FILE]` shows every metric, and `model2data metrics +export -f MODEL [--metrics FILE] --to ossie [-o FILE]` writes the Ossie file without generating. +Without `--metrics`, both use the model's inferred metrics. A sibling metrics file is not picked up +by `generate` on its own: a file appearing beside a model must not change what an unchanged +command writes. From Python, `model2data.metrics` loads and checks a file (`load`, `validate`), +resolves it with the model into one semantic representation (`resolve`), and computes +(`known_values`) and exports (`to_ossie`) from it. + +The e-commerce example has a metrics file too, [`examples/ecommerce.metrics.yml`](../examples/ecommerce.metrics.yml) +(revenue, orders, average order value, units sold); the [README](../README.md#before-and-after) +shows what a run with it writes. diff --git a/docs/model-file.md b/docs/model-file.md new file mode 100644 index 0000000..49ac74d --- /dev/null +++ b/docs/model-file.md @@ -0,0 +1,76 @@ +# The model file + +A model is one YAML document, `.model2data.yml` — or the same document as JSON. Its +format is [spec 0.4.0](../model2data/spec/README.md), with a JSON Schema +([`model.schema.json`](../model2data/spec/model.schema.json)) your editor can autocomplete and check +against: + +```yaml +# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/0.2.0/model.schema.json +model2data: 0.2.0 +name: coffee_webshop + +enums: + order_status: [pending, paid, shipped, delivered, cancelled] + +tables: + customers: + description: One row per registered customer account + columns: + id: {type: bigint, pk: true} + email: {type: email, unique: true, not_null: true} + phone: + type: phone_number + generate: {null_rate: 0.4} + + orders: + columns: + id: {type: bigint, pk: true} + customer_id: + type: bigint + not_null: true + references: customers.id + generate: {skew: 0.8} + total_amount: + type: numeric + not_null: true + generate: + min: 10 + max: 5000 + distribution: {kind: lognormal, median: 120, spread: 0.7} + status: + type: order_status + generate: + weights: {delivered: 20, cancelled: 1} + +run: + seed: 1 + as_of: 2026-01-01 +``` + +A column's `type` is an SQL type, an enum of the model, or a generator (`email`, `first_name`, +`ean13`, ...). `generate` holds its hints, each a typed value the schema documents. `run` saves +the generation settings with the model; every CLI option overrides the one it names. The +[reference example](../model2data/spec/examples/coffee_webshop.model2data.yml) is a complete model, +and every schema in [`examples/`](../examples/) comes as one. + +Check a model without generating anything — every issue is printed with its path in the document: + +```bash +model2data validate examples/ecommerce.model2data.yml +``` + +## DBML is supported input + +`--file` also takes a `.dbml` file, which model2data converts to the same model before generating +(parsed with [pydbml](https://github.com/Vanderhoof/PyDBML)). Hints written as a JSON note on a +column (`[note: '{"min": 1, "max": 5}']`) become its `generate`; any other note is its +description. `convert` writes the model a DBML file converts to, for you to keep: + +```bash +model2data convert examples/ecommerce.dbml -o ecommerce.model2data.yml +``` + +The spec's [From 0.1](../model2data/spec/README.md#from-01) section lists how each DBML feature +converts. A few newer or rarer DBML features — `check` constraints, `Records`, `TablePartial`, +unquoted non-ASCII names — can't be read yet, and are refused with the line and what to change. diff --git a/docs/time-shapes.md b/docs/time-shapes.md new file mode 100644 index 0000000..4d0f3c5 --- /dev/null +++ b/docs/time-shapes.md @@ -0,0 +1,60 @@ +# When things happen + +By default, every date and timestamp is drawn uniformly across its window. `--business-hours`, +`--growth`, and `--seasonality` shape that instead — weekdays and working hours, a trend across +the window, and an annual cycle peaking in Q4: + +```bash +model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42 \ + --business-hours --growth 0.5 --seasonality 0.3 +``` + +Within a row, created/updated/deleted-style columns are ordered automatically — `updated_at` never +lands before its own `created_at` — under any profile, uniform included. A column whose name +doesn't say what it depends on can say so explicitly with an `after` hint: + +```yaml +shipped_at: + type: timestamp + generate: {after: ordered_at} +``` + +A column that only has a value in some states says which with `when` (spec 0.4.0: the model says +`model2data: 0.4.0`): it holds a value on +exactly the rows whose named column holds one of the listed values, and is null on every other +row — no `pending` order with a `shipped_at`, no `cancelled` subscription without a +`cancelled_at`: + +```yaml +shipped_at: + type: timestamp + generate: {after: ordered_at, when: {status: [shipped, delivered]}} +``` + +The named column is another column of the table (an enum, boolean, number or text column; several +named must all match), and the column carrying `when` is nullable. On the matching rows it is +never null, unless it has a `null_rate`, which then counts only those rows. On days after the +first (`--days`), a row that moves into a listed status gets its value that day and a row that +moves out loses it. Every other column comes out exactly as it would without the hint. + +The flags above (or `run.shape` in the model) shape every date and timestamp column the same +way, run-wide. `business_hours`, `growth`, and `seasonality` column hints override that for one +column at a time — the whole point being a run can be uniform everywhere except the one column +that needs shaping, or shaped everywhere except the one column that shouldn't be: + +```yaml +orders: + columns: + id: {type: int, pk: true} + created_at: + type: timestamp + generate: {business_hours: true, growth: 0.4} + refunded_at: + type: timestamp + generate: {growth: 0} +``` + +Here `created_at` gets business hours and growth even on an otherwise-uniform run, while +`refunded_at` stays flat even under `--growth 0.5` — each hint only replaces the fields it names, +so a partial hint like `{growth: 0}` leaves that column's `business_hours`/`seasonality` at +whatever the run-level flags set. diff --git a/examples/README.md b/examples/README.md index bcdc051..d34346b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -79,6 +79,17 @@ dbt seed && dbt run - **Unique constraints**: Email uniqueness per customer - **Realistic data generation**: Names, emails, phone numbers generated via Faker library +**Metrics**: `ecommerce.metrics.yml` beside it ([metrics spec 0.1.0](../model2data/spec/metrics/README.md)) +defines revenue, orders, average order value and units sold. Pass it with `--metrics` and the run +also writes each metric's known value (`metric_values.json`), a dbt test per metric and an Apache +Ossie file; the data is the same with or without it: + +```bash +model2data --file examples/ecommerce.model2data.yml --metrics examples/ecommerce.metrics.yml \ + --rows 200 --seed 42 --as-of 2026-01-31 +cd dbt_ecommerce && dbt build +``` + **Use case**: Perfect for learning how to model transactional systems. Run this to generate realistic e-commerce data for testing, analytics, or building dbt pipelines. **Try it**: diff --git a/examples/ecommerce.metrics.yml b/examples/ecommerce.metrics.yml new file mode 100644 index 0000000..2f31750 --- /dev/null +++ b/examples/ecommerce.metrics.yml @@ -0,0 +1,33 @@ +# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json +model2data-metrics: 0.1.0 +model: ecommerce + +metrics: + revenue: + label: Revenue + description: Order value of the orders that were not cancelled. + measure: orders.total_amount + agg: sum + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + format: currency + + orders: + label: Orders + count: orders + where: + orders.status: [processing, shipped, delivered] + time: orders.order_date + + average_order_value: + label: Average order value + ratio: {numerator: revenue, denominator: orders} + format: currency + + units_sold: + label: Units sold + measure: order_items.quantity # dated by orders.order_date, through order_id + agg: sum + where: + orders.status: {ne: cancelled} diff --git a/model2data/cli.py b/model2data/cli.py index c9af042..1f2045b 100644 --- a/model2data/cli.py +++ b/model2data/cli.py @@ -46,6 +46,22 @@ ) from model2data.generate.history import history_tables from model2data.generate.options import TimeProfile +from model2data.metrics import ( + SPEC_VERSION as METRICS_SPEC_VERSION, +) +from model2data.metrics import ( + Metrics, + MetricsError, + is_metrics_file, + known_values, + metrics_stem, + sibling_model, + to_ossie, + write_metric_tests, +) +from model2data.metrics import load as load_metrics +from model2data.metrics import resolve as resolve_metrics +from model2data.metrics import validate as validate_metrics from model2data.model import ( DEFECT_PRESETS, Issue, @@ -198,6 +214,18 @@ def _read_model(file: Path) -> Model: return model +def _read_metrics(file: Path, model: Model, model_file: Path) -> Metrics: + """The metrics in `file`, checked against `model`, or exit 1 listing every issue.""" + try: + found = load_metrics(file, model, model_name=_model_stem(model_file)) + except MetricsError as error: + _print_issues(file, [*error.issues, *error.warnings]) + raise typer.Exit(1) from None + if found.warnings: + _print_issues(file, found.warnings) + return found + + def _print_issues(file: Path, issues: list[Issue], label: Optional[str] = None) -> None: """Errors, then warnings, each with its document path.""" label = label or file.name @@ -428,6 +456,21 @@ def main( "(changelog/TABLE.csv, every row with _day and _op), or 'final' (only the seeds)." ), ), + metrics_file: Optional[Path] = typer.Option( # noqa: B008 + None, + "--metrics", + exists=True, + file_okay=True, + dir_okay=False, + readable=True, + resolve_path=True, + help=( + "The model's metrics file (.metrics.yml). Adds metric_values.json (each\n" + "metric's known value over the generated data), osi/NAME.yml (Apache Ossie 0.1.1)\n" + "and a dbt test per metric (data-tests/metrics/). The data is the same with or\n" + "without it. Not picked up from beside the model: pass it." + ), + ), ): """ Generate synthetic data and a dbt project from a model. @@ -474,6 +517,8 @@ def main( # Read the model (names untouched) # ------------------------- model = _read_model(file) + metrics_file = _given(metrics_file) + metrics_doc = _read_metrics(metrics_file, model, file) if metrics_file is not None else None inputs = to_engine(model) tables, refs = inputs.tables, inputs.refs run = inputs.run @@ -677,6 +722,28 @@ def pick(option: Any, from_run: Any, default: Any) -> Any: write_defects_report(report, dest) write_expected_failures(report, dest) + # ------------------------- + # Metrics: only added files, computed from what the seeds hold + # ------------------------- + metrics_count: Optional[int] = None + if metrics_doc is not None and metrics_file is not None: + typer.echo("📏 Computing each metric's known value and its dbt test...") + semantic = resolve_metrics(model, metrics_doc, model_name=_model_stem(file)) + values = known_values( + semantic, + generated_tables, + seed=seed, + as_of=anchor.isoformat() if anchor is not None else None, + ) + (dest / "metric_values.json").write_text(values.to_json(), encoding="utf-8") + write_metric_tests(dest, semantic, values) + (dest / "osi").mkdir(exist_ok=True) + (dest / "osi" / f"{project_name}.yml").write_text( + to_ossie(semantic).to_yaml(), encoding="utf-8" + ) + shutil.copy(metrics_file, dest / metrics_file.name) + metrics_count = len(semantic.metrics) + # ------------------------- # Summary # ------------------------- @@ -724,6 +791,12 @@ def pick(option: Any, from_run: Any, default: Any) -> Any: for label in duplicate_unique: typer.echo(f" - {label}") + if metrics_count is not None: + typer.echo( + f" Metrics: {metrics_count} (metric_values.json, " + f"osi/{project_name}.yml, data-tests/metrics/)" + ) + if report is not None: _print_defects(report) @@ -820,11 +893,25 @@ def validate_command( require_files: bool = typer.Option( False, "--require-files", help="Exit 1 when no model file was given or matched." ), + model_option: Optional[Path] = typer.Option( # noqa: B008 + None, + "--model", + "-m", + exists=True, + file_okay=True, + dir_okay=False, + readable=True, + help=( + "The model a metrics file (*.metrics.yml) is checked against. Default: the model " + "beside it with the same stem (coffee.model2data.yml for coffee.metrics.yml)." + ), + ), ): """Check that models conform to spec 0.4.0 (or 0.2.x, 0.3.x), printing every issue with its path. - Takes one or more files and/or --glob patterns. Exits 1 when any file has an error; - warnings are printed, and the model conforms. + Takes one or more files and/or --glob patterns. A metrics file (*.metrics.yml) is checked + against metrics spec 0.1.0 and against its model. Exits 1 when any file has an error; + warnings are printed, and the file conforms. """ if output_format not in ("text", "github"): raise typer.BadParameter("must be 'text' or 'github'", param_hint="--format") @@ -837,7 +924,11 @@ def validate_command( failed = 0 for file in targets: - issues = validate(file) + against: Optional[Path] = None + if is_metrics_file(file): + issues, against = _metrics_issues(file, _given(model_option)) + else: + issues = validate(file) if output_format == "github": _print_github_issues(file, issues) else: @@ -845,9 +936,14 @@ def validate_command( if any(issue.is_error for issue in issues): failed += 1 elif output_format == "text": - typer.echo( - f"✅ {file if len(targets) > 1 else file.name} conforms to spec {_spec_of(file)}." - ) + shown = file if len(targets) > 1 else file.name + if is_metrics_file(file): + model_note = f", checked against {against.name}" if against is not None else "" + typer.echo( + f"✅ {shown} conforms to metrics spec {METRICS_SPEC_VERSION}{model_note}." + ) + else: + typer.echo(f"✅ {shown} conforms to spec {_spec_of(file)}.") if len(targets) > 1 or output_format == "github": count = len(targets) checked = f"{count} model file{'s' if count != 1 else ''}" @@ -860,6 +956,32 @@ def validate_command( raise typer.Exit(1) +def _metrics_issues(file: Path, model_option: Optional[Path]) -> tuple[list[Issue], Optional[Path]]: + """A metrics file's issues, checked against `model_option` or the model beside it, and + the model it was checked against (None when there was none to find).""" + model_file = model_option or sibling_model(file) + if model_file is None: + note = Issue( + "", + "no model to check it against: there is no " + f"{metrics_stem(file)}.model2data.yml beside it. Checked on its own; " + "pass --model to check its tables, columns and filters too", + severity="warning", + ) + return [*validate_metrics(file), note], None + try: + model = load(model_file) + except ModelError: + return [ + Issue( + "", + f"its model, {model_file.name}, does not conform: run `model2data validate " + f"{model_file.name}` and fix it first", + ) + ], model_file + return validate_metrics(file, model, model_name=_model_stem(model_file)), model_file + + @app.command("convert") def convert_command( file: Path = typer.Argument( # noqa: B008 @@ -941,3 +1063,115 @@ def guide_command( typer.echo(f"# model2data guide: {reason} -> {topic.value}\n") page = resources.files("model2data").joinpath(f"guide/{topic.value}.md") typer.echo(page.read_text("utf-8"), nl=False) + + +metrics_app = typer.Typer( + help=( + "The model's metrics (metrics spec 0.1.0): list them, or export them without " + "generating. `generate --metrics FILE` computes their known values and dbt tests." + ), + no_args_is_help=True, +) +app.add_typer(metrics_app, name="metrics") + +_METRICS_EXPORTS = ("ossie",) + + +def _metrics_inputs(file: Path, metrics_file: Optional[Path]) -> Any: + """The semantic model of a model file and its metrics file (None: the inferred metrics).""" + model = _read_model(file) + found = _read_metrics(metrics_file, model, file) if metrics_file is not None else None + return resolve_metrics(model, found, model_name=_model_stem(file)) + + +_MODEL_OPTION = typer.Option( + ..., + "--file", + "-f", + exists=True, + file_okay=True, + dir_okay=False, + readable=True, + help="The model: a .model2data.yml / .yaml / .json document, or a .dbml file.", +) +_METRICS_OPTION = typer.Option( + None, + "--metrics", + exists=True, + file_okay=True, + dir_okay=False, + readable=True, + help=( + "The model's metrics file. Without it, the metrics are the ones the model implies: " + "one per `measure` column and a row count per fact." + ), +) + + +@metrics_app.command("list") +def metrics_list_command( + file: Path = _MODEL_OPTION, + metrics_file: Optional[Path] = _METRICS_OPTION, +): + """List every metric: its kind, where it comes from, what it is dated by.""" + semantic = _metrics_inputs(file, _given(metrics_file)) + if not semantic.metrics: + typer.echo("No metrics: the model has no `measure` column and the file adds none.") + return + width = max(len(name) for name in semantic.metrics) + for name, metric in semantic.metrics.items(): + origin = "inferred" if metric.inferred else "file" + if metric.kind == "simple": + what = f"{metric.agg} of {metric.table}.{metric.column}" + elif metric.kind == "count": + what = f"rows of {metric.table}" + elif metric.kind == "ratio": + what = f"{metric.numerator} / {metric.denominator}" + else: + what = str(metric.expression) + filtered = ", filtered" if metric.where is not None else "" + dated = f", by {metric.time}" if metric.time else "" + typer.echo(f"{name:<{width}} {metric.kind:<7} {origin:<8} {what}{filtered}{dated}") + + +@metrics_app.command("export") +def metrics_export_command( + file: Path = _MODEL_OPTION, + metrics_file: Optional[Path] = _METRICS_OPTION, + to: str = typer.Option("ossie", "--to", help="The format: 'ossie' (Apache Ossie 0.1.1, YAML)."), + output: Optional[Path] = typer.Option( # noqa: B008 + None, + "--output", + "-o", + dir_okay=False, + help="Where to write it (default: print it).", + ), + force: bool = typer.Option(False, "--force", help="Overwrite the output file if it exists."), +): + """Export the model's metrics, and the model as a semantic model, without generating data. + + Ossie datasets read the staging models of the dbt project `generate` writes + (staging.stg_
). What Ossie 0.1.1 cannot say is listed at the top of the file, + and carried in custom_extensions. + """ + to = (_given(to) or "ossie").lower() + if to not in _METRICS_EXPORTS: + raise typer.BadParameter( + f"Choose one of: {', '.join(_METRICS_EXPORTS)}.", param_hint="--to" + ) + semantic = _metrics_inputs(file, _given(metrics_file)) + export = to_ossie(semantic) + text = export.to_yaml() + output = _given(output) + if output is None: + typer.echo(text, nl=False) + return + if output.exists() and not _given(force): + typer.echo(f"❌ {output} already exists. Use --force to overwrite it.") + raise typer.Exit(1) + output.write_text(text, encoding="utf-8") + typer.echo(f"✅ Wrote {output} (Apache Ossie 0.1.1, {len(semantic.metrics)} metrics)") + if export.lossiness: + typer.echo("ℹ️ Not expressible in Ossie 0.1.1 itself, carried in custom_extensions:") + for loss in export.lossiness: + typer.echo(f" - {loss}") diff --git a/model2data/guide/triage.md b/model2data/guide/triage.md index f9a0a10..c3fbd66 100644 --- a/model2data/guide/triage.md +++ b/model2data/guide/triage.md @@ -18,6 +18,12 @@ says it created the project. Fix the **model file**, never the generated project (`tables.orders.columns.status.generate.weights`). Fix that value. Exit 1 until all are gone. - `⚠️ : N warnings`: it conforms. A reference to a parent column that is not a key: make the parent `pk` or `unique`. A `grain` without a key: add the key, or drop `grain`. +- A metrics file (`.metrics.yml`) is checked against the model beside it. An error + naming two paths: the filter's table is reached twice (two foreign keys to one table); + filter on a column of the table in between instead. `cannot be reached`: the column is on + a child table; put the metric on that table, or drop the filter. `compares by order`: list + the values with `in`. A warning `no date or timestamp column`: set `time`. A warning + `can never match`: the value is outside what the model generates; fix the value or the hint. ## generate @@ -51,9 +57,13 @@ Take the first rule that applies to each line: 3. Any other failing `not_null`, `unique`, `relationships`, `accepted_values` test: the generated data broke a key the model declares. Re-read the generate summary; rule 4 or 5 above applies. -4. A WARN from a hint test (`min`/`max`, `after`, `null_rate`, `distinct`, `grain`): the hint +4. A failing `metric_` test (with `--metrics`): the warehouse computed the metric + differently from `metric_values.json`. Regenerate with `--force` first (the test and the + seeds must come from the same run); if it still fails, report it with both files. Never + edit `metric_values.json` or the test to make it pass. +5. A WARN from a hint test (`min`/`max`, `after`, `null_rate`, `distinct`, `grain`): the hint and the data disagree. Fix the hint in the model. -5. Still failing after the model is fixed: report it with the model file and the output. +6. Still failing after the model is fixed: report it with the model file and the output. Done looks like: `validate` exits 0, the summary has no ⚠️ lines, `dbt build` ends `ERROR=0` (and `WARN=0` unless the model asks for defects). diff --git a/model2data/guide/tune.md b/model2data/guide/tune.md index 4bfc165..4003a11 100644 --- a/model2data/guide/tune.md +++ b/model2data/guide/tune.md @@ -5,8 +5,10 @@ | Command | Does | Exit | |---|---|---| | `model2data --file M` | same as `model2data generate --file M` | see below | -| `model2data validate M...` | check models against spec 0.4.0 | 0 conforms (warnings allowed), 1 any error | +| `model2data validate M...` | check models against spec 0.4.0, and `*.metrics.yml` files against the metrics spec and their model (`--model M`, else the model beside it with the same stem) | 0 conforms (warnings allowed), 1 any error | | `model2data convert M` | print M as `.model2data.yml`; `-o FILE` writes it, `--force` overwrites | 0 ok, 1 unreadable model or output exists | +| `model2data metrics list -f M [--metrics F]` | every metric: the file's, then the inferred ones | 0, 1 on a model or metrics error | +| `model2data metrics export -f M [--metrics F] --to ossie` | the model and its metrics as Apache Ossie 0.1.1; `-o FILE` writes it, `--force` overwrites | 0, 1 on an error or an existing output, 2 on another `--to` | | `model2data guide [TOPIC]` | this page; topics `setup`, `triage`, `tune` | 0 | `generate` exits 0 on success; 1 on a model error, an existing `dbt_/` without @@ -37,6 +39,7 @@ Every option overrides the model's `run` setting of the same meaning. | `--unit-tests` | | off | | `--name NAME` (project becomes `dbt_NAME`) | `name` (top level) | the model's `name`, else the file stem | | `--force` (replace `dbt_/`) | | off | +| `--metrics FILE` (the model's `*.metrics.yml`; never picked up on its own) | | none | ## Generator types @@ -136,4 +139,29 @@ With `--days N` the seeds hold the state after the last day, and `--days-format` Only tables with `incremental` change after the first day. +With `--metrics FILE`, and nothing else changed: `metric_values.json` (each metric's known +value over the seeds, overall and by month), `data-tests/metrics/metric_.sql` (a dbt +test per metric that fails unless the warehouse gets that value) and `osi/.yml` +(Apache Ossie 0.1.1). + +## Metrics file + +`.metrics.yml` beside the model, version 0.1.0 of its own spec +(`model2data/spec/metrics/README.md`, `metrics.schema.json` in the package). Top level: +`model2data-metrics` (0.1.0), `model` (the model's `name`), `metrics`, `dimensions`, `infer`. +A metric (name `[a-z][a-z0-9_]*`) has exactly one of `measure` (`table.column`, aggregated +by `agg`: `sum`, `average`, `min`, `max`, `median`, `count`, `count_distinct`, or else by +the column's own `measure`), `count` (a table: its rows), `ratio` (`{numerator, denominator}`, +two metric names) or `expression` (metric names, numbers, `+ - * /`, parentheses), and may +have `label`, `description`, `ai_context`, `format` (`number`, `currency`, `percent`: a fraction, 0.6 shows as 60%, never `* 100`), and, +with `measure` or `count`, `time` (a date or timestamp column) and `where`. `where` maps +`table.column` to a value (equals), a list (one of) or operators `eq`, `ne`, `in`, +`not_in`, `gt`, `gte`, `lt`, `lte`, `between: [a, b]`, `is_null: true`; `any: [...]` and +`all: [...]` combine filters. A null fails every condition but `is_null: true`. A filter +column is on the metric's table or reached through `references` child to parent along +exactly one path. Order operators only on numbers, dates and timestamps. Every `measure` +column also yields a metric `
_` and a fact `
_count`, unless +`infer: false`; a metric of the same name in the file replaces it. `dimensions` maps a +column to `false` (never a dimension), `true`, or `{label, description}`. + next: `model2data guide triage` diff --git a/model2data/metrics/__init__.py b/model2data/metrics/__init__.py new file mode 100644 index 0000000..446f218 --- /dev/null +++ b/model2data/metrics/__init__.py @@ -0,0 +1,100 @@ +"""Metrics next to the model: metrics spec 0.1.0, read, checked, resolved, computed and exported. + +A metrics file is `.metrics.yml` beside `.model2data.yml`, as +`model2data/spec/metrics/README.md` and `metrics.schema.json` define it. The +generator never reads it: a model generates the same data with or without one. + + >>> from model2data.model import load as load_model + >>> from model2data import metrics + >>> model = load_model("coffee_webshop.model2data.yml") + >>> found = metrics.load("coffee_webshop.metrics.yml", model) # MetricsError on errors + >>> semantic = metrics.resolve(model, found) # entities, relationships, dimensions, metrics + >>> values = metrics.known_values(semantic, frames) # over a run's generated tables + >>> export = metrics.to_ossie(semantic) # Apache Ossie 0.1.1 + >>> export.to_yaml(), export.lossiness + +`resolve(model)` with no metrics file gives the metrics the model implies on +its own: one per `measure` column, and a row count per fact. +""" + +from model2data.metrics.check import SCHEMA_URL, SPEC_VERSION, check_against, check_document +from model2data.metrics.dbt import metric_test_sql, write_metric_tests +from model2data.metrics.expression import ExpressionError +from model2data.metrics.expression import parse as parse_expression +from model2data.metrics.graph import Graph, Join +from model2data.metrics.infer import inferred_metrics +from model2data.metrics.ossie import OSSIE_VERSION, Loss, OssieExport, to_ossie +from model2data.metrics.reader import ( + MetricsError, + from_dict, + is_metrics_file, + load, + metrics_stem, + sibling_model, + validate, +) +from model2data.metrics.semantic import ( + Dimension, + Entity, + Measure, + Relationship, + ResolvedMetric, + SemanticModel, + resolve, +) +from model2data.metrics.types import ( + AGGREGATIONS, + FORMATS, + OPERATORS, + Condition, + DimensionOverride, + Metric, + Metrics, + Ratio, + Where, +) +from model2data.metrics.values import KnownValue, KnownValues, known_values + +__all__ = [ + "AGGREGATIONS", + "FORMATS", + "OPERATORS", + "OSSIE_VERSION", + "SCHEMA_URL", + "SPEC_VERSION", + "Condition", + "Dimension", + "DimensionOverride", + "Entity", + "ExpressionError", + "Graph", + "Join", + "KnownValue", + "KnownValues", + "Loss", + "Measure", + "Metric", + "Metrics", + "MetricsError", + "OssieExport", + "Ratio", + "Relationship", + "ResolvedMetric", + "SemanticModel", + "Where", + "check_against", + "check_document", + "from_dict", + "inferred_metrics", + "is_metrics_file", + "known_values", + "load", + "metric_test_sql", + "metrics_stem", + "parse_expression", + "resolve", + "sibling_model", + "to_ossie", + "validate", + "write_metric_tests", +] diff --git a/model2data/metrics/check.py b/model2data/metrics/check.py new file mode 100644 index 0000000..6d9cf4e --- /dev/null +++ b/model2data/metrics/check.py @@ -0,0 +1,566 @@ +"""Does a metrics file conform: the schema, the checks within the file, then against its model. + +`check_document(document)` is what a reader can say from the file alone: the +schema, the version, each metric having exactly one kind, every `expression` +parsing, and no metric depending on itself. `check_against(metrics, model)` +is the rest, which needs the model: every table and column named exists, every +aggregation fits its column, every filter value fits its column and every +filter and time column is reachable along exactly one path. Both return every +issue they find, each with the document path of the value at fault, like the +model's own checks (`model2data.model.validate`), whose schema messages they +reuse. +""" + +from __future__ import annotations + +import difflib +import json +from collections.abc import Mapping +from datetime import date, datetime +from functools import lru_cache +from importlib import resources +from typing import Any, Optional + +import jsonschema +from jsonschema.exceptions import ValidationError + +from model2data.metrics import expression as expr +from model2data.metrics.columns import ( + ColumnKind, + column_kind, + find_column, + is_numeric, + is_temporal, +) +from model2data.metrics.graph import Graph, describe +from model2data.metrics.infer import aggregation_of, inferred_metrics +from model2data.metrics.types import ( + ANY_KIND_AGGREGATIONS, + KIND_KEYS, + ORDER_OPERATORS, + Condition, + Metric, + Metrics, + Where, +) +from model2data.model.errors import Issue, PathPart, format_path +from model2data.model.types import Column, Model +from model2data.model.validate import _describe, _schema_issues, _show + +SPEC_VERSION = "0.1.0" +# The minor versions this reader implements. +READS = ("0.1",) +SCHEMA_URL = ( + f"https://www.jbanalytica.com/model2data/spec/metrics/{SPEC_VERSION}/metrics.schema.json" +) + +_PATTERN_WORDS = { + "metricName": "must be a metric name: a lower-case letter, then lower-case letters, " + "digits and underscores", + "columnPath": "must name a column as `table.column` or `schema.table.column`", + "tableKey": "must name a table as `table` or `schema.table`", +} + + +@lru_cache(maxsize=1) +def schema() -> dict[str, Any]: + """The packaged, normative JSON Schema of the metrics spec.""" + text = ( + resources.files("model2data") + .joinpath("spec/metrics/metrics.schema.json") + .read_text("utf-8") + ) + return json.loads(text) + + +@lru_cache(maxsize=1) +def _validator() -> Any: + return jsonschema.Draft202012Validator(schema()) + + +def _mapping(value: Any) -> Mapping: + return value if isinstance(value, Mapping) else {} + + +# --------------------------------------------------------------------------- +# The file alone +# --------------------------------------------------------------------------- +def check_document(document: Any) -> list[Issue]: + """Every way `document` (a parsed YAML or JSON value) fails to conform on its own.""" + if not isinstance(document, Mapping): + return [ + Issue("", f"a metrics file is a mapping at the top level, not {_describe(document)}") + ] + issues = _version_issues(document) + for error in _validator().iter_errors(document): + path = list(error.absolute_path) + if error.validator == "oneOf" and len(path) == 2 and path[0] == "metrics": + continue # said better by _kind_issues + issues.extend(_schema_error(error)) + metrics = _mapping(document.get("metrics")) + for name, metric in metrics.items(): + if isinstance(metric, Mapping): + issues.extend(_kind_issues(str(name), metric)) + issues.extend(_expression_issues(str(name), metric)) + issues.extend(_cycle_issues(metrics)) + return list(dict.fromkeys(issues)) + + +def _schema_error(error: ValidationError) -> list[Issue]: + if error.validator == "pattern": + for name, words in _PATTERN_WORDS.items(): + if schema()["$defs"][name]["pattern"] == error.validator_value: + where = format_path(error.absolute_path) + if error.relative_schema_path and "propertyNames" in error.relative_schema_path: + return [Issue(where, f"{_show(error.instance)} {words}")] + return [Issue(where, f"{words} (got {_show(error.instance)})")] + return _schema_issues(error) + + +def _version_issues(document: Mapping) -> list[Issue]: + version = document.get("model2data-metrics") + if not isinstance(version, str): + return [] # missing, or not a string: the schema says so + parts = version.split(".") + if len(parts) == 3 and all(part.isdigit() for part in parts): + if ".".join(parts[:2]) in READS: + return [] + return [ + Issue( + "model2data-metrics", + f"the file is written against metrics spec {version}, and this reader " + f"implements metrics spec {SPEC_VERSION} (0.1.x). Upgrade model2data to read it.", + ) + ] + return [] + + +def _kind_issues(name: str, metric: Mapping) -> list[Issue]: + where = f"metrics.{name}" + given = [key for key in KIND_KEYS if key in metric] + if not given: + return [ + Issue( + where, + "a metric needs one of `measure` (a column), `count` (a table's rows), " + "`ratio` or `expression`", + ) + ] + if len(given) > 1: + both = " and ".join(f"`{key}`" for key in given) + return [Issue(where, f"a metric is of one kind, but {both} are both given: keep one")] + kind = KIND_KEYS[given[0]] + issues = [] + if "agg" in metric and kind != "simple": + issues.append( + Issue(f"{where}.agg", "`agg` says how a `measure` aggregates; this metric has none") + ) + if kind in ("ratio", "derived"): + for key in ("time", "where"): + if key in metric: + issues.append( + Issue( + f"{where}.{key}", + f"`{key}` is for a metric with `measure` or `count`: a " + f"{'ratio' if kind == 'ratio' else 'derived metric'} is " + f"{'dated' if key == 'time' else 'filtered'} through the metrics it uses", + ) + ) + return issues + + +def _expression_issues(name: str, metric: Mapping) -> list[Issue]: + text = metric.get("expression") + if not isinstance(text, str) or not text: + return [] + try: + node = expr.parse(text) + except expr.ExpressionError as error: + return [Issue(f"metrics.{name}.expression", str(error))] + if not expr.names(node): + return [ + Issue( + f"metrics.{name}.expression", + "the expression uses no metric: a derived metric is arithmetic over metrics", + ) + ] + return [] + + +def dependencies(metric: Mapping) -> list[str]: + """The metrics a metric (as written in the file) is computed from.""" + ratio = metric.get("ratio") + if isinstance(ratio, Mapping): + return [ + value + for value in (ratio.get("numerator"), ratio.get("denominator")) + if isinstance(value, str) + ] + text = metric.get("expression") + if isinstance(text, str): + try: + return expr.names(expr.parse(text)) + except expr.ExpressionError: + return [] + return [] + + +def _cycle_issues(metrics: Mapping) -> list[Issue]: + """A metric that depends on itself, directly or through others: one issue per cycle.""" + graph = { + str(name): [dep for dep in dependencies(metric) if dep in metrics] + for name, metric in metrics.items() + if isinstance(metric, Mapping) + } + issues: list[Issue] = [] + reported: set[frozenset[str]] = set() + state: dict[str, int] = {} + + def visit(name: str, stack: list[str]) -> None: + state[name] = 1 + for dep in graph.get(name, []): + if state.get(dep) == 1: + cycle = [*stack[stack.index(dep) :], dep] + members = frozenset(cycle) + if members not in reported: + reported.add(members) + issues.append( + Issue( + f"metrics.{cycle[0]}", + "metrics depend on each other in a cycle: " + " -> ".join(cycle), + ) + ) + elif state.get(dep) is None: + visit(dep, [*stack, dep]) + state[name] = 2 + + for name in graph: + if state.get(name) is None: + visit(name, [name]) + return issues + + +# --------------------------------------------------------------------------- +# Against the model +# --------------------------------------------------------------------------- +def all_metrics(metrics: Metrics, model: Optional[Model]) -> dict[str, Metric]: + """The file's metrics and, with `infer`, the model's: explicit ones first, in file + order, then the inferred ones they do not replace, in model order.""" + out = dict(metrics.metrics) + if metrics.infer and model is not None: + for name, metric in inferred_metrics(model).items(): + out.setdefault(name, metric) + return out + + +def check_references(metrics: Metrics, known: Mapping[str, Metric]) -> list[Issue]: + """Every ratio and expression names metrics that exist.""" + issues = [] + for name, metric in metrics.metrics.items(): + if metric.ratio is not None: + for part in ("numerator", "denominator"): + target = getattr(metric.ratio, part) + if target not in known: + issues.append( + Issue(f"metrics.{name}.ratio.{part}", _unknown_metric(target, known)) + ) + elif metric.expression is not None: + for target in expr.names(expr.parse(metric.expression)): + if target not in known: + issues.append( + Issue(f"metrics.{name}.expression", _unknown_metric(target, known)) + ) + return issues + + +def _unknown_metric(name: str, known: Mapping[str, Metric]) -> str: + return f"there is no metric {name!r}{_did_you_mean(name, list(known))}" + + +def _did_you_mean(name: str, options: list[str]) -> str: + close = difflib.get_close_matches(name, options, n=1) + return f" (did you mean {close[0]!r}?)" if close else "" + + +def check_against( + metrics: Metrics, model: Model, *, model_name: Optional[str] = None +) -> list[Issue]: + """Every way `metrics` does not fit `model`: errors and warnings. + + `model_name` is the name the model goes by when it has no `name` of its own + (the CLI passes the file's stem). + """ + return _ModelChecks(metrics, model, model_name).run() + + +class _ModelChecks: + def __init__(self, metrics: Metrics, model: Model, model_name: Optional[str]): + self.metrics = metrics + self.model = model + self.model_name = model.name or model_name + self.graph = Graph(model) + self.issues: list[Issue] = [] + + def error(self, path: list[PathPart], message: str) -> None: + self.issues.append(Issue(format_path(path), message)) + + def warn(self, path: list[PathPart], message: str) -> None: + self.issues.append(Issue(format_path(path), message, severity="warning")) + + def run(self) -> list[Issue]: + if self.model_name is not None and self.metrics.model != self.model_name: + self.error( + ["model"], + f"the metrics are for model {self.metrics.model!r}, and the model is " + f"{self.model_name!r}", + ) + known = all_metrics(self.metrics, self.model) + self.issues.extend(check_references(self.metrics, known)) + for name, metric in self.metrics.metrics.items(): + self._metric(name, metric) + for column in self.metrics.dimensions: + self._column(["dimensions", column], column) + return self.issues + + # -- tables and columns -------------------------------------------------- + def _table(self, path: list[PathPart], key: str) -> bool: + if key in self.model.tables: + return True + self.error( + path, f"the model has no table {key!r}{_did_you_mean(key, list(self.model.tables))}" + ) + return False + + def _column(self, path: list[PathPart], column_path: str) -> Optional[Column]: + table_key, column = find_column(self.model, column_path) + if table_key is None: + key = column_path.rpartition(".")[0] + self.error( + path, + f"the model has no table {key!r}{_did_you_mean(key, list(self.model.tables))}", + ) + return None + if column is None: + name = column_path.rpartition(".")[2] + options = list(self.model.tables[table_key].columns) + self.error( + path, f"table {table_key!r} has no column {name!r}{_did_you_mean(name, options)}" + ) + return None + return column + + def _reach(self, path: list[PathPart], start: str, column_path: str, what: str) -> None: + target = column_path.rpartition(".")[0] + paths = self.graph.paths(start, target) + if not paths: + self.error( + path, + f"{what} {column_path} cannot be reached from {start}: a column has to be on " + f"the metric's table, or on one its foreign keys lead to (many-to-one)", + ) + elif len(paths) > 1: + self.error( + path, + f"{what} {column_path} is reached from {start} along two paths, so which " + f"{target} it means is unclear: {describe(paths[0])}; or {describe(paths[1])}", + ) + + # -- one metric ------------------------------------------------------------ + def _metric(self, name: str, metric: Metric) -> None: + base = ["metrics", name] + table: Optional[str] = None + if metric.kind == "simple" and metric.measure is not None: + column = self._column([*base, "measure"], metric.measure) + if column is not None: + table = metric.measure.rpartition(".")[0] + self._aggregation(base, metric, column) + elif metric.kind == "count" and metric.count is not None: + if self._table([*base, "count"], metric.count): + table = metric.count + if table is None: + return + if metric.time is not None: + column = self._column([*base, "time"], metric.time) + if column is not None: + if not is_temporal(column_kind(self.model, column)): + self.error( + [*base, "time"], + f"{metric.time} is {column.type}, not a date or timestamp column", + ) + else: + self._reach([*base, "time"], table, metric.time, "time column") + elif self.graph.first_time_column(table) is None: + self.warn( + base, + f"no date or timestamp column on {table}, or on a table it reaches, dates this " + "metric: it has a total but no value per month. Set `time` to date it", + ) + if metric.where is not None: + self._where([*base, "where"], table, metric.where) + + def _aggregation(self, base: list[PathPart], metric: Metric, column: Column) -> None: + assert metric.measure is not None + kind = column_kind(self.model, column) + if metric.agg is None: + implied = aggregation_of(column.measure) + if implied is None: + self.error( + [*base, "measure"], + f"{metric.measure} has no `measure` in the model, so say how it " + "aggregates with `agg` (sum, average, min, max, median, count, " + "count_distinct)", + ) + return + if metric.agg not in ANY_KIND_AGGREGATIONS and not is_numeric(kind): + self.error( + [*base, "agg"], + f"{metric.agg} needs a numeric column, and {metric.measure} is " + f"{column.type}: count and count_distinct apply to any column", + ) + + # -- filters ---------------------------------------------------------------- + def _where(self, path: list[PathPart], table: str, where: Where) -> None: + for term in where.terms: + if isinstance(term, Where): + group = "any" if term.any else "all" + for index, alternative in enumerate(term.terms): + assert isinstance(alternative, Where) + self._where([*path, group, index], table, alternative) + else: + self._condition(path, table, term) + + def _condition(self, path: list[PathPart], table: str, condition: Condition) -> None: + where = [*path, condition.column] + column = self._column(where, condition.column) + if column is None: + return + self._reach(where, table, condition.column, "filter column") + kind = column_kind(self.model, column) + for operator, operand in condition.operators.items(): + at = [*where, operator] if condition.written == "operators" else where + if operator in ORDER_OPERATORS and not (is_numeric(kind) or is_temporal(kind)): + self.error( + at, + f"`{operator}` compares by order, and {condition.column} is " + f"{'an enum' if kind == 'enum' else kind}: list the values it may hold " + "with `in` instead", + ) + continue + if operator == "is_null": + if operand is True and _never_null(column): + self.warn( + at, + f"can never match: {condition.column} is never null in the generated data", + ) + continue + values = operand if isinstance(operand, list) else [operand] + ok = True + for index, value in enumerate(values): + message = _value_problem(self.model, column, kind, value) + if message is not None: + ok = False + value_at = [*at, index] if isinstance(operand, list) else at + self.error(value_at, message) + if operator == "between" and ok and len(values) == 2: + low, high = (_comparable(kind, value) for value in values) + if low is not None and high is not None and low > high: + self.warn( + at, f"can never match: {_show(values[0])} is above {_show(values[1])}" + ) + continue + if ok and is_numeric(kind): + self._never_matches(at, condition.column, column, operator, operand) + + def _never_matches( + self, at: list[PathPart], name: str, column: Column, operator: str, operand: Any + ) -> None: + low, high = column.generate.get("min"), column.generate.get("max") + low = low if _number(low) else None + high = high if _number(high) else None + if low is None and high is None: + return + bounds = ( + f"between {low} and {high}" + if low is not None and high is not None + else (f"at least {low}" if low is not None else f"at most {high}") + ) + + def outside(value: Any) -> bool: + return (low is not None and value < low) or (high is not None and value > high) + + never = False + if operator == "eq": + never = outside(operand) + elif operator == "in": + never = all(outside(value) for value in operand) + elif operator == "gt": + never = high is not None and operand >= high + elif operator == "gte": + never = high is not None and operand > high + elif operator == "lt": + never = low is not None and operand <= low + elif operator == "lte": + never = low is not None and operand < low + elif operator == "between": + never = (high is not None and operand[0] > high) or ( + low is not None and operand[1] < low + ) + if never: + self.warn(at, f"can never match: {name} is generated {bounds}") + + +def _number(value: Any) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) + + +def _never_null(column: Column) -> bool: + return column.pk or column.not_null + + +def _parse_temporal(kind: ColumnKind, value: str) -> Optional[datetime]: + """`value` as an ISO 8601 date (or, for a timestamp column, timestamp), else None.""" + try: + if kind == "date" or len(value) == 10: + if len(value) != 10: + return None + day = date.fromisoformat(value) + return datetime(day.year, day.month, day.day) + parsed = datetime.fromisoformat(value) + except ValueError: + return None + return parsed if parsed.tzinfo is None else None + + +def _comparable(kind: ColumnKind, value: Any) -> Any: + if is_temporal(kind) and isinstance(value, str): + return _parse_temporal(kind, value) + return value if _number(value) else None + + +def _value_problem(model: Model, column: Column, kind: ColumnKind, value: Any) -> Optional[str]: + """Why `value` cannot be a value of `column`, or None when it can.""" + if kind == "enum": + enum = model.enum_for(column.type) + assert enum is not None + text = str(value) if isinstance(value, int) and not isinstance(value, bool) else value + if isinstance(text, str) and text in enum.members: + return None + members = ", ".join(enum.members) + hint = _did_you_mean(text, enum.members) if isinstance(text, str) else "" + return f"{_show(value)} is not a member of {column.type} ({members}){hint}" + if kind == "boolean": + return None if isinstance(value, bool) else f"must be true or false, not {_show(value)}" + if is_numeric(kind): + if not _number(value): + return f"must be a number, as {column.type} is numeric, not {_describe(value)}" + if kind == "integer" and isinstance(value, float) and not value.is_integer(): + return f"must be a whole number, as {column.type} is an integer type (got {value!r})" + return None + if is_temporal(kind): + if isinstance(value, str) and _parse_temporal(kind, value) is not None: + return None + shape = "YYYY-MM-DD" if kind == "date" else "YYYY-MM-DD or YYYY-MM-DD HH:MM:SS" + return f"must be an ISO 8601 {kind} as a string, {shape} (got {_show(value)})" + if not isinstance(value, str): + return f"must be text, as {column.type} is a text column: quote it (got {_show(value)})" + return None diff --git a/model2data/metrics/columns.py b/model2data/metrics/columns.py new file mode 100644 index 0000000..bcb5933 --- /dev/null +++ b/model2data/metrics/columns.py @@ -0,0 +1,66 @@ +"""What kind of values a model's column holds, as the metrics spec reads it. + +The kinds are the model spec's ("Generation hints"), answered by +`model2data.generate.kinds` so the two specs can never disagree: an enum is an +enum whatever its name contains, and is never numeric, boolean or temporal. +Within numeric, an integer and a decimal are told apart because they are cast +differently in SQL; within temporal, a date and a timestamp, for the same +reason. +""" + +from __future__ import annotations + +from typing import Literal, Optional + +from model2data.generate import kinds +from model2data.model.types import Column, Model + +ColumnKind = Literal["enum", "boolean", "integer", "decimal", "date", "timestamp", "text"] + + +def column_kind(model: Model, column: Column) -> ColumnKind: + if model.enum_for(column.type) is not None: + return "enum" + if kinds.is_boolean_type(column.type): + return "boolean" + if kinds.is_integer_type(column.type): + return "integer" + if kinds.is_decimal_type(column.type): + return "decimal" + temporal = kinds.temporal_kind(column.type) + if temporal is not None: + return temporal + return "text" + + +def is_numeric(kind: ColumnKind) -> bool: + return kind in ("integer", "decimal") + + +def is_temporal(kind: ColumnKind) -> bool: + return kind in ("date", "timestamp") + + +def find_column(model: Model, path: str) -> tuple[Optional[str], Optional[Column]]: + """`(table key, column)` a column path names; the column is None when the table has + no such column, and both are None when no table matches.""" + table_key, _, name = path.rpartition(".") + table = model.tables.get(table_key) + if table is None: + return None, None + return table_key, table.columns.get(name) + + +# The SQL type each kind is read as, in the known values and in the dbt tests. +# DuckDB and Postgres both read every one of them. A decimal is an exact +# decimal rather than a double, so a sum is the same whatever order the rows are +# added in -- which is what keeps a known value byte-identical run after run. +SQL_TYPES: dict[str, str] = { + "enum": "varchar", + "text": "varchar", + "boolean": "boolean", + "integer": "bigint", + "decimal": "decimal(38, 9)", + "date": "date", + "timestamp": "timestamp", +} diff --git a/model2data/metrics/dbt.py b/model2data/metrics/dbt.py new file mode 100644 index 0000000..9b15396 --- /dev/null +++ b/model2data/metrics/dbt.py @@ -0,0 +1,86 @@ +"""One singular dbt test per metric: its value over the project's models is its known value. + +`write_metric_tests` writes `data-tests/metrics/metric_.sql` into a +generated project (`data-tests` is the project's `test-paths`). Each test +computes the metric over the staging models (`stg_
`), with the same SQL +the known values were computed with (`model2data.metrics.sql`), and returns a +row -- failing -- unless the result is within the tolerance of the value in +`metric_values.json`. So `dbt build` proves that the metric's logic, run by +the warehouse over the project's own models, gives the number the run says +it should. + +The tolerance is `0.000001 + 0.000000001 * |known value|`: the known value is +rounded to 6 decimal places, and a warehouse may add a column of doubles in +another order than DuckDB did. A known value of null passes only when the +metric is null too. +""" + +from __future__ import annotations + +from decimal import Decimal +from pathlib import Path +from typing import Optional + +from model2data.metrics.semantic import SemanticModel +from model2data.metrics.sql import scalar +from model2data.metrics.values import KnownValues, Number + +ABSOLUTE_TOLERANCE = Decimal("0.000001") +RELATIVE_TOLERANCE = Decimal("0.000000001") +TEST_DIRECTORY = Path("data-tests") / "metrics" + + +def tolerance(value: Number) -> Decimal: + return ABSOLUTE_TOLERANCE + RELATIVE_TOLERANCE * abs(Decimal(value)) + + +def _number(value: Number) -> str: + return format(Decimal(value), "f") + + +def metric_test_sql(semantic: SemanticModel, name: str, expected: Optional[Number]) -> str: + """The singular test for metric `name`, given its known value.""" + metric = semantic.metrics[name] + + def relation(table: str) -> str: + return f"{{{{ ref('stg_{semantic.entities[table].name}') }}}}" + + value_sql = scalar(semantic, name, relation).replace("\n", "\n ") + label = metric.label.replace("\n", " ") + if expected is None: + header = ( + f"-- model2data metric `{name}` ({label}): its known value in metric_values.json is\n" + "-- null (no row to aggregate, or a division by zero), so it must be null here too.\n" + ) + check = "value is not null" + else: + slack = tolerance(expected) + header = ( + f"-- model2data metric `{name}` ({label}): its value over the staging models must be\n" + f"-- its known value in metric_values.json, {_number(expected)}, within " + f"{_number(slack)}.\n" + ) + check = f"value is null or abs(value - {_number(expected)}) > {_number(slack)}" + return ( + f"{header}" + "with actual as (\n" + f" select {value_sql} as value\n" + ")\n" + "select value as actual_value\n" + "from actual\n" + f"where {check}\n" + ) + + +def write_metric_tests(dest: Path, semantic: SemanticModel, values: KnownValues) -> list[Path]: + """One test per metric under `dest/data-tests/metrics/`, in metric order; the paths.""" + directory = dest / TEST_DIRECTORY + directory.mkdir(parents=True, exist_ok=True) + written = [] + for name in semantic.metrics: + path = directory / f"metric_{name}.sql" + path.write_text( + metric_test_sql(semantic, name, values.values[name].value), encoding="utf-8" + ) + written.append(path) + return written diff --git a/model2data/metrics/expression.py b/model2data/metrics/expression.py new file mode 100644 index 0000000..e8688c7 --- /dev/null +++ b/model2data/metrics/expression.py @@ -0,0 +1,209 @@ +"""A derived metric's `expression`: arithmetic over metric names and numbers. + +This is the one place the metrics spec encodes something in a string, so it is +kept as small as a grammar can be and read with a parser of its own, never +`eval`: + + expression := term (("+" | "-") term)* + term := factor (("*" | "/") factor)* + factor := ("+" | "-") factor | NUMBER | NAME | "(" expression ")" + NAME := [a-z][a-z0-9_]* a metric + NUMBER := [0-9]+ ("." [0-9]+)? + +Whitespace between tokens is ignored. Operators keep the usual precedence and +associate to the left (`a - b - c` is `(a - b) - c`). Division by zero is +null, as a ratio's is, and so is any arithmetic with a null in it. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation +from typing import Optional, Union + +_TOKEN = re.compile( + r"\s*(?:(?P[0-9]+(?:\.[0-9]+)?)|(?P[a-z][a-z0-9_]*)|(?P[-+*/()]))" +) + + +class ExpressionError(ValueError): + """An expression that does not parse; the message says where.""" + + +@dataclass(frozen=True) +class Number: + value: Decimal + + +@dataclass(frozen=True) +class Name: + name: str + + +@dataclass(frozen=True) +class Negate: + operand: Node + + +@dataclass(frozen=True) +class BinaryOp: + op: str + left: Node + right: Node + + +Node = Union[Number, Name, Negate, BinaryOp] + + +def _tokens(text: str) -> list[tuple[str, str, int]]: + """`(kind, text, position)` for each token; position is 1-based, for messages.""" + out: list[tuple[str, str, int]] = [] + position = 0 + while position < len(text): + if text[position:].strip() == "": + break + match = _TOKEN.match(text, position) + if match is None or match.end() == position: + offset = len(text[position:]) - len(text[position:].lstrip()) + char = text[position + offset] + raise ExpressionError( + f"unexpected {char!r} at character {position + offset + 1}: an expression " + "holds metric names, numbers, + - * / and parentheses" + ) + kind = match.lastgroup or "op" + out.append((kind, match.group(kind), match.start(kind) + 1)) + position = match.end() + return out + + +class _Parser: + def __init__(self, text: str): + self.tokens = _tokens(text) + self.index = 0 + + def peek(self) -> Optional[tuple[str, str, int]]: + return self.tokens[self.index] if self.index < len(self.tokens) else None + + def take(self) -> tuple[str, str, int]: + token = self.tokens[self.index] + self.index += 1 + return token + + def expression(self) -> Node: + node = self.term() + while (token := self.peek()) is not None and token[1] in ("+", "-"): + self.take() + node = BinaryOp(token[1], node, self.term()) + return node + + def term(self) -> Node: + node = self.factor() + while (token := self.peek()) is not None and token[1] in ("*", "/"): + self.take() + node = BinaryOp(token[1], node, self.factor()) + return node + + def factor(self) -> Node: + token = self.peek() + if token is None: + raise ExpressionError("the expression ends where a metric name or number should be") + kind, text, position = self.take() + if text in ("+", "-"): + operand = self.factor() + return Negate(operand) if text == "-" else operand + if kind == "number": + return Number(Decimal(text)) + if kind == "name": + return Name(text) + if text == "(": + node = self.expression() + closing = self.peek() + if closing is None or closing[1] != ")": + raise ExpressionError(f"the parenthesis at character {position} is never closed") + self.take() + return node + raise ExpressionError( + f"unexpected {text!r} at character {position}: a metric name or number goes here" + ) + + +def parse(text: str) -> Node: + """The expression's tree, or `ExpressionError` saying what is wrong and where.""" + parser = _Parser(text) + if not parser.tokens: + raise ExpressionError("the expression is empty") + node = parser.expression() + rest = parser.peek() + if rest is not None: + raise ExpressionError( + f"unexpected {rest[1]!r} at character {rest[2]}: an operator goes between " + "two metric names or numbers" + ) + return node + + +def names(node: Node) -> list[str]: + """The metric names an expression uses, each once, in the order they appear.""" + found: dict[str, None] = {} + + def walk(current: Node) -> None: + if isinstance(current, Name): + found[current.name] = None + elif isinstance(current, Negate): + walk(current.operand) + elif isinstance(current, BinaryOp): + walk(current.left) + walk(current.right) + + walk(node) + return list(found) + + +def evaluate(node: Node, values: Mapping[str, Optional[Decimal]]) -> Optional[Decimal]: + """The expression's value given each metric's, with SQL's nulls: null in, null out, + and division by zero is null.""" + if isinstance(node, Number): + return node.value + if isinstance(node, Name): + return values[node.name] + if isinstance(node, Negate): + operand = evaluate(node.operand, values) + return None if operand is None else -operand + left = evaluate(node.left, values) + right = evaluate(node.right, values) + if left is None or right is None: + return None + if node.op == "+": + return left + right + if node.op == "-": + return left - right + if node.op == "*": + return left * right + if right == 0: + return None + try: + return left / right + except InvalidOperation: # pragma: no cover - 0/0 is caught above + return None + + +def to_sql(node: Node, render_name: Callable[[str], str]) -> str: + """The expression as SQL, each metric name replaced by `render_name(name)`. + + Every operation is parenthesised, so the SQL means what the tree means + whatever the reader's precedence rules, and a division divides by + `nullif(..., 0)`, so dividing by zero is null rather than an error. + """ + if isinstance(node, Number): + return format(node.value, "f") + if isinstance(node, Name): + return render_name(node.name) + if isinstance(node, Negate): + return f"(-{to_sql(node.operand, render_name)})" + left = to_sql(node.left, render_name) + right = to_sql(node.right, render_name) + if node.op == "/": + return f"({left} / nullif({right}, 0))" + return f"({left} {node.op} {right})" diff --git a/model2data/metrics/graph.py b/model2data/metrics/graph.py new file mode 100644 index 0000000..586445d --- /dev/null +++ b/model2data/metrics/graph.py @@ -0,0 +1,151 @@ +"""How a metric's table reaches the columns its filters and time name: many-to-one joins. + +A metric is aggregated over the rows of one table. A filter or a time column on +another table is reached by following foreign keys from child to parent -- +`order_items.order_id -> orders.id -> orders.customer_id -> customers.id` -- +because each step joins a row to at most one parent row, so joining never +duplicates the rows being aggregated. Going the other way (parent to children) +would, so it is never taken. + +A table is reachable when exactly one path of such steps leads to it. Two +paths (`orders.billed_to` and `orders.shipped_to` both to `customers`) leave it +unclear which customer a filter means, which the metrics spec makes an error +rather than a guess. +""" + +from __future__ import annotations + +from collections import deque +from dataclasses import dataclass + +from model2data.generate import kinds +from model2data.model.types import Model + +Path = tuple["Join", ...] + + +@dataclass(frozen=True) +class Join: + """One many-to-one step: rows of `child` to the row of `parent` their key names.""" + + child: str + child_columns: tuple[str, ...] + parent: str + parent_columns: tuple[str, ...] + + def __str__(self) -> str: + left = ", ".join(f"{self.child}.{column}" for column in self.child_columns) + right = ", ".join(f"{self.parent}.{column}" for column in self.parent_columns) + return f"{left} -> {right}" + + +def describe(path: Path) -> str: + """A path as `orders.customer_id -> customers.id`, steps joined by `, then `.""" + return ", then ".join(str(step) for step in path) if path else "the table itself" + + +class Graph: + """The many-to-one steps of a model, child to parent, in document order. + + A column's `references` comes before the table's `foreign_keys`, columns in + document order, so the first path found is the one a reader of the model + would find first. + """ + + def __init__(self, model: Model): + self.model = model + self.steps: dict[str, list[Join]] = {key: [] for key in model.tables} + for key, table in model.tables.items(): + for name, column in table.columns.items(): + if column.references is not None and column.references.table in model.tables: + self.steps[key].append( + Join( + key, + (name,), + column.references.table, + (column.references.column,), + ) + ) + for fk in table.foreign_keys: + if fk.references in model.tables: + self.steps[key].append( + Join(key, tuple(fk.columns), fk.references, tuple(fk.to_columns)) + ) + + def _reaching(self, target: str) -> set[str]: + """Every table with a path to `target`, `target` included.""" + parents_of: dict[str, list[str]] = {} + for steps in self.steps.values(): + for step in steps: + parents_of.setdefault(step.parent, []).append(step.child) + seen = {target} + queue = deque([target]) + while queue: + for child in parents_of.get(queue.popleft(), []): + if child not in seen: + seen.add(child) + queue.append(child) + return seen + + def paths(self, start: str, target: str, limit: int = 2) -> list[Path]: + """Up to `limit` paths from `start` to `target`, each visiting a table once. + + The empty path when they are the same table. Only tables that can reach + `target` are entered, so the search never wanders, and it stops at + `limit` (two is enough to know a path is not the only one). + """ + if start == target: + return [()] + reaching = self._reaching(target) + if start not in reaching: + return [] + found: list[Path] = [] + + def walk(table: str, path: list[Join], visited: set[str]) -> None: + for step in self.steps.get(table, []): + if len(found) >= limit: + return + if step.parent in visited or step.parent not in reaching: + continue + if step.parent == target: + found.append((*path, step)) + continue + walk(step.parent, [*path, step], visited | {step.parent}) + + walk(start, [], {start}) + return found + + def first_time_column(self, start: str) -> tuple[str, Path] | None: + """`(column path, join path)` of the default time of a metric on `start`. + + The table's own first date or timestamp column; else, nearest first and + in document order, the first date or timestamp column of a table it + reaches along exactly one path. None when there is no such column. + """ + own = self.temporal_columns(start) + if own: + return f"{start}.{own[0]}", () + seen = {start} + queue = deque([start]) + while queue: + table = queue.popleft() + for step in self.steps.get(table, []): + if step.parent in seen: + continue + seen.add(step.parent) + queue.append(step.parent) + columns = self.temporal_columns(step.parent) + if not columns: + continue + paths = self.paths(start, step.parent) + if len(paths) == 1: + return f"{step.parent}.{columns[0]}", paths[0] + return None + + def temporal_columns(self, table: str) -> list[str]: + """The date and timestamp columns of `table`, in document order.""" + found = [] + for name, column in self.model.tables[table].columns.items(): + if self.model.enum_for(column.type) is None and kinds.is_temporal_type(column.type): + found.append(name) + return found diff --git a/model2data/metrics/infer.py b/model2data/metrics/infer.py new file mode 100644 index 0000000..231e5c9 --- /dev/null +++ b/model2data/metrics/infer.py @@ -0,0 +1,113 @@ +"""The metrics a model implies on its own, without a metrics file. + +They are named the way model2data studio's semantic-layer export names its +metrics (`api/modelling/semantic.py` there), so the studio can move onto this +module without renaming a metric anyone already uses: + +- one simple metric for each column with a `measure`, named + `
_`, with `_` appended when it is not `sum` + (`orders_total_amount`, `order_items_unit_price_average`); +- one row count for each fact, named `
_count`. A fact is a table whose + `role` is `fact`, or which has no `role` and at least one measure; the + studio's classifier decides the second case from the data too, which the + engine does not have here, and a table with a measure is the case it decides + as a fact. + +A name is made of lower-case letters, digits and single underscores, starts +with a letter (else `c_` is put in front) and is at least two characters; a +name already handed out gets `_2`, `_3`. Columns with `measure: false`, or no +`measure`, imply nothing: the studio also infers measures for facts from the +data, which the engine leaves to the metrics file. +""" + +from __future__ import annotations + +import re + +from model2data.metrics.types import Metric +from model2data.model.types import Model + +_AGG_WORDS = { + "sum": "Sum", + "average": "Average", + "min": "Smallest", + "max": "Largest", + "median": "Median", + "count": "Count", + "count_distinct": "Distinct count", +} + + +def metric_name(text: str) -> str: + """`text` as a metric name: lower case, digits and single underscores, starting with a + letter, at least two characters, not ending in an underscore.""" + name = re.sub(r"_+", "_", re.sub(r"[^a-z0-9_]+", "_", text.lower())).strip("_") + if not name or not name[0].isalpha(): + name = f"c_{name}" if name else "unnamed" + return name if len(name) >= 2 else f"{name}_x" + + +class _Names: + """Names handed out once each, the second taker of a name getting `_2`.""" + + def __init__(self) -> None: + self.taken: set[str] = set() + + def take(self, text: str) -> str: + base = metric_name(text) + name, suffix = base, 1 + while name in self.taken: + suffix += 1 + name = f"{base}_{suffix}" + self.taken.add(name) + return name + + +def _label(text: str) -> str: + words = text.replace("_", " ").strip() + return words[:1].upper() + words[1:] + + +def aggregation_of(measure: object) -> str | None: + """How a column's `measure` aggregates: `True` is `sum`; None when it is no measure.""" + if measure is True: + return "sum" + if isinstance(measure, str): + return measure + return None + + +def inferred_metrics(model: Model) -> dict[str, Metric]: + """Every metric the model implies, by name, tables and columns in document order.""" + names = _Names() + out: dict[str, Metric] = {} + for key, table in model.tables.items(): + measured = False + for column_name, column in table.columns.items(): + aggregation = aggregation_of(column.measure) + if aggregation is None: + continue + measured = True + suffix = "" if aggregation == "sum" else f"_{aggregation}" + name = names.take(f"{key}_{column_name}{suffix}") + label = f"{_label(key)} {column_name.replace('_', ' ')}" + if aggregation != "sum": + label += f" ({_AGG_WORDS[aggregation].lower()})" + out[name] = Metric( + kind="simple", + measure=f"{key}.{column_name}", + agg=aggregation, + label=label, + description=f"{_AGG_WORDS[aggregation]} of {key}.{column_name}.", + inferred=True, + ) + if table.role == "fact" or (table.role is None and measured): + name = names.take(f"{key}_count") + out[name] = Metric( + kind="count", + count=key, + label=f"{_label(key)} count", + description=f"Rows of {key}.", + inferred=True, + ) + return out diff --git a/model2data/metrics/ossie.py b/model2data/metrics/ossie.py new file mode 100644 index 0000000..66b2398 --- /dev/null +++ b/model2data/metrics/ossie.py @@ -0,0 +1,310 @@ +"""A semantic model as an Apache Ossie (Open Semantic Interchange) 0.1.1 document. + +Ossie 0.1.1 is read from github.com/apache/ossie at the `osi-0.1.1-rc1` tag +(`core-spec/spec.md` and `osi-schema.json`; its `version.txt` says 0.1.1): no +final 0.1.1 tag exists yet. 0.2.0 is an unreleased draft that changes the +format, so this module does not write it. The mapping: + +| model2data | Ossie 0.1.1 | +|---|---| +| the model | one `semantic_model`, named after it, with its description | +| a table | a `dataset` named by its dbt name; `source` the staging model the generated dbt project builds (`staging.stg_` by default) | +| its primary key, `unique` columns and `unique` keys | `primary_key`, `unique_keys` | +| a column | a `field` whose expression is the column; a dimension gets `dimension: {is_time: ...}` and its label | +| a foreign key | a `relationship` from the many side to the one side, `from_columns` paired with `to_columns` | +| a metric | a model-level `metric` with one `ANSI_SQL` expression over `dataset.field` | +| `description`, `ai_context` | `description`, `ai_context` (a string) | + +A metric's filter becomes `CASE WHEN THEN END` inside its +aggregate, a ratio `() / NULLIF(, 0)`, a derived +metric its expression with each input written out in full, and a row count +`COUNT()`, so the expression names the dataset whose rows it +counts. + +What Ossie 0.1.1 has no place for -- a metric's `label`, `format` and time +column, a table's `role` and `grain`, an enum's members, a one-to-one +relationship -- is carried in a `custom_extensions` entry of vendor `COMMON`, +whose `data` is a JSON object under the key `model2data`, and listed in the +export's lossiness report, as is anything a reader could take another way (a +filter that reaches its table through another, since Ossie leaves the join +path to the reader, or a row count of a table without a one-column key). +""" + +from __future__ import annotations + +import json +import textwrap +from collections.abc import Callable +from dataclasses import dataclass, field +from typing import Any, Optional + +import yaml + +from model2data.metrics.semantic import Entity, SemanticModel +from model2data.metrics.sql import ansi, identifier + +OSSIE_VERSION = "0.1.1" + + +def default_source(entity: Entity) -> str: + """The relation a dataset reads: the staging model of the generated dbt project.""" + return f"staging.stg_{entity.name}" + + +@dataclass(frozen=True) +class Loss: + """Something the export could not say in Ossie 0.1.1 itself. + + `subject` is what it is about (`metric revenue`, `dataset orders`, `field + orders.status`), `item` which part (`label`, `time`, ...), and `reason` what + happened to it. + """ + + subject: str + item: str + reason: str + + def __str__(self) -> str: + return f"{self.subject}: {self.item}: {self.reason}" + + +@dataclass +class OssieExport: + document: dict[str, Any] + lossiness: list[Loss] = field(default_factory=list) + + def to_yaml(self) -> str: + """The document as YAML, the lossiness report as comments above it.""" + name = self.document["semantic_model"][0]["name"] + lines = [ + f"# Apache Ossie (Open Semantic Interchange) {OSSIE_VERSION}: the semantic model of " + f"{name},", + "# written by model2data from the model and its metrics.", + ] + if self.lossiness: + lines.append("#") + lines.append("# Not expressible in Ossie 0.1.1 itself (see custom_extensions):") + for loss in self.lossiness: + lines.extend( + textwrap.wrap( + str(loss), + width=100, + initial_indent="# - ", + subsequent_indent="# ", + break_on_hyphens=False, + ) + ) + body = yaml.dump( + self.document, + Dumper=_Dumper, + sort_keys=False, + allow_unicode=True, + default_flow_style=False, + # One line per expression, however long: a folded SQL expression is + # valid YAML but hard to read and to diff. + width=1_000_000, + ) + return "\n".join(lines) + "\n" + body + + +class _Quoted(str): + """A string written double-quoted: the version, which must read back as text.""" + + +class _Dumper(yaml.SafeDumper): + pass + + +def _quoted(dumper: yaml.SafeDumper, value: _Quoted) -> yaml.Node: + return dumper.represent_scalar("tag:yaml.org,2002:str", str(value), style='"') + + +def _flow_list(dumper: yaml.SafeDumper, value: _FlowList) -> yaml.Node: + return dumper.represent_sequence("tag:yaml.org,2002:seq", list(value), flow_style=True) + + +class _FlowList(list): + """A key's columns, written `[a, b]` as the Ossie examples write them.""" + + +_Dumper.add_representer(_Quoted, _quoted) +_Dumper.add_representer(_FlowList, _flow_list) + + +def _expression(text: str) -> dict[str, Any]: + return {"dialects": [{"dialect": "ANSI_SQL", "expression": text}]} + + +def _extension(data: dict[str, Any]) -> list[dict[str, str]]: + return [ + { + "vendor_name": "COMMON", + "data": json.dumps({"model2data": data}, ensure_ascii=False), + } + ] + + +def to_ossie( + semantic: SemanticModel, *, source: Optional[Callable[[Entity], str]] = None +) -> OssieExport: + """The Ossie 0.1.1 document of `semantic`, and what it could not express. + + `source` names the relation each dataset reads; by default the staging model + of the dbt project `model2data generate` writes, `staging.stg_`. + """ + source = source or default_source + losses: list[Loss] = [] + model: dict[str, Any] = {"name": semantic.name} + if semantic.description: + model["description"] = semantic.description.strip() + enum_fields: list[str] = [] + model["datasets"] = [ + _dataset(semantic, entity, source, losses, enum_fields) + for entity in semantic.entities.values() + ] + if enum_fields: + losses.append( + Loss( + f"{'field' if len(enum_fields) == 1 else 'fields'} {', '.join(enum_fields)}", + "values", + "Ossie fields have no list of allowed values; the enum's members are carried " + "in custom_extensions", + ) + ) + relationships = [] + for relationship in semantic.relationships: + entry: dict[str, Any] = { + "name": relationship.name, + "from": semantic.entities[relationship.from_table].name, + "to": semantic.entities[relationship.to_table].name, + "from_columns": _FlowList(relationship.from_columns), + "to_columns": _FlowList(relationship.to_columns), + } + if relationship.one_to_one: + entry["custom_extensions"] = _extension({"one_to_one": True}) + losses.append( + Loss( + f"relationship {relationship.name}", + "one_to_one", + "Ossie relationships are many-to-one; that it is one-to-one is in " + "custom_extensions", + ) + ) + relationships.append(entry) + if relationships: + model["relationships"] = relationships + carried: dict[str, list[str]] = {} + metrics = [_metric(semantic, name, losses, carried) for name in semantic.metrics] + if metrics: + model["metrics"] = metrics + for item, reason in _CARRIED.items(): + if carried.get(item): + subject = "metric" if len(carried[item]) == 1 else "metrics" + losses.append( + Loss( + f"{subject} {', '.join(carried[item])}", + item, + f"{reason}; carried in custom_extensions", + ) + ) + document = {"version": _Quoted(OSSIE_VERSION), "semantic_model": [model]} + return OssieExport(document, losses) + + +def _dataset( + semantic: SemanticModel, + entity: Entity, + source: Callable[[Entity], str], + losses: list[Loss], + enum_fields: list[str], +) -> dict[str, Any]: + table = semantic.model.tables[entity.table] + dataset: dict[str, Any] = {"name": entity.name, "source": source(entity)} + if entity.primary_key: + dataset["primary_key"] = _FlowList(entity.primary_key) + if entity.unique_keys: + dataset["unique_keys"] = [_FlowList(key) for key in entity.unique_keys] + if entity.description: + dataset["description"] = entity.description.strip() + dimensions = {d.column: d for d in semantic.dimensions if d.table == entity.table} + fields = [] + for name, column in table.columns.items(): + item: dict[str, Any] = {"name": name, "expression": _expression(identifier(name))} + dimension = dimensions.get(name) + if dimension is not None: + item["dimension"] = {"is_time": dimension.kind == "time"} + item["label"] = dimension.label + description = dimension.description if dimension is not None else column.description + if description: + item["description"] = description.strip() + if dimension is not None and dimension.members is not None: + item["custom_extensions"] = _extension({"values": list(dimension.members)}) + enum_fields.append(f"{entity.name}.{name}") + fields.append(item) + if fields: + dataset["fields"] = fields + extra: dict[str, Any] = {} + if entity.role: + extra["role"] = entity.role + if entity.grain: + extra["grain"] = list(entity.grain) + if extra: + dataset["custom_extensions"] = _extension(extra) + losses.append( + Loss( + f"dataset {entity.name}", + " and ".join(extra), + "Ossie datasets have no " + " or ".join(extra) + "; carried in custom_extensions", + ) + ) + return dataset + + +def _metric( + semantic: SemanticModel, name: str, losses: list[Loss], carried: dict[str, list[str]] +) -> dict[str, Any]: + metric = semantic.metrics[name] + entry: dict[str, Any] = {"name": name, "expression": _expression(ansi(semantic, name))} + if metric.description: + entry["description"] = metric.description.strip() + if metric.ai_context: + entry["ai_context"] = metric.ai_context.strip() + extra: dict[str, Any] = {"label": metric.label} + if metric.format: + extra["format"] = metric.format + if metric.time: + extra["time"] = metric.time + for item in extra: + carried.setdefault(item, []).append(name) + subject = f"metric {name}" + for table, path in metric.joins.items(): + if len(path) > 1: + via = ", ".join(semantic.entities[step.parent].name for step in path[:-1]) + losses.append( + Loss( + subject, + "join path", + f"reaches {semantic.entities[table].name} through {via}; Ossie leaves the " + "join path to the reader, which finds it from the relationships", + ) + ) + if metric.kind == "count" and metric.table is not None: + if len(semantic.entities[metric.table].primary_key) != 1: + losses.append( + Loss( + subject, + "row count", + f"{semantic.entities[metric.table].name} has no one-column primary key, so " + "COUNT(*) names no dataset: a reader cannot tell whose rows it counts", + ) + ) + entry["custom_extensions"] = _extension(extra) + return entry + + +# What an Ossie metric has no place for, as the lossiness report says it. +_CARRIED = { + "label": "Ossie metrics have no label", + "format": "Ossie metrics have no format (`percent` is a fraction: 0.6 is shown as 60%)", + "time": "Ossie metrics name no time dimension (the column is marked is_time on its dataset)", +} diff --git a/model2data/metrics/reader.py b/model2data/metrics/reader.py new file mode 100644 index 0000000..e11ffc2 --- /dev/null +++ b/model2data/metrics/reader.py @@ -0,0 +1,243 @@ +"""Reading a metrics file from a file or from text, and checking it against its model.""" + +from __future__ import annotations + +import copy +import json +import os +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any, Optional, Union + +from model2data.metrics.check import ( + all_metrics, + check_against, + check_document, + check_references, +) +from model2data.metrics.types import ( + KIND_KEYS, + Condition, + DimensionOverride, + Metric, + Metrics, + Ratio, + Where, +) +from model2data.model._yaml import locate_issues, parse_yaml +from model2data.model.errors import Issue, ModelError +from model2data.model.reader import read_text +from model2data.model.types import Model + +Source = Union[str, "os.PathLike[str]"] + +# The suffixes a metrics file is recognised by, beside `.model2data.yml`. +SUFFIXES = (".metrics.yml", ".metrics.yaml", ".metrics.json") + + +class MetricsError(ModelError): + """A metrics file that does not conform, with every issue found in it. + + A `ModelError`, so code that reports a model's issues reports these the same way. + """ + + def __init__(self, issues: Sequence[Issue], warnings: Sequence[Issue] = ()): + super().__init__(issues, warnings) + count = len(self.issues) + heading = "1 issue" if count == 1 else f"{count} issues" + lines = "\n".join(f" - {issue}" for issue in self.issues) + ValueError.__init__(self, f"The metrics file has {heading}:\n{lines}") + + +def is_metrics_file(path: Union[str, Path]) -> bool: + """Whether a file's name says it is a metrics file (`*.metrics.yml`, `.yaml`, `.json`).""" + return str(path).lower().endswith(SUFFIXES) + + +def metrics_stem(path: Path) -> str: + """`coffee.metrics.yml` -> `coffee`.""" + name = path.name + for suffix in SUFFIXES: + if name.lower().endswith(suffix): + return name[: -len(suffix)] + return path.stem + + +def sibling_model(path: Path) -> Optional[Path]: + """The model beside a metrics file, by stem: `coffee.model2data.yml` (or `.yaml`, + `.json`, or `coffee.dbml`) next to `coffee.metrics.yml`; None when there is none.""" + stem = metrics_stem(path) + for suffix in (".model2data.yml", ".model2data.yaml", ".model2data.json", ".dbml"): + candidate = path.with_name(stem + suffix) + if candidate.is_file(): + return candidate + return None + + +def _as_path(source: Source) -> Optional[Path]: + if isinstance(source, os.PathLike): + return Path(source) + if "\n" in source or len(source) > 4096: + return None + path = Path(source) + if path.suffix.lower() in (".yml", ".yaml", ".json") and path.is_file(): + return path + return None + + +def _parse(source: Source) -> tuple[str, Any]: + path = _as_path(source) + try: + if path is not None: + text = read_text(path) + is_json = path.suffix.lower() == ".json" + else: + text = str(source) + is_json = text.lstrip().startswith("{") + if is_json: + try: + return text, json.loads(text) + except json.JSONDecodeError as error: + raise ModelError( + [Issue("", f"not valid JSON: {error.msg} (column {error.colno})", error.lineno)] + ) from None + return text, parse_yaml(text) + except ModelError as error: + raise MetricsError(error.issues) from None + + +def load( + source: Source, + model: Optional[Model] = None, + *, + model_name: Optional[str] = None, +) -> Metrics: + """Read a metrics file from a file, or the text of one, and check it conforms. + + `source` is a path (`coffee.metrics.yml`, `.json`) or the document's text, + YAML under the model spec's YAML profile, or JSON. With `model`, the file is + also checked against it: every table and column it names, every filter and + time path. `model_name` is the name a model without `name` goes by. Raises + `MetricsError` listing every error; the warnings of a file that conforms are + on the returned `Metrics.warnings`. + """ + text, document = _parse(source) + return from_dict(document, model, model_name=model_name, text=text) + + +def from_dict( + document: Any, + model: Optional[Model] = None, + *, + model_name: Optional[str] = None, + text: Optional[str] = None, +) -> Metrics: + """The metrics a parsed document describes, as `load` reads them; `text`, when given, + is the document's source, and puts a line on each issue.""" + issues = check_document(document) + metrics: Optional[Metrics] = None + if not any(issue.is_error for issue in issues): + metrics = _build(document) + if model is not None: + issues += check_against(metrics, model, model_name=model_name) + elif not metrics.infer: + issues += check_references(metrics, all_metrics(metrics, None)) + if text is not None: + issues = locate_issues(text, issues) + errors = [issue for issue in issues if issue.is_error] + warnings = [issue for issue in issues if not issue.is_error] + if errors or metrics is None: + raise MetricsError(errors, warnings) + metrics.warnings = warnings + return metrics + + +def validate( + source: Source, model: Optional[Model] = None, *, model_name: Optional[str] = None +) -> list[Issue]: + """Every issue in a metrics file, errors and warnings; no errors means it conforms.""" + try: + metrics = load(source, model, model_name=model_name) + except ModelError as error: + return [*error.issues, *error.warnings] + return list(metrics.warnings) + + +# --------------------------------------------------------------------------- +# Building the typed value from a document that passed `check_document` +# --------------------------------------------------------------------------- +def _extensions(mapping: Mapping) -> dict[str, Any]: + return { + key: copy.deepcopy(value) + for key, value in mapping.items() + if isinstance(key, str) and key.startswith("x-") + } + + +def _build(document: Mapping) -> Metrics: + return Metrics( + version=document["model2data-metrics"], + model=document["model"], + metrics={name: _metric(metric) for name, metric in (document.get("metrics") or {}).items()}, + dimensions={ + path: _dimension(value) for path, value in (document.get("dimensions") or {}).items() + }, + infer=document.get("infer", True), + extensions=_extensions(document), + ) + + +def _metric(data: Mapping) -> Metric: + key = next(key for key in KIND_KEYS if key in data) + ratio = data.get("ratio") + return Metric( + kind=KIND_KEYS[key], # type: ignore[arg-type] + measure=data.get("measure"), + agg=data.get("agg"), + count=data.get("count"), + ratio=Ratio(ratio["numerator"], ratio["denominator"], _extensions(ratio)) + if ratio is not None + else None, + expression=data.get("expression"), + label=data.get("label"), + description=data.get("description"), + format=data.get("format"), + ai_context=data.get("ai_context"), + time=data.get("time"), + where=_where(data["where"]) if "where" in data else None, + extensions=_extensions(data), + ) + + +def _where(data: Mapping, any_: bool = False) -> Where: + terms: list[Union[Condition, Where]] = [] + for key, value in data.items(): + if key in ("any", "all"): + terms.append(Where([_where(item) for item in value], any=key == "any")) + elif "." in key: + terms.append(_condition(key, value)) + return Where(terms, any=any_, extensions=_extensions(data)) + + +def _condition(column: str, value: Any) -> Condition: + if isinstance(value, list): + return Condition(column, {"in": list(value)}, written="list") + if isinstance(value, Mapping): + operators = { + key: list(item) if isinstance(item, list) else item + for key, item in value.items() + if not key.startswith("x-") + } + return Condition(column, operators, extensions=_extensions(value)) + return Condition(column, {"eq": value}, written="value") + + +def _dimension(value: Any) -> DimensionOverride: + if isinstance(value, bool): + return DimensionOverride(offered=value) + return DimensionOverride( + offered=True, + label=value.get("label"), + description=value.get("description"), + extensions=_extensions(value), + ) diff --git a/model2data/metrics/semantic.py b/model2data/metrics/semantic.py new file mode 100644 index 0000000..80f4634 --- /dev/null +++ b/model2data/metrics/semantic.py @@ -0,0 +1,336 @@ +"""One semantic representation of a model and its metrics, which every exporter reads. + +`resolve(model, metrics)` turns a model and its metrics file (or none: then +the metrics are the ones the model implies) into a `SemanticModel`: + +- **entities**, one per table: its keys, grain, role and description, and the + dbt name its seed and staging model go by (`stg_`); +- **relationships**, one per foreign key: many side to one side, the columns + paired in order; +- **dimensions**: by default every enum, boolean, date and timestamp column + that is not a key, a foreign key or a measure (dates and timestamps are + time dimensions whether keyed or not); `dimensions:` in the metrics file + adds a column (`true`, or a label) or takes one away (`false`); +- **measures**: every column with a `measure`, and how it aggregates; +- **metrics**: the file's metrics and the inferred ones it does not replace, + each resolved -- the table it aggregates, the aggregation, the join path to + every table its filter and time column are on, and its time column, the + default one filled in. + +The known values (`model2data.metrics.values`), the dbt tests +(`model2data.metrics.dbt`) and the Ossie export (`model2data.metrics.ossie`) +all read this and nothing else, so they cannot disagree about what a metric +is. model2data studio's semantic-layer export is meant to move onto it too. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Literal, Optional + +from model2data.dbt.naming import dbt_identifier +from model2data.metrics import expression as expr +from model2data.metrics.check import all_metrics, check_against +from model2data.metrics.columns import ColumnKind, column_kind, find_column, is_temporal +from model2data.metrics.graph import Graph, Join, Path +from model2data.metrics.infer import _label, _Names, aggregation_of +from model2data.metrics.reader import MetricsError +from model2data.metrics.types import Metric, Metrics, Where +from model2data.model.types import Model + + +@dataclass(frozen=True) +class Entity: + """A table as a business entity. `name` is its dbt name (`stg_` in the project).""" + + table: str + name: str + primary_key: tuple[str, ...] + unique_keys: tuple[tuple[str, ...], ...] + grain: Optional[tuple[str, ...]] + role: Optional[str] + description: Optional[str] + + +@dataclass(frozen=True) +class Relationship: + """A foreign key: rows of `from_table` (the many side) to one row of `to_table`.""" + + name: str + from_table: str + from_columns: tuple[str, ...] + to_table: str + to_columns: tuple[str, ...] + one_to_one: bool = False + + +@dataclass(frozen=True) +class Dimension: + """A column to group or filter by: `time` for a date or timestamp, else `categorical`.""" + + table: str + column: str + kind: Literal["categorical", "time"] + data_kind: ColumnKind + label: str + description: Optional[str] = None + members: Optional[tuple[str, ...]] = None + + @property + def path(self) -> str: + return f"{self.table}.{self.column}" + + +@dataclass(frozen=True) +class Measure: + table: str + column: str + agg: str + + +@dataclass +class ResolvedMetric: + """A metric with everything a consumer needs to compute it. + + For `simple` and `count`: `table` is aggregated -- `agg` over `column`, or + `count` over its rows -- after joining, along `joins`, every table its + `where` or `time` is on (by table key; the metric's own table is not in + it). `time` is the column path it is dated by, None for none. For `ratio` + and `derived`: `inputs` are the metrics it is computed from, `ratio` or + `node` how. + """ + + name: str + kind: Literal["simple", "count", "ratio", "derived"] + label: str + description: Optional[str] = None + format: Optional[str] = None + ai_context: Optional[str] = None + inferred: bool = False + table: Optional[str] = None + column: Optional[str] = None + column_kind: Optional[ColumnKind] = None + agg: Optional[str] = None + where: Optional[Where] = None + time: Optional[str] = None + time_kind: Optional[ColumnKind] = None + joins: dict[str, Path] = field(default_factory=dict) + inputs: tuple[str, ...] = () + numerator: Optional[str] = None + denominator: Optional[str] = None + node: Optional[expr.Node] = None + expression: Optional[str] = None + + +@dataclass +class SemanticModel: + name: str + description: Optional[str] + entities: dict[str, Entity] + relationships: list[Relationship] + dimensions: list[Dimension] + measures: list[Measure] + metrics: dict[str, ResolvedMetric] + model: Model = field(repr=False, compare=False) + + def simple_inputs(self, name: str) -> list[str]: + """The simple metrics and row counts `name` is computed from, itself if it is one, + each once, in the order they are first used.""" + found: dict[str, None] = {} + + def walk(current: str) -> None: + metric = self.metrics[current] + if metric.kind in ("simple", "count"): + found[current] = None + for dep in metric.inputs: + walk(dep) + + walk(name) + return list(found) + + +def resolve( + model: Model, metrics: Optional[Metrics] = None, *, model_name: Optional[str] = None +) -> SemanticModel: + """The semantic model of `model` and `metrics` (None: the inferred metrics only). + + `metrics` should be what `load(..., model)` returned; one that does not fit + the model raises `MetricsError`. + """ + if metrics is None: + metrics = Metrics(model=model.name or model_name or "model") + else: + issues = check_against(metrics, model, model_name=model_name) + errors = [issue for issue in issues if issue.is_error] + if errors: + raise MetricsError(errors) + graph = Graph(model) + return SemanticModel( + name=model.name or model_name or metrics.model, + description=model.description, + entities=_entities(model), + relationships=_relationships(graph), + dimensions=_dimensions(model, metrics), + measures=[ + Measure(key, name, agg) + for key, table in model.tables.items() + for name, column in table.columns.items() + if (agg := aggregation_of(column.measure)) is not None + ], + metrics={ + name: _resolve_metric(model, graph, name, metric) + for name, metric in all_metrics(metrics, model).items() + }, + model=model, + ) + + +def _entities(model: Model) -> dict[str, Entity]: + out = {} + for key, table in model.tables.items(): + unique: list[tuple[str, ...]] = [ + (name,) for name, column in table.columns.items() if column.unique + ] + unique += [tuple(k.columns) for k in table.keys if k.kind == "unique"] + out[key] = Entity( + table=key, + name=dbt_identifier(key), + primary_key=tuple(table.primary_key()), + unique_keys=tuple(unique), + grain=tuple(table.grain) if table.grain else None, + role=table.role, + description=table.description, + ) + return out + + +def _relationships(graph: Graph) -> list[Relationship]: + names = _Names() + out = [] + for steps in graph.steps.values(): + for step in steps: + out.append( + Relationship( + name=names.take( + f"{dbt_identifier(step.child)}_{'_'.join(step.child_columns)}" + f"_to_{dbt_identifier(step.parent)}" + ), + from_table=step.child, + from_columns=step.child_columns, + to_table=step.parent, + to_columns=step.parent_columns, + one_to_one=_one_to_one(graph.model, step), + ) + ) + return out + + +def _one_to_one(model: Model, step: Join) -> bool: + table = model.tables[step.child] + if len(step.child_columns) == 1: + reference = table.columns[step.child_columns[0]].references + if reference is not None and reference.table == step.parent: + return reference.one_to_one + return any( + fk.one_to_one + for fk in table.foreign_keys + if tuple(fk.columns) == step.child_columns and fk.references == step.parent + ) + + +def _dimensions(model: Model, metrics: Metrics) -> list[Dimension]: + out = [] + for key, table in model.tables.items(): + keyed = set(table.primary_key()) + for k in table.keys: + keyed.update(k.columns) + for fk in table.foreign_keys: + keyed.update(fk.columns) + for name, column in table.columns.items(): + kind = column_kind(model, column) + override = metrics.dimensions.get(f"{key}.{name}") + if override is not None and not override.offered: + continue + temporal = is_temporal(kind) + default = temporal or ( + kind in ("enum", "boolean") + and not ( + name in keyed + or column.pk + or column.unique + or column.references is not None + or column.measure not in (None, False) + ) + ) + if not default and override is None: + continue + enum = model.enum_for(column.type) + out.append( + Dimension( + table=key, + column=name, + kind="time" if temporal else "categorical", + data_kind=kind, + label=(override.label if override and override.label else None) or _label(name), + description=(override.description if override else None) or column.description, + members=tuple(enum.members) if enum is not None else None, + ) + ) + return out + + +def _resolve_metric(model: Model, graph: Graph, name: str, metric: Metric) -> ResolvedMetric: + out = ResolvedMetric( + name=name, + kind=metric.kind, + label=metric.label or _label(name), + description=metric.description, + format=metric.format, + ai_context=metric.ai_context, + inferred=metric.inferred, + ) + if metric.kind == "ratio": + assert metric.ratio is not None + out.numerator, out.denominator = metric.ratio.numerator, metric.ratio.denominator + out.inputs = tuple(dict.fromkeys((out.numerator, out.denominator))) + return out + if metric.kind == "derived": + assert metric.expression is not None + out.node = expr.parse(metric.expression) + out.expression = metric.expression + out.inputs = tuple(expr.names(out.node)) + return out + if metric.kind == "simple": + assert metric.measure is not None + table_key, column = find_column(model, metric.measure) + assert table_key is not None and column is not None + out.table = table_key + out.column = metric.measure.rpartition(".")[2] + out.column_kind = column_kind(model, column) + out.agg = metric.agg or aggregation_of(column.measure) + else: + assert metric.count is not None + out.table = metric.count + out.agg = "count" + out.where = metric.where + joined: list[str] = [] + if metric.time is not None: + out.time = metric.time + joined.append(metric.time.rpartition(".")[0]) + else: + found = graph.first_time_column(out.table) + if found is not None: + out.time = found[0] + joined.append(found[0].rpartition(".")[0]) + if out.time is not None: + time_table, time_column = find_column(model, out.time) + assert time_table is not None and time_column is not None + out.time_kind = column_kind(model, time_column) + if metric.where is not None: + joined.extend( + condition.column.rpartition(".")[0] for condition in metric.where.conditions() + ) + for table in joined: + if table != out.table and table not in out.joins: + out.joins[table] = graph.paths(out.table, table, limit=1)[0] + return out diff --git a/model2data/metrics/sql.py b/model2data/metrics/sql.py new file mode 100644 index 0000000..fb02103 --- /dev/null +++ b/model2data/metrics/sql.py @@ -0,0 +1,270 @@ +"""A resolved metric as SQL: one way of writing it for every place it is computed. + +Two dialects of the same thing: + +- **warehouse SQL** (`query`, `scalar`), which the known values run on DuckDB + and the generated dbt tests run on DuckDB or Postgres. It reads relations a + caller names (a DuckDB table, a dbt `ref()`), joins the tables a metric's + filter and time are on along their many-to-one path with `left join` (a row + whose parent is missing stays, and fails any filter on the parent, as a null + does), and casts every column it reads to the type its kind is read as + (`model2data.metrics.columns.SQL_TYPES`), so the answer does not depend on + how a CSV loader guessed the column's type. +- **ANSI SQL for Ossie** (`ansi`), over `dataset.field` names, with no joins: + an Ossie consumer joins along the relationships the document declares. + +A filter on a simple metric goes inside its aggregate, `sum(case when +then end)`, so rows that fail it count as nulls: `sum`, `avg` and +the rest skip them, and an aggregate over no rows at all is null (a count is +0). Conditions follow SQL's three-valued logic, which is exactly the spec's +null rule: a null fails every condition but `is_null: true`. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable +from decimal import Decimal +from typing import Any + +from model2data.metrics import expression as expr +from model2data.metrics.columns import SQL_TYPES, ColumnKind, column_kind, find_column +from model2data.metrics.semantic import ResolvedMetric, SemanticModel +from model2data.metrics.types import Condition, Where + +Relation = Callable[[str], str] + +_BARE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") + + +def quote(name: str) -> str: + """An identifier, double-quoted the ANSI way (a `"` inside doubled).""" + return '"' + name.replace('"', '""') + '"' + + +def identifier(name: str) -> str: + """A name as an SQL identifier: bare when it can be, else double-quoted.""" + return name if _BARE.match(name) else quote(name) + + +def _text(value: str) -> str: + return "'" + value.replace("'", "''") + "'" + + +def literal(value: Any, kind: ColumnKind, upper: bool = False) -> str: + """A filter value as an SQL literal of the column's kind.""" + if isinstance(value, bool): + text = "true" if value else "false" + return text.upper() if upper else text + if kind in ("date", "timestamp"): + word = "date" if kind == "date" else "timestamp" + return f"{word.upper() if upper else word} {_text(str(value))}" + if kind in ("integer", "decimal"): + if isinstance(value, float): + return format(Decimal(repr(value)), "f") + return str(value) + return _text(str(value)) + + +def _kw(word: str, upper: bool) -> str: + return word.upper() if upper else word + + +def condition_sql(condition: Condition, column: str, kind: ColumnKind, upper: bool = False) -> str: + """Every operator of `condition` on `column` (SQL text), all of them holding.""" + k = lambda word: _kw(word, upper) # noqa: E731 + lit = lambda value: literal(value, kind, upper) # noqa: E731 + parts = [] + for operator, operand in condition.operators.items(): + if operator == "eq": + parts.append(f"{column} = {lit(operand)}") + elif operator == "ne": + parts.append(f"{column} <> {lit(operand)}") + elif operator in ("in", "not_in"): + values = ", ".join(lit(value) for value in operand) + word = k("in") if operator == "in" else f"{k('not')} {k('in')}" + parts.append(f"{column} {word} ({values})") + elif operator in ("gt", "gte", "lt", "lte"): + sign = {"gt": ">", "gte": ">=", "lt": "<", "lte": "<="}[operator] + parts.append(f"{column} {sign} {lit(operand)}") + elif operator == "between": + parts.append(f"{column} {k('between')} {lit(operand[0])} {k('and')} {lit(operand[1])}") + else: # is_null + parts.append( + f"{column} {k('is')} {k('null') if operand else k('not') + ' ' + k('null')}" + ) + return parts[0] if len(parts) == 1 else "(" + f" {k('and')} ".join(parts) + ")" + + +ColumnOf = Callable[[str], tuple[str, ColumnKind]] + + +def where_sql(where: Where, column_of: ColumnOf, upper: bool = False) -> str: + """A filter as one SQL condition. `column_of(path)` is `(SQL for the column, its kind)`.""" + parts = [] + for term in where.terms: + if isinstance(term, Where): + joiner = f" {_kw('or' if term.any else 'and', upper)} " + alternatives = [ + where_sql(alt, column_of, upper) for alt in term.terms if isinstance(alt, Where) + ] + parts.append( + alternatives[0] if len(alternatives) == 1 else "(" + joiner.join(alternatives) + ")" + ) + else: + column, kind = column_of(term.column) + parts.append(condition_sql(term, column, kind, upper)) + return parts[0] if len(parts) == 1 else "(" + f" {_kw('and', upper)} ".join(parts) + ")" + + +def aggregate_sql( + metric: ResolvedMetric, value: str | None, condition: str | None, upper: bool = False +) -> str: + """The aggregate of a simple metric (`value` is its column's SQL) or a row count + (`value` None), with `condition` inside it when it has a filter.""" + k = lambda word: _kw(word, upper) # noqa: E731 + if metric.kind == "count": + if condition is None: + return f"{k('count')}(*)" + return f"{k('count')}({k('case')} {k('when')} {condition} {k('then')} 1 {k('end')})" + assert value is not None and metric.agg is not None + inner = ( + value + if condition is None + else (f"{k('case')} {k('when')} {condition} {k('then')} {value} {k('end')}") + ) + if metric.agg == "sum": + return f"{k('sum')}({inner})" + if metric.agg == "average": + return f"{k('avg')}({inner})" + if metric.agg in ("min", "max"): + return f"{k(metric.agg)}({inner})" + if metric.agg == "median": + return f"{k('percentile_cont')}(0.5) {k('within')} {k('group')} ({k('order')} {k('by')} {inner})" + if metric.agg == "count": + return f"{k('count')}({inner})" + return f"{k('count')}({k('distinct')} {inner})" + + +# --------------------------------------------------------------------------- +# Warehouse SQL +# --------------------------------------------------------------------------- +def _cast(alias: str, column: str, kind: ColumnKind) -> str: + return f"cast({alias}.{quote(column)} as {SQL_TYPES[kind]})" + + +class _From: + """The `from` and `join` clauses of a simple metric, and the alias of each table in them.""" + + def __init__(self, metric: ResolvedMetric, relation: Relation): + assert metric.table is not None + self.lines = [f"from {relation(metric.table)} as t0"] + self.alias = {metric.table: "t0"} + by_prefix: dict[tuple, str] = {(): "t0"} + for table, path in metric.joins.items(): + for length in range(1, len(path) + 1): + prefix = path[:length] + if prefix in by_prefix: + continue + alias = f"t{len(by_prefix)}" + previous = by_prefix[path[: length - 1]] + step = path[length - 1] + on = " and ".join( + f"cast({previous}.{quote(child)} as varchar) = cast({alias}.{quote(parent)} as varchar)" + for child, parent in zip(step.child_columns, step.parent_columns, strict=True) + ) + self.lines.append(f"left join {relation(step.parent)} as {alias} on {on}") + by_prefix[prefix] = alias + self.alias[table] = by_prefix[path] + + +def _parts( + semantic: SemanticModel, metric: ResolvedMetric, relation: Relation +) -> tuple[str, _From]: + clauses = _From(metric, relation) + + def column_of(path: str) -> tuple[str, ColumnKind]: + table, column = find_column(semantic.model, path) + assert table is not None and column is not None + kind = column_kind(semantic.model, column) + return _cast(clauses.alias[table], path.rpartition(".")[2], kind), kind + + condition = where_sql(metric.where, column_of) if metric.where is not None else None + value = None + if metric.kind == "simple": + assert metric.column is not None and metric.column_kind is not None + value = _cast("t0", metric.column, metric.column_kind) + return aggregate_sql(metric, value, condition), clauses + + +def query(semantic: SemanticModel, name: str, relation: Relation, *, by_month: bool = False) -> str: + """A simple metric or row count as a query: one row with its `value`, or, `by_month`, a + row per calendar month of its time column (`month` the month's first day).""" + metric = semantic.metrics[name] + aggregate, clauses = _parts(semantic, metric, relation) + joins = "\n".join(clauses.lines) + if not by_month: + return f"select {aggregate} as value\n{joins}" + assert metric.time is not None + table, column = metric.time.rpartition(".")[0], metric.time.rpartition(".")[2] + month = f"cast(date_trunc('month', cast({clauses.alias[table]}.{quote(column)} as timestamp)) as date)" + return f"select {month} as month, {aggregate} as value\n{joins}\ngroup by 1\norder by 1" + + +def scalar(semantic: SemanticModel, name: str, relation: Relation) -> str: + """Any metric as one scalar SQL expression: a simple one as a subquery, a ratio or a + derived one composed from its inputs' subqueries.""" + metric = semantic.metrics[name] + if metric.kind in ("simple", "count"): + indented = query(semantic, name, relation).replace("\n", "\n ") + return f"(\n {indented}\n)" + if metric.kind == "ratio": + assert metric.numerator is not None and metric.denominator is not None + numerator = scalar(semantic, metric.numerator, relation) + denominator = scalar(semantic, metric.denominator, relation) + return f"({numerator} / nullif({denominator}, 0))" + assert metric.node is not None + return expr.to_sql(metric.node, lambda dep: scalar(semantic, dep, relation)) + + +# --------------------------------------------------------------------------- +# ANSI SQL over dataset.field, for Ossie +# --------------------------------------------------------------------------- +def field_ref(semantic: SemanticModel, path: str) -> str: + """`orders.total_amount` as Ossie names it: the dataset (the table's dbt name), a dot, + the field (the column), each quoted only when it has to be.""" + table, column = path.rpartition(".")[0], path.rpartition(".")[2] + return f"{identifier(semantic.entities[table].name)}.{identifier(column)}" + + +def ansi(semantic: SemanticModel, name: str) -> str: + """A metric as one ANSI SQL aggregate expression over `dataset.field` names, its inputs + written out in full when it is a ratio or derived.""" + metric = semantic.metrics[name] + if metric.kind == "ratio": + assert metric.numerator is not None and metric.denominator is not None + return f"({ansi(semantic, metric.numerator)}) / NULLIF({ansi(semantic, metric.denominator)}, 0)" + if metric.kind == "derived": + assert metric.node is not None + text = expr.to_sql(metric.node, lambda dep: ansi(semantic, dep)) + return text.replace("nullif(", "NULLIF(") + + def column_of(path: str) -> tuple[str, ColumnKind]: + _, column = find_column(semantic.model, path) + assert column is not None + return field_ref(semantic, path), column_kind(semantic.model, column) + + condition = where_sql(metric.where, column_of, upper=True) if metric.where is not None else None + value = None + if metric.kind == "simple": + assert metric.table is not None and metric.column is not None + value = field_ref(semantic, f"{metric.table}.{metric.column}") + elif metric.table is not None: + # COUNT(*) names no dataset, so a reader could not tell whose rows it counts: + # a one-column primary key, never null, counts the same rows and says whose. + entity = semantic.entities[metric.table] + if len(entity.primary_key) == 1: + key = field_ref(semantic, f"{metric.table}.{entity.primary_key[0]}") + inner = key if condition is None else f"CASE WHEN {condition} THEN {key} END" + return f"COUNT({inner})" + return aggregate_sql(metric, value, condition, upper=True) diff --git a/model2data/metrics/types.py b/model2data/metrics/types.py new file mode 100644 index 0000000..422a771 --- /dev/null +++ b/model2data/metrics/types.py @@ -0,0 +1,163 @@ +"""A metrics file as typed values: what a `.metrics.yml` document says, read. + +These are the file's own words, before they meet the model: a column is still +the `table.column` path the file wrote, and a ratio still names its inputs. +`model2data.metrics.semantic` resolves them against a model (which table a +filter joins through, which column dates a metric) and adds the metrics the +model implies on its own. + +Like the model's types, every class is a plain, comparable dataclass, and the +`x-*` keys of the document, at any level, are kept in `extensions`. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Literal, Optional, Union + +if TYPE_CHECKING: + from model2data.model.errors import Issue + +# The aggregations the model spec's `measure` names, in its order. +AGGREGATIONS: tuple[str, ...] = ( + "sum", + "average", + "min", + "max", + "median", + "count", + "count_distinct", +) +# The two that apply to a column of any kind; the rest need a numeric one. +ANY_KIND_AGGREGATIONS: tuple[str, ...] = ("count", "count_distinct") +FORMATS: tuple[str, ...] = ("number", "currency", "percent") +# Each operator of a condition, in the order a condition is written out. +OPERATORS: tuple[str, ...] = ( + "eq", + "ne", + "in", + "not_in", + "gt", + "gte", + "lt", + "lte", + "between", + "is_null", +) +# The operators that compare by order, which a text, enum or boolean column has none of. +ORDER_OPERATORS: tuple[str, ...] = ("gt", "gte", "lt", "lte", "between") +# A metric's kind, by the one key that says it. +KIND_KEYS: dict[str, str] = { + "measure": "simple", + "count": "count", + "ratio": "ratio", + "expression": "derived", +} + +MetricKind = Literal["simple", "count", "ratio", "derived"] +Value = Union[str, int, float, bool] + + +@dataclass +class Condition: + """What one column must hold: every operator in `operators` must hold at once. + + A scalar in the file reads as `{eq: value}` and a list as `{in: [...]}`, so + `operators` always maps an operator of `OPERATORS` to its operand: a value, + a list of values (`in`, `not_in`, and `[low, high]` for `between`), or true + or false for `is_null`. + """ + + column: str + operators: dict[str, Any] + # How the file wrote it: a lone value (`eq`), a list (`in`), or operators. + written: Literal["value", "list", "operators"] = "operators" + extensions: dict[str, Any] = field(default_factory=dict) + + +@dataclass +class Where: + """A filter: every term must hold (or, with `any`, one of them). + + A term is a `Condition`, or a group: the `any` or `all` key of the file, a + `Where` whose terms are the filters listed under it, each a `Where` itself. + `any=True` on the group means one of them must hold, `any=False` (`all`) + that every one must. Terms are in document order. + """ + + terms: list[Union[Condition, Where]] = field(default_factory=list) + any: bool = False + extensions: dict[str, Any] = field(default_factory=dict) + + def conditions(self) -> list[Condition]: + """Every condition in the filter, nested ones included, in document order.""" + found: list[Condition] = [] + for term in self.terms: + found.extend(term.conditions() if isinstance(term, Where) else [term]) + return found + + +@dataclass +class Ratio: + numerator: str + denominator: str + extensions: dict[str, Any] = field(default_factory=dict) + + +@dataclass +class Metric: + """One metric. `kind` says which of `measure`, `count`, `ratio` and `expression` is set. + + - `simple`: `measure` is a column path, aggregated by `agg`, or, without + it, by the column's own `measure` in the model. + - `count`: `count` is a table key; the metric counts its rows. + - `ratio`: the numerator metric's total over the denominator's. + - `derived`: `expression` is arithmetic over metric names and numbers. + + `time` and `where` are for the first two only: a ratio or a derived metric + is filtered and dated through its inputs. `inferred` is True for a metric + the model implies (a `measure` column) rather than one the file wrote. + """ + + kind: MetricKind + measure: Optional[str] = None + agg: Optional[str] = None + count: Optional[str] = None + ratio: Optional[Ratio] = None + expression: Optional[str] = None + label: Optional[str] = None + description: Optional[str] = None + format: Optional[str] = None + ai_context: Optional[str] = None + time: Optional[str] = None + where: Optional[Where] = None + inferred: bool = False + extensions: dict[str, Any] = field(default_factory=dict) + + +@dataclass +class DimensionOverride: + """`dimensions:` for one column: offered (with a label or description) or never.""" + + offered: bool = True + label: Optional[str] = None + description: Optional[str] = None + extensions: dict[str, Any] = field(default_factory=dict) + + +@dataclass +class Metrics: + """A metrics file: the model it is for, its metrics and dimension overrides. + + `infer` (default True) adds a metric for every `measure` column of the + model; a metric of the same name here replaces the inferred one. + """ + + model: str + metrics: dict[str, Metric] = field(default_factory=dict) + dimensions: dict[str, DimensionOverride] = field(default_factory=dict) + infer: bool = True + version: str = "0.1.0" + extensions: dict[str, Any] = field(default_factory=dict) + # What the reader pointed out in a file that conforms. Not part of the value. + warnings: list[Issue] = field(default_factory=list, compare=False, repr=False) diff --git a/model2data/metrics/values.py b/model2data/metrics/values.py new file mode 100644 index 0000000..d1813ea --- /dev/null +++ b/model2data/metrics/values.py @@ -0,0 +1,284 @@ +"""The known value of every metric over a run's generated tables. + +A run knows its data, so it knows what each metric comes to: `known_values` +computes them over the frames the run wrote as seeds, overall and per calendar +month of the metric's time column. The generated dbt project then tests that +its own models give the same numbers (`model2data.metrics.dbt`), and anyone +building a semantic layer over the data can check theirs against them. + +How the values are computed: + +- Each table is read the way its seed is: the frame is written as CSV, exactly + as the seed file is, and read back as text, an empty field and `null` (in + any case) being null as dbt's seed loader reads them. Each column a metric + touches is then cast to the SQL type of its kind, in the same SQL the dbt + tests run (`model2data.metrics.sql`). DuckDB runs it, on one thread. +- Simple metrics and row counts are SQL aggregates. A ratio is its + numerator's total over its denominator's, null when the denominator is null + or zero; a derived metric is its expression over its inputs' totals, with + nulls in giving null out and division by zero giving null. Both are computed + from their inputs' values in decimal arithmetic. +- `by_month` holds a value for each calendar month (`YYYY-MM`) in which a row + of the metric's table has a time; rows with no time count only in the + total. A month in which a filter keeps no row has the value of an aggregate + over no rows: 0 for a count, null for anything else. A ratio or derived + metric has `by_month` when every metric it is computed from has a time, + each input taken in its own months. +- Values are rounded once, at the end: counts, and the sum, minimum and + maximum of an integer column, are exact integers; everything else is + rounded to 6 decimal places, half to even, and written without trailing + zeros. The same frames give the same JSON, byte for byte. +""" + +from __future__ import annotations + +import io +import json +from collections.abc import Mapping +from dataclasses import dataclass, field +from decimal import ROUND_HALF_EVEN, Context, Decimal, localcontext +from typing import Any, Optional, Union + +import pandas as pd + +from model2data.metrics import expression as expr +from model2data.metrics.check import SPEC_VERSION +from model2data.metrics.semantic import ResolvedMetric, SemanticModel +from model2data.metrics.sql import query, quote + +Number = Union[int, Decimal] +_PLACES = Decimal("0.000001") +ROUNDING = ( + "counts, and sums, minimums and maximums of integer columns, are exact; every other " + "value is rounded to 6 decimal places, half to even" +) + + +@dataclass +class KnownValue: + """One metric's value over the run's data, and per month when it has a time.""" + + name: str + kind: str + label: str + value: Optional[Number] + time: Optional[str] = None + by_month: Optional[dict[str, Optional[Number]]] = None + + +@dataclass +class KnownValues: + model: str + values: dict[str, KnownValue] = field(default_factory=dict) + seed: Optional[int] = None + as_of: Optional[str] = None + + def to_json(self) -> str: + """`metric_values.json`: stable text, keys in a fixed order, two-space indent.""" + document: dict[str, Any] = { + "model2data-metrics": SPEC_VERSION, + "model": self.model, + } + if self.seed is not None or self.as_of is not None: + document["run"] = { + key: value + for key, value in (("seed", self.seed), ("as_of", self.as_of)) + if value is not None + } + document["rounding"] = ROUNDING + metrics: dict[str, Any] = {} + for name, known in self.values.items(): + entry: dict[str, Any] = {"kind": known.kind, "label": known.label, "value": known.value} + if known.time is not None: + entry["time"] = known.time + if known.by_month is not None: + entry["by_month"] = known.by_month + metrics[name] = entry + document["metrics"] = metrics + return _dump(document) + "\n" + + +def _dump(value: Any, indent: int = 0) -> str: + """JSON text with a decimal written as its exact digits (the `json` module has no + way to write a `Decimal` but as a float, which would round it again).""" + pad = " " * (indent + 1) + if isinstance(value, dict): + if not value: + return "{}" + items = [ + f"{pad}{json.dumps(key, ensure_ascii=False)}: {_dump(item, indent + 1)}" + for key, item in value.items() + ] + return "{\n" + ",\n".join(items) + "\n" + " " * indent + "}" + if isinstance(value, Decimal): + return format(value, "f") + return json.dumps(value, ensure_ascii=False) + + +def _rounded(value: Any, exact: bool) -> Optional[Number]: + """A value from DuckDB or the arithmetic, as `known_values` writes it.""" + if value is None: + return None + if exact: + return int(value) + if isinstance(value, float): + value = Decimal(repr(value)) + elif not isinstance(value, Decimal): + value = Decimal(value) + rounded = value.quantize(_PLACES, rounding=ROUND_HALF_EVEN) + if rounded == 0: + return Decimal(0) + normal = rounded.normalize() + # normalize() writes 1200 as 1.2E+3; quantize back to no decimals for those. + return normal if normal.as_tuple().exponent < 0 else normal.quantize(Decimal(1)) # type: ignore[operator] + + +def _exact(metric: ResolvedMetric) -> bool: + if metric.kind == "count" or metric.agg in ("count", "count_distinct"): + return True + return metric.agg in ("sum", "min", "max") and metric.column_kind == "integer" + + +def _empty(metric: ResolvedMetric) -> Optional[Number]: + """The value of a metric over no rows: 0 for a count, null otherwise.""" + return 0 if metric.kind == "count" or metric.agg in ("count", "count_distinct") else None + + +def _as_text(frame: pd.DataFrame) -> pd.DataFrame: + """`frame` as its seed file reads back: every column text, `""` and `null` null.""" + buffer = io.StringIO(frame.to_csv(index=False)) + text = pd.read_csv(buffer, dtype=str, keep_default_na=False, na_values=[], na_filter=False) + return text.apply(lambda column: column.map(_null_or_text)).astype(object) + + +def _null_or_text(value: Any) -> Any: + return None if value == "" or str(value).lower() == "null" else value + + +def known_values( + semantic: SemanticModel, + frames: Mapping[str, pd.DataFrame], + *, + seed: Optional[int] = None, + as_of: Optional[str] = None, +) -> KnownValues: + """Every metric of `semantic` over `frames` (by table key: what the seeds hold).""" + import duckdb + + connection = duckdb.connect(":memory:") + try: + connection.execute("set threads = 1") + names: dict[str, str] = {} + used = { + table + for metric in semantic.metrics.values() + if metric.table is not None + for table in (metric.table, *metric.joins) + } + for index, key in enumerate(table for table in semantic.entities if table in used): + if key not in frames: + raise ValueError(f"no generated rows for table {key!r}, which a metric reads") + names[key] = f"m2d_{index}" + text = _as_text(frames[key]) + connection.register(f"{names[key]}_frame", text) + columns = ", ".join(f"cast({quote(c)} as varchar) as {quote(c)}" for c in text.columns) + connection.execute( + f"create table {names[key]} as select {columns} from {names[key]}_frame" + ) + connection.unregister(f"{names[key]}_frame") + raw: dict[str, tuple[Any, Optional[dict[str, Any]]]] = {} + for name, metric in semantic.metrics.items(): + if metric.kind not in ("simple", "count"): + continue + relation = names.__getitem__ + total = connection.execute(query(semantic, name, relation)).fetchone() + by_month = None + if metric.time is not None: + rows = connection.execute(query(semantic, name, relation, by_month=True)).fetchall() + by_month = { + month.strftime("%Y-%m"): value for month, value in rows if month is not None + } + raw[name] = (total[0] if total else None, by_month) + finally: + connection.close() + return _assemble(semantic, raw, seed=seed, as_of=as_of) + + +def _assemble( + semantic: SemanticModel, + raw: dict[str, tuple[Any, Optional[dict[str, Any]]]], + *, + seed: Optional[int], + as_of: Optional[str], +) -> KnownValues: + out = KnownValues(model=semantic.name, seed=seed, as_of=as_of) + for name, metric in semantic.metrics.items(): + if metric.kind in ("simple", "count"): + total, months = raw[name] + exact = _exact(metric) + value = _rounded(total, exact) + by_month = ( + {month: _rounded(item, exact) for month, item in months.items()} + if months is not None + else None + ) + else: + value, by_month = _composite(semantic, name, raw) + out.values[name] = KnownValue( + name=name, + kind=metric.kind, + label=metric.label, + value=value, + time=metric.time, + by_month=by_month, + ) + return out + + +def _composite( + semantic: SemanticModel, name: str, raw: dict[str, tuple[Any, Optional[dict[str, Any]]]] +) -> tuple[Optional[Number], Optional[dict[str, Optional[Number]]]]: + inputs = semantic.simple_inputs(name) + + def compute(pick: Mapping[str, Any]) -> Optional[Number]: + with localcontext(Context(prec=40)): + result = _evaluate(semantic, name, pick) + return _rounded(result, exact=False) + + total = compute({dep: raw[dep][0] for dep in inputs}) + if not all(raw[dep][1] is not None for dep in inputs): + return total, None + months = sorted({month for dep in inputs for month in (raw[dep][1] or {})}) + by_month = {} + for month in months: + picked = {} + for dep in inputs: + values = raw[dep][1] or {} + picked[dep] = values[month] if month in values else _empty(semantic.metrics[dep]) + by_month[month] = compute(picked) + return total, by_month + + +def _decimal(value: Any) -> Optional[Decimal]: + if value is None: + return None + if isinstance(value, float): + return Decimal(repr(value)) + return Decimal(value) + + +def _evaluate(semantic: SemanticModel, name: str, values: Mapping[str, Any]) -> Optional[Decimal]: + metric = semantic.metrics[name] + if metric.kind in ("simple", "count"): + return _decimal(values[name]) + if metric.kind == "ratio": + assert metric.numerator is not None and metric.denominator is not None + numerator = _evaluate(semantic, metric.numerator, values) + denominator = _evaluate(semantic, metric.denominator, values) + if numerator is None or denominator is None or denominator == 0: + return None + return numerator / denominator + assert metric.node is not None + return expr.evaluate( + metric.node, {dep: _evaluate(semantic, dep, values) for dep in metric.inputs} + ) diff --git a/model2data/spec/PUBLISHING.md b/model2data/spec/PUBLISHING.md index 050abfa..07810fb 100644 --- a/model2data/spec/PUBLISHING.md +++ b/model2data/spec/PUBLISHING.md @@ -22,6 +22,8 @@ On the website (`JB-Analytica/jba-website`), serve this directory's files at the | URL | File | | --- | --- | +| `https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json` | `model2data/spec/metrics/metrics.schema.json` of engine 1.13.0 | +| `https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/README.md` (optional, or render it as a page) | `model2data/spec/metrics/README.md` of engine 1.13.0 | | `https://www.jbanalytica.com/model2data/spec/0.4.0/model.schema.json` | `model2data/spec/model.schema.json` of engine 1.11.0 | | `https://www.jbanalytica.com/model2data/spec/0.4.0/README.md` (optional, or render it as a page) | `model2data/spec/README.md` of engine 1.11.0 | | `https://www.jbanalytica.com/model2data/spec/0.3.0/model.schema.json` | `model2data/spec/model.schema.json` of engine 1.9.0 | @@ -37,7 +39,11 @@ Requirements: Monaco playgrounds) can fetch it. - **`Content-Type: application/schema+json`** (or `application/json`). - Copy the file from the released engine version, not from a branch: the tag that ships each - version (0.2.0: engine 1.8.0; 0.3.0: engine 1.9.0; 0.4.0: engine 1.11.0). + version (0.2.0: engine 1.8.0; 0.3.0: engine 1.9.0; 0.4.0: engine 1.11.0; metrics 0.1.0: + engine 1.13.0). +- The metrics spec is versioned on its own, under `/spec/metrics//`, beside the model + spec's versions; a metrics file's first line points at it the same way: + `# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json`. Check it: @@ -46,7 +52,8 @@ curl -sI https://www.jbanalytica.com/model2data/spec/0.4.0/model.schema.json | g curl -s https://www.jbanalytica.com/model2data/spec/0.4.0/model.schema.json | python -c "import json,sys; print(json.load(sys.stdin)['\$id'])" ``` -The second command must print the URL itself. +The second command must print the URL itself. Check the metrics schema the same way, at +`/model2data/spec/metrics/0.1.0/metrics.schema.json`. ## 2. Register the file pattern with SchemaStore @@ -66,6 +73,20 @@ adding this entry to `src/api/json/catalog.json` (the list is kept alphabetical } ``` +A second entry registers the metrics files: + +```json +{ + "name": "model2data metrics", + "description": "The metrics of a model2data model: simple metrics, row counts, ratios and derived metrics, with filters (metrics spec 0.1.0)", + "fileMatch": ["*.metrics.yml", "*.metrics.yaml"], + "url": "https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json" +} +``` + +(`*.metrics.yml` is a common name; if SchemaStore asks for something narrower, register it +without `fileMatch`, so a file that points at the schema on its first line still validates.) + The schema stays hosted on our site (SchemaStore links to it, it does not copy it), so a new spec version only needs the `url` updated in a follow-up PR (an entry already registered with the 0.3.0 URL moves to 0.4.0, which also reads 0.2 and 0.3 documents), or a `versions` map diff --git a/model2data/spec/README.md b/model2data/spec/README.md index 4dba23b..4736936 100644 --- a/model2data/spec/README.md +++ b/model2data/spec/README.md @@ -54,6 +54,10 @@ a description prose. Nothing is encoded inside a string. The [model2data](https://github.com/JB-Analytica/model2data) engine reads this document and nothing else. DBML, dbt YAML and other formats are read by converting them into it. +What the model's numbers mean -- revenue, orders, average order value -- is written in a second, +optional document beside it, `.metrics.yml`, defined by the [metrics +spec](metrics/README.md) and versioned on its own. The generator never reads it. + ## What is normative This README and `model.schema.json`, including every `description` in the schema. A document @@ -221,7 +225,8 @@ they sit beside `generate`, not in it, because they are about the model rather t model. - `measure` is `true` (a measure, aggregated by `sum`), `false` (not a measure), or how it aggregates: `sum`, `average`, `min`, `max`, `median`, `count`, `count_distinct`. The last two - apply to any column, the rest to numeric ones. + apply to any column, the rest to numeric ones. Each measure also implies a metric, and a + [metrics file](metrics/README.md) adds named, filtered, derived ones. ```yaml orders: diff --git a/model2data/spec/examples/coffee_webshop.metrics.yml b/model2data/spec/examples/coffee_webshop.metrics.yml new file mode 100644 index 0000000..a583523 --- /dev/null +++ b/model2data/spec/examples/coffee_webshop.metrics.yml @@ -0,0 +1,58 @@ +# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json +model2data-metrics: 0.1.0 +model: coffee_webshop + +metrics: + revenue: + label: Revenue + description: Order value of the orders that were not cancelled or left unpaid. + ai_context: > + What customers paid for. Use it for sales and turnover questions; it leaves out pending + and cancelled orders. + measure: orders.total_amount # summed: the column's own `measure` is true + where: + orders.status: [paid, shipped, delivered] + time: orders.order_date + format: currency + + orders: + label: Orders + measure: orders.id + agg: count_distinct + + order_lines: + label: Order lines + count: order_items # the rows of a table + + average_order_value: + label: Average order value + ratio: {numerator: revenue, denominator: orders} + format: currency + + web_revenue: + label: Web revenue + measure: orders.total_amount + where: + orders.status: [paid, shipped, delivered] + orders.channel: web + + web_share: + label: Web share of revenue + expression: web_revenue / revenue # a fraction: 0.6 shows as 60% + format: percent + + coffee_units_sold: + label: Coffee units sold + description: Bags and capsules on orders that were not cancelled, sold online or in bulk. + measure: order_items.quantity # dated by orders.order_date, through order_id + where: + orders.status: {ne: cancelled} + products.category: [beans, ground, capsules] + customers.phone: {is_null: false} # two steps away: order_items -> orders -> customers + any: + - orders.channel: web + - order_items.quantity: {gte: 3} + +dimensions: + products.name: {label: Product} + customers.email: false diff --git a/model2data/spec/examples/coffee_webshop.model2data.yml b/model2data/spec/examples/coffee_webshop.model2data.yml index ec489c1..e6e2669 100644 --- a/model2data/spec/examples/coffee_webshop.model2data.yml +++ b/model2data/spec/examples/coffee_webshop.model2data.yml @@ -87,6 +87,7 @@ tables: total_amount: type: numeric not_null: true + measure: true generate: min: 10 max: 5000 @@ -106,6 +107,7 @@ tables: quantity: type: int not_null: true + measure: true generate: {min: 1, max: 10} unit_price: type: numeric diff --git a/model2data/spec/metrics/README.md b/model2data/spec/metrics/README.md new file mode 100644 index 0000000..2f8e6f6 --- /dev/null +++ b/model2data/spec/metrics/README.md @@ -0,0 +1,308 @@ +# model2data metrics — spec 0.1.0 + +A metrics file says what a model2data model's numbers mean: revenue, orders, average order value, +each written once, next to the model they are computed from. It is a second, optional document, +`.metrics.yml` beside `.model2data.yml`, versioned on its own. +[`metrics.schema.json`](metrics.schema.json) is its JSON Schema. The model it is for is defined by +the [model spec](../README.md). + +```yaml +# yaml-language-server: $schema=https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json +model2data-metrics: 0.1.0 +model: coffee_webshop # the model's `name` + +metrics: + revenue: + label: Revenue + description: Order value of orders that were not cancelled. + measure: orders.total_amount # aggregated by the column's own `measure`, or by `agg` + where: + orders.status: [paid, shipped, delivered] + time: orders.order_date + format: currency + orders: + measure: orders.id + agg: count_distinct + order_lines: + count: order_items # the rows of a table + average_order_value: + ratio: {numerator: revenue, denominator: orders} + format: currency + web_revenue: + measure: orders.total_amount + where: + orders.status: [paid, shipped, delivered] + orders.channel: web + web_share: + expression: web_revenue / revenue # a fraction: format shows 0.6 as 60% + format: percent + +dimensions: + products.name: {label: Product} + customers.email: false + +infer: true +``` + +A complete example, which conforms against the example model beside it, is +[`../examples/coffee_webshop.metrics.yml`](../examples/coffee_webshop.metrics.yml). + +The model and its generated data do not depend on the metrics file: a generator never reads it, +and a model generates the same rows with or without one. What a metrics file adds is meaning +(names, labels, formats, filters) and, for a consumer that has the data, numbers. + +## What is normative + +This README and `metrics.schema.json`, including every `description` in the schema. A document +**conforms** when, read under the model spec's [YAML profile](../README.md#the-yaml-profile), it +validates against the schema and meets every check under [Checks beyond the +schema](#checks-beyond-the-schema), and, read **against its model**, every check under [Checks +against the model](#checks-against-the-model). Examples are informative. + +A reader reports issues as the model spec does: each with the document path of the value +(`metrics.revenue.where.orders.status`) and a severity, an **error** where the document does not +conform and a **warning** for something it points out in a document that does. + +## Versioning + +The metrics spec is versioned on its own, apart from the model spec, the engine and the studio, +with [semantic versioning](https://semver.org). The version is in the schema's `$id`: + +``` +https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json +``` + +A document names the version it is written against in `model2data-metrics:`. A patch release +changes wording only; a minor release adds something optional, and, before 1.0.0, may also make a +valid document invalid or change what one means, and says so. A reader refuses a minor version it +does not implement (from 1.0.0, a major one) rather than guess. + +## The YAML profile, and extensions + +A metrics file is read under the model spec's YAML profile: the YAML 1.2 core schema (`no` is a +string, `2026-01-01` is a string), no duplicate keys, no anchors, aliases, merge keys or tags, +one UTF-8 document. JSON is read as JSON. + +A key beginning `x-` is an extension, allowed at the top level, on a metric, in a `ratio`, in a +`where` (a key with no `.` in it), in a condition's mapping of operators and on a dimension. It +carries no meaning here; a consumer ignores the ones it does not know and keeps them when it +rewrites the document. Every other unknown key makes the document invalid. + +## The document + +### `model` + +The `name` of the model the metrics are for. A model without a `name` goes by the stem of its +file (`coffee_webshop` for `coffee_webshop.model2data.yml`). Read against a model of another +name, the document is in error, not quietly applied to the wrong model. + +### Metrics + +`metrics` maps a **metric name** to a metric. A name is a lower-case letter, then lower-case +letters, digits and underscores (`^[a-z][a-z0-9_]*$`): it is used unquoted in SQL, in file names +and in other semantic layers. Order is significant: consumers list metrics in document order. + +A metric's **kind** comes from exactly one of four keys. None, or more than one, is an error. + +| Key | Kind | The metric is | +|-----|------|---------------| +| `measure: table.column` | simple | the column aggregated: by `agg`, or without it by the column's own `measure` in the model (`true` is `sum`) | +| `count: table` | row count | the number of rows of the table | +| `ratio: {numerator: a, denominator: b}` | ratio | metric `a`'s total divided by metric `b`'s | +| `expression: a - b` | derived | arithmetic over other metrics and numbers | + +The other keys: + +- `agg` (simple only): one of the model spec's seven aggregations, `sum`, `average`, `min`, + `max`, `median`, `count`, `count_distinct`. `count` (the non-null values) and `count_distinct` + apply to a column of any kind; the rest to a numeric one, by the model spec's [kind + rules](../README.md#generation-hints) (an enum is never numeric). Without `agg`, the column's + `measure` decides; a column with no `measure` (or `measure: false`) and no `agg` is an error. +- `label`: the metric's name for people. `description`: what it is, in prose. `ai_context`: + prose for an AI tool, on when to use it and what it is also called. +- `format`: how a value is shown, `number`, `currency` or `percent`. It changes no value. + `percent` expects a **fraction**: a value of 0.6 is shown as 60%, as Lightdash, Looker, Power + BI and Cube format percentages. So a share is written `web_revenue / revenue`, never + `* 100`. A consumer or exporter passes `percent` on as a percent format and does not multiply + the value, and a known value of a `percent` metric is the fraction. +- `time` (simple and row count only): the date or timestamp column the metric is dated by. See + [Time](#time). +- `where` (simple and row count only): a filter. See [Filters](#filters). A ratio or a derived + metric is filtered, and dated, through the metrics it is computed from. + +**A ratio** is a ratio of totals: the numerator's value over the whole of what is being looked at, +divided by the denominator's, never an average of row-level ratios. It is null when the +denominator is zero or null. + +**A derived metric's** `expression` is the one place this spec encodes something in a string, so +its grammar is as small as one can be: + +``` +expression := term (("+" | "-") term)* +term := factor (("*" | "/") factor)* +factor := ("+" | "-") factor | NUMBER | NAME | "(" expression ")" +NAME := [a-z][a-z0-9_]* a metric +NUMBER := [0-9]+ ("." [0-9]+)? +``` + +Whitespace between tokens is ignored; `*` and `/` bind tighter than `+` and `-`, and operators of +one precedence associate to the left. An expression uses at least one metric. It is evaluated over +its metrics' values in decimal arithmetic: any null makes the result null, and so does a division +by zero. A reader parses it; it never evaluates it as code. + +A ratio or an expression may use any metric: one written in the file, or one [inferred from the +model](#inferred-metrics). Metrics that depend on each other in a cycle are an error. + +### Filters + +`where` maps a **column path** (`table.column`, or `schema.table.column`: the last `.` separates +the column) to a **condition**, and every entry must hold. Two keys combine filters: + +- `any: [filter, ...]`: one of the filters holds; +- `all: [filter, ...]`: every one holds (needed inside `any`). + +A column path always contains a `.`, so it never collides with `any`, `all` or an extension. + +A condition is + +- a value: the column equals it (`orders.channel: web`); +- a list of values: the column is one of them (`orders.status: [paid, shipped]`); +- a mapping of operators, every one of which must hold: + +| Operator | Holds when the column | +|----------|------------------------| +| `eq: v` | equals `v` | +| `ne: v` | differs from `v` | +| `in: [v, ...]` | equals one of them | +| `not_in: [v, ...]` | equals none of them | +| `gt`, `gte`, `lt`, `lte: v` | is greater than, at least, less than, at most `v` | +| `between: [a, b]` | is at least `a` and at most `b` | +| `is_null: true` / `false` | is null / is not null | + +**Nulls** behave as in SQL: a null fails every condition except `is_null: true`. So `ne: cancelled` +and `not_in: [cancelled]` leave out the rows with no status as well as the cancelled ones; write +`any: [{orders.status: {ne: cancelled}}, {orders.status: {is_null: true}}]` to keep them. + +A **value** is checked against its column, by the model spec's kinds: + +| The column is | A value is | +|---------------|------------| +| an enum | one of its members (an integer member may be written as a number) | +| boolean | `true` or `false` | +| numeric | a number; a whole one for an integer column | +| a date | an ISO 8601 date string, `"2026-01-01"` | +| a timestamp | an ISO 8601 date or timestamp string, `"2026-01-01 08:30:00"`, with no time zone | +| anything else | text (quote one that YAML would read as a number) | + +The order operators (`gt`, `gte`, `lt`, `lte`, `between`) apply to numeric, date and timestamp +columns only. On an enum, a boolean or text they are an error: an enum's order is not its +members', and text order depends on the warehouse's collation, so a filter on it would not mean +one thing. List the values with `in` instead. + +Not in 0.1.0: `not`, text matching (`like`, prefixes), dates relative to today. + +### Reaching other tables + +A metric aggregates the rows of one table: a simple metric's column's, a row count's table. A +filter column, and its `time`, are on that table or on one it **reaches**: by following foreign +keys from child to parent (`references` and `foreign_keys`, the many side to the one side), +one or more steps. Each step joins a row to at most one parent row, so the rows being aggregated +are never multiplied. A step from parent to child is never taken. + +A column is reachable when **exactly one** such path leads to its table. When two do (`orders` +with both `billed_to` and `shipped_to` referencing `customers`), which one the filter means is +unclear, and it is an error naming both paths. A column no path leads to is an error too. + +A row whose foreign key is null, or names no parent row, reaches no parent: its parent's columns +are null, and fail every condition but `is_null: true`. + +### Time + +A simple metric's or a row count's `time` names the date or timestamp column it is dated by: its +value per month is computed over the rows whose time falls in that month. The column is on the +metric's table or one it reaches, as a filter column is. Without `time`: + +1. the metric's table's first date or timestamp column, in document order; +2. else, nearest first and in document order of the foreign keys, the first date or timestamp + column of a table it reaches along exactly one path (`order_items` is dated by + `orders.order_date`, through `order_id`). + +A metric with neither is valid, and a reader warns: it has a total but no value per month. + +### Dimensions + +The **dimensions** of a model are the columns to group and filter metrics by. By default they are +every date and timestamp column (a time dimension), and every enum and boolean column that is not +a primary key, in a key, `unique`, a foreign key, or a `measure` (a categorical dimension). + +`dimensions` changes that, by column path: + +- `false`: never offered as a dimension (`customers.email: false`); +- `true`: offered, as a time dimension when the column is a date or timestamp, else categorical; +- a mapping: offered, with a `label` and a `description`. + +### Inferred metrics + +With `infer: true` (the default), a model implies metrics on its own, beside the file's: + +- one **simple metric for each column with a `measure`**, aggregated by it, named + `
_`, with `_` appended when it is not `sum` + (`orders_total_amount`, `products_price_average`); +- one **row count for each fact**, named `
_count`: a table whose `role` is `fact`, or which + has no `role` and at least one `measure` column. + +A name is the text lower-cased, with every run of other characters made one `_`, trimmed of `_`; +it starts with a letter (else `c_` is put in front) and has at least two characters (else `_x` is +appended), and a name already taken gets `_2`, `_3`, in document order of tables and columns. An +inferred metric's label is the table and the column in words (`Orders total amount`), with the +aggregation in brackets when it is not `sum`. + +A metric of the same name in the file **replaces** the inferred one. Consumers list the file's +metrics first, in document order, then the inferred ones it did not replace, in model order. +`infer: false` turns inference off: the metrics are only the ones written here. + +A model with no metrics file has the inferred metrics, wherever a consumer exposes metrics. + +These are the names model2data studio's semantic-layer export gave its metrics, so a metric keeps +its name when the studio moves onto this spec. + +## Checks beyond the schema + +A conforming document also meets these, read on its own: + +1. `model2data-metrics` is a version this reader implements (0.1.x). +2. Each metric has exactly one of `measure`, `count`, `ratio` and `expression`; `agg` only with + `measure`; `time` and `where` only with `measure` or `count`. +3. Every `expression` parses by the grammar above and uses at least one metric. +4. No metric depends on itself, directly or through other metrics. + +## Checks against the model + +Read against its model, a conforming document also meets these: + +1. `model` is the model's name. +2. Every column path and table key names a column or table of the model (`measure`, `count`, + `time`, every column of a `where`, every key of `dimensions`). +3. A simple metric's aggregation fits its column (see `agg`); without `agg`, the column has a + `measure`. +4. Every metric a `ratio` or an `expression` names exists, in the file or inferred. +5. `time` is a date or timestamp column. +6. Every filter column and `time` is reachable from the metric's table along exactly one path. +7. Every filter value fits its column, and no order operator is on an enum, boolean or text + column. + +### Warnings + +1. A simple metric or row count of the file has no time: no `time`, and no date or timestamp + column on its table or a table it reaches. +2. A filter can never match, as far as the model's generation hints cheaply tell: + `is_null: true` on a column that is never null (`pk` or `not_null`), a comparison or value + outside a numeric column's `min` and `max`, or a `between` whose low end is above its high end. + +## What a reader computes + +This spec fixes what a metric means. How a consumer computes and publishes metrics is its own +business; the [model2data engine](https://github.com/JB-Analytica/model2data) computes each one's +known value over a generated run (`metric_values.json`), writes a dbt test per metric that checks +the warehouse gets the same number, and exports Apache Ossie 0.1.1 as tagged `osi-0.1.1-rc1` in +apache/ossie (no final 0.1.1 tag exists yet). Its README says how. diff --git a/model2data/spec/metrics/metrics.schema.json b/model2data/spec/metrics/metrics.schema.json new file mode 100644 index 0000000..d46cde4 --- /dev/null +++ b/model2data/spec/metrics/metrics.schema.json @@ -0,0 +1,177 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json", + "title": "model2data metrics 0.1.0", + "description": "The metrics of a model2data model: a `.metrics.yml` file beside `.model2data.yml`. Normative together with README.md in this directory, which says what a document means and the checks a reader makes beyond this schema (against the model, among others).", + "type": "object", + "required": ["model2data-metrics", "model"], + "properties": { + "model2data-metrics": { + "description": "The version of the metrics spec the document is written against. A reader refuses a minor version it does not implement (the spec is before 1.0.0).", + "type": "string", + "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" + }, + "model": { + "description": "The `name` of the model the metrics are for. A metrics file read against a model of another name is an error.", + "type": "string", + "minLength": 1 + }, + "metrics": { + "description": "The metrics, by name. Order is significant: consumers list them in document order.", + "type": "object", + "propertyNames": {"$ref": "#/$defs/metricName"}, + "additionalProperties": {"$ref": "#/$defs/metric"} + }, + "dimensions": { + "description": "Which columns are offered as dimensions, beyond the default (every enum, boolean, date and timestamp column that is not a key), by column path. `false`: never offered. `true` or a mapping: offered, with its label and description.", + "type": "object", + "propertyNames": {"$ref": "#/$defs/columnPath"}, + "additionalProperties": {"$ref": "#/$defs/dimension"} + }, + "infer": { + "description": "Whether every `measure` column of the model also yields a metric (default true). false: only the metrics written here.", + "type": "boolean" + } + }, + "patternProperties": {"^x-": {}}, + "additionalProperties": false, + "$defs": { + "metricName": { + "description": "A metric name: a lower-case letter, then lower-case letters, digits and underscores.", + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$" + }, + "columnPath": { + "description": "A column of the model: `table.column`, or `schema.table.column`. The last `.` separates the column.", + "type": "string", + "pattern": "^[^.]+(\\.[^.]+)?\\.[^.]+$" + }, + "tableKey": { + "description": "A table of the model: its name, or `schema.name`.", + "type": "string", + "pattern": "^[^.]+(\\.[^.]+)?$" + }, + "metric": { + "type": "object", + "description": "One metric. Exactly one of `measure` (simple), `count` (a row count), `ratio` and `expression` (derived) says its kind.", + "properties": { + "label": {"type": "string", "description": "The metric's name for people.", "minLength": 1}, + "description": {"type": "string", "description": "What the metric is, in prose."}, + "ai_context": {"type": "string", "description": "Prose for an AI tool: when to use the metric, what it is called, what it is not."}, + "format": { + "description": "How a value is shown. It changes no value: `percent` expects a fraction (0.6 is shown as 60%), and a consumer passes it on as a percent format without multiplying.", + "enum": ["number", "currency", "percent"] + }, + "measure": { + "$ref": "#/$defs/columnPath", + "description": "A simple metric: this column, aggregated by `agg`, or without it by the column's own `measure` in the model (`true` is sum)." + }, + "agg": { + "description": "How a simple metric aggregates its column. `count` and `count_distinct` apply to any column; the rest to numeric ones.", + "enum": ["sum", "average", "min", "max", "median", "count", "count_distinct"] + }, + "count": { + "$ref": "#/$defs/tableKey", + "description": "A row count: the rows of this table (that pass `where`)." + }, + "ratio": { + "description": "The numerator metric's total divided by the denominator metric's: a ratio of totals, null when the denominator is zero or null.", + "type": "object", + "required": ["numerator", "denominator"], + "properties": { + "numerator": {"$ref": "#/$defs/metricName"}, + "denominator": {"$ref": "#/$defs/metricName"} + }, + "patternProperties": {"^x-": {}}, + "additionalProperties": false + }, + "expression": { + "description": "A derived metric: arithmetic over metric names and numbers, with + - * / and parentheses (the grammar is in README.md). Division by zero is null.", + "type": "string", + "minLength": 1 + }, + "time": { + "$ref": "#/$defs/columnPath", + "description": "The date or timestamp column a simple metric or row count is dated by: on its table, or reachable through many-to-one references. Default: the table's first date or timestamp column, else the first one reachable." + }, + "where": {"$ref": "#/$defs/where"} + }, + "patternProperties": {"^x-": {}}, + "additionalProperties": false, + "oneOf": [ + {"required": ["measure"], "not": {"anyOf": [{"required": ["count"]}, {"required": ["ratio"]}, {"required": ["expression"]}]}}, + {"required": ["count"], "not": {"anyOf": [{"required": ["measure"]}, {"required": ["ratio"]}, {"required": ["expression"]}, {"required": ["agg"]}]}}, + {"required": ["ratio"], "not": {"anyOf": [{"required": ["measure"]}, {"required": ["count"]}, {"required": ["expression"]}, {"required": ["agg"]}, {"required": ["time"]}, {"required": ["where"]}]}}, + {"required": ["expression"], "not": {"anyOf": [{"required": ["measure"]}, {"required": ["count"]}, {"required": ["ratio"]}, {"required": ["agg"]}, {"required": ["time"]}, {"required": ["where"]}]}} + ] + }, + "where": { + "description": "A filter on a simple metric or a row count: every entry must hold. A key containing a `.` is a column path and maps to its condition; `any` holds when one of its filters does, `all` when every one does. A column must be on the metric's table or reachable through many-to-one references along exactly one path.", + "type": "object", + "minProperties": 1, + "properties": { + "any": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/where"}}, + "all": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/where"}} + }, + "patternProperties": { + "^x-[^.]*$": {}, + "\\.": {"$ref": "#/$defs/condition"} + }, + "additionalProperties": false + }, + "value": { + "description": "A value a column is compared with: an enum member, true or false for a boolean column, a number for a numeric one, an ISO 8601 date or timestamp string for a date or timestamp one, text for any other.", + "type": ["string", "number", "boolean"] + }, + "values": { + "type": "array", + "minItems": 1, + "items": {"$ref": "#/$defs/value"} + }, + "condition": { + "description": "What a column must hold. A value: equal to it. A list: one of them. A mapping of operators: every one must hold. A null fails every condition but `is_null: true`, as in SQL.", + "oneOf": [ + {"$ref": "#/$defs/value"}, + {"$ref": "#/$defs/values"}, + { + "type": "object", + "minProperties": 1, + "properties": { + "eq": {"$ref": "#/$defs/value"}, + "ne": {"$ref": "#/$defs/value"}, + "in": {"$ref": "#/$defs/values"}, + "not_in": {"$ref": "#/$defs/values"}, + "gt": {"$ref": "#/$defs/value"}, + "gte": {"$ref": "#/$defs/value"}, + "lt": {"$ref": "#/$defs/value"}, + "lte": {"$ref": "#/$defs/value"}, + "between": { + "description": "[low, high], both inclusive.", + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": {"$ref": "#/$defs/value"} + }, + "is_null": {"type": "boolean"} + }, + "patternProperties": {"^x-": {}}, + "additionalProperties": false + } + ] + }, + "dimension": { + "oneOf": [ + {"type": "boolean"}, + { + "type": "object", + "properties": { + "label": {"type": "string", "minLength": 1}, + "description": {"type": "string"} + }, + "patternProperties": {"^x-": {}}, + "additionalProperties": false + } + ] + } + } +} diff --git a/pyproject.toml b/pyproject.toml index 56b8fc3..94e9ee4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta" [project] name = "model2data" -version = "1.12.0" -description = "Generate analytics-ready datasets from DBML models" +version = "1.13.0" +description = "Turn a data model into realistic synthetic data, a runnable dbt project and metrics with known values" readme = "README_PYPI.md" requires-python = ">=3.10" license = "MIT" diff --git a/tests/fixtures/determinism.json b/tests/fixtures/determinism.json index 60a4f53..08866f6 100644 --- a/tests/fixtures/determinism.json +++ b/tests/fixtures/determinism.json @@ -533,6 +533,101 @@ "dbt_det/seeds/raw/product_reviews.csv": "bcc6a29dba1224d32aca2bcdb1e0d83d618dfa8e8168d6c1381b5bb549f6b1d6", "dbt_det/seeds/raw/products.csv": "89a4ff01ab5a53712fc8b47728dd463104a008eb3d31b3fee1b6a95e0d8e1188" }, + "metrics": { + "dbt_det/coffee_webshop.metrics.yml": "16981620dc46a34c01f2c6cfc36644522bf9c5a80defb49206cc2a82eca67c32", + "dbt_det/coffee_webshop.model2data.yml": "ae20f85480b4e5503e2f54ea5a71d24673b8bb5990644c850e51f91c5eddbc4a", + "dbt_det/data-tests/metrics/metric_average_order_value.sql": "d47204215d92b41b5216e61559e5593745d3ab13e0549045a6f9a816a0c1171b", + "dbt_det/data-tests/metrics/metric_coffee_units_sold.sql": "e8f15c7726dcbce43d0e72caca13d537fba8eb93a60e19e1ebbed780ce41ee7b", + "dbt_det/data-tests/metrics/metric_order_items_count.sql": "3d0e8d87ceb9c396106908800b18872c499f9927b634780868271d003df3a415", + "dbt_det/data-tests/metrics/metric_order_items_quantity.sql": "918e178f40d3deb4ccb9bce56adb09ded838e7bdeebe75c219474a1f50037799", + "dbt_det/data-tests/metrics/metric_order_lines.sql": "20bd6bf0823be4a1e62cec088f29ad84273ff9854599125862b7d1783deefc5d", + "dbt_det/data-tests/metrics/metric_orders.sql": "8f0ec4c078bfe0d28a5efb93c9e1763a19f88c2314f0cb6ffb3ab7709d7a1c49", + "dbt_det/data-tests/metrics/metric_orders_count.sql": "628a2ac01a89dbf22c8b83618704b2be1d674b440e23af6aceb4246398b0944d", + "dbt_det/data-tests/metrics/metric_orders_total_amount.sql": "072dcd484960eddac5609721af41e0efca0e6ba1ae2cf1cab3500bf7a212ccd4", + "dbt_det/data-tests/metrics/metric_revenue.sql": "23235c3ffb49ed5a7ee427628ce64921d49034bedbc0da360f1afcde26a4639b", + "dbt_det/data-tests/metrics/metric_web_revenue.sql": "c13c50a47a735df3a275d45b4bcaa7e97f028bbd7dd1451116bf02c83e01eddb", + "dbt_det/data-tests/metrics/metric_web_share.sql": "e6b2d720239105a41b1a75164c2f07f0051aa9e2f21ce2b078526af377abb58a", + "dbt_det/dbt_project.yml": "631ba06ded8db3ebbfa85351d29ef0953dc5b92992235b786a21424e1db3ddda", + "dbt_det/macros/generate_schema_name.sql": "60f37d9f6fc022c0f482c2699a2e1a4dc61207cbb0a7f355788b4c1d43066438", + "dbt_det/macros/model2data_hint_tests.sql": "d0b45f48d4740c22b3881654fc704f48ca08404b8d574f37680c7237ba4b08f8", + "dbt_det/metric_values.json": "37a66e08fbbe5d227a141d6be3efd31be6925b835e67cc95cbfd6b3922ec1316", + "dbt_det/models/staging/stg_customers.sql": "44df48978d6ac653bdf1c8ed58d8884068c066f5131aeb8f4822f8dc4c424b0a", + "dbt_det/models/staging/stg_customers.yml": "02f18d31e87a4bfb8314588180ef3382b8db5e5b5460e4222f6e094e4006582c", + "dbt_det/models/staging/stg_order_items.sql": "d9ffb6428433e8ad821637b07a40f88037201b1ea3ad4d05c06ace185f496b39", + "dbt_det/models/staging/stg_order_items.yml": "2d90adc3a2e3beeddab1b0b8a0140309a1bb5b667337d1e187b2ab40333cd966", + "dbt_det/models/staging/stg_orders.sql": "ce9432edd2e51bd8d6caa6e887268ae57ebad6041881548e140e172fc4420c91", + "dbt_det/models/staging/stg_orders.yml": "5733941fb5c1419091efa5a9044030c4f205be95a2b68413df81fdd220eaa8c7", + "dbt_det/models/staging/stg_product_reviews.sql": "939f77269e38a2b8265ce001e62a8194fd5335d340f97168a67f6915c8881f75", + "dbt_det/models/staging/stg_product_reviews.yml": "62bab917381e2a43ee1067a5cd56119082cb7f7dff10f23a687c2a9345b2a07c", + "dbt_det/models/staging/stg_products.sql": "7455128b10e0357213f26c209913a73df3f417be28ddd9d4dd8681390e7dcdab", + "dbt_det/models/staging/stg_products.yml": "a4c6588fa066b5bb5ac7a9b66db33dd15a77dea40c2688f9520e4ceaf53ab240", + "dbt_det/osi/det.yml": "5d005ea6c0d5af8887e3c43bb9dea9a1be080e94cc6b913253c95e86d7b64dc7", + "dbt_det/profiles.yml": "c40b04badc35b941a8de1079c8d0a298f6446f0d5caf5bd50dbe3797ab2a2255", + "dbt_det/seeds/raw/__seed_config.yml": "8e3df67807677a8ed3e8949f8dd7c7dba6a088d4f5221236a770638802d62105", + "dbt_det/seeds/raw/customers.csv": "f97be9bf40f3f5f3c97eb9697a866733d5118518ed1e48475e3d4363b3febbb4", + "dbt_det/seeds/raw/order_items.csv": "77134c2df8747a7fd348846a2cc7a233f6e0b4a8159204ec86aa7484747837c0", + "dbt_det/seeds/raw/orders.csv": "1b4b057db7bfd59f49182f3effd13ecafa5ff3dba1de0ef766162e2b90b53e6b", + "dbt_det/seeds/raw/product_reviews.csv": "dd0b9a9087b147836358609c315cf2f15a83c9770c225bfee2f26742eab8e48a", + "dbt_det/seeds/raw/products.csv": "980c75d598d2bbb80b768df98c0958789f6720627c94bc542a40e1acac9c8ee5" + }, + "metrics-defects-days": { + "dbt_det/EXPECTED_FAILURES.md": "233eff9d9cd993bd7a2b6621870f8ef315785230eda5e23dca07286b6f68c192", + "dbt_det/data-tests/metrics/metric_average_order.sql": "d80e42fddda755c509ba6b8234fc5a7148439200562b615f7ad672b2f559a9d8", + "dbt_det/data-tests/metrics/metric_customers_ordering.sql": "59e9c3257ec14d3f88ef1f8e12abd01988d46caccccd5357fbac943542e6ee41", + "dbt_det/data-tests/metrics/metric_delivered_revenue.sql": "ba26ce0181f17bc1875d476c6aba9481cb242b1c7d13cf332ade14d05f0cdedf", + "dbt_det/data-tests/metrics/metric_item_value.sql": "a20af11fe63a2be7aa8fd678903fd1b112e47dc7580c4290e204c94c4f49daba", + "dbt_det/data-tests/metrics/metric_items_of_recent_orders.sql": "8278a4996aedd85c53d3872ce84fffafb4dad8dd1f366b1cb5add01803c5b898", + "dbt_det/data-tests/metrics/metric_largest_quantity.sql": "c4bacbc70b08ccacaac62a0e2b88229d10ac57674c40f45c89945bf976ced4e5", + "dbt_det/data-tests/metrics/metric_median_order.sql": "c31f160119f37f0b9da2e1e28fc6e7cfded949c1c9fc52e318ec88cf4d8134bc", + "dbt_det/data-tests/metrics/metric_odd_mix.sql": "1cfaae567a6832ea2ae0991f18ecb3fded0a8fe5896da9f5c0d6478eedebfcab", + "dbt_det/data-tests/metrics/metric_open_orders.sql": "2e17d92acec3359b59999af6038b7b61dfc5f65cf7a5abf97b2c2a235426b13f", + "dbt_det/data-tests/metrics/metric_order_items_quantity.sql": "56203c4e6fb6484b4edd93c71a4b65112988acefd88ced83920523675ed4bb40", + "dbt_det/data-tests/metrics/metric_orders_count.sql": "592476a2c421c58056ad647ca7ad0a777a6642bbedc31bbb853aba9dc34ff396", + "dbt_det/data-tests/metrics/metric_per_order.sql": "441cc6ad7c7c41652bd90be2571caf22c54100b5a16bc133b8ac1b1b5182e530", + "dbt_det/data-tests/metrics/metric_rows_with_a_status.sql": "eaa990b17ba39de81eab1b85412944a609da72da59df8a6cc02f1e348a29d874", + "dbt_det/data-tests/metrics/metric_smallest_order.sql": "19a15f9618e90b2fa00c9fe61e5ba94335ee94b4e43fdb31bef5222db037de57", + "dbt_det/data-tests/metrics/metric_units_per_order.sql": "a2fd7b54c5c331564cf9912c2c6cb331c4249817c0cc3681a7ce8168ceb361db", + "dbt_det/days/customers/day_000.csv": "b0f8a38e0fb85d2a5ebd8ca80d56d9439b5938cbce6b4013bf91c4fb24502552", + "dbt_det/days/customers/day_001.csv": "220e7214482b5ac03b117c6d5c0fa2007f8ce17e1dcc9fc4c253a41df6acaa28", + "dbt_det/days/customers/day_002.csv": "f23b8f5fdc0e9abdb5efd01f6090d5c4f10d99fd47b07e228cd882bffa4c9738", + "dbt_det/days/order_items/day_000.csv": "68fda4e887148c46a19e39d5cc4ce3df20adbc5e846e2406b7fec80ac85d7594", + "dbt_det/days/order_items/day_001.csv": "c745a9e7052352d861d82b57b5a58371f3d026ecb739125deb60a00237e03b18", + "dbt_det/days/order_items/day_002.csv": "d22872e01590564174fbaa5f0f399a67b71a7f365cb3171e58259d8676c5cbf4", + "dbt_det/days/orders/day_000.csv": "a640c05ab85ad9125a030b24cb81932845ea56f0f0561229849c82208ee6a1b4", + "dbt_det/days/orders/day_001.csv": "4340b37b948e5b999c412d9d0542cdacef598bee46cd8f82c407a1759c744d74", + "dbt_det/days/orders/day_002.csv": "5de8d8e169904130aec139b1b65221ecd4c994d640f4e1dee72c332811d7941e", + "dbt_det/days/product_reviews/day_000.csv": "f9f33d24a151c6ff680ce5c4521f87418284a4de045dcb0eacb11fba048b7324", + "dbt_det/days/products/day_000.csv": "9274598e42761eea40a269c36524bc83eab320de4160c2fe0f0a5584fd8aebb3", + "dbt_det/dbt_project.yml": "631ba06ded8db3ebbfa85351d29ef0953dc5b92992235b786a21424e1db3ddda", + "dbt_det/defects_report.json": "02983affaff02f2248b52acc957c9c5f9a1dd3c071d72708b008d7707e39355c", + "dbt_det/ecommerce_training.metrics.yml": "01324c4ec6103dee37f66d254c82d85515e894ccabd13262368814ab2fc9c6c1", + "dbt_det/ecommerce_training.model2data.yml": "ab1cfa8d1b9b9638a4e8767cb02209ab2033beb1bdeff470f4bf0b2e596612ed", + "dbt_det/macros/generate_schema_name.sql": "60f37d9f6fc022c0f482c2699a2e1a4dc61207cbb0a7f355788b4c1d43066438", + "dbt_det/macros/model2data_hint_tests.sql": "d0b45f48d4740c22b3881654fc704f48ca08404b8d574f37680c7237ba4b08f8", + "dbt_det/macros/model2data_history_tests.sql": "f202816bc0a415be7f3a2059bda83233cce37a88004d8c7d6a5fcf24964355ad", + "dbt_det/metric_values.json": "1f52439dfaa03f459e00e3fb6d7c24e6e5cd672b5005353596ac33acc2e25a1f", + "dbt_det/models/staging/stg_customers.sql": "44df48978d6ac653bdf1c8ed58d8884068c066f5131aeb8f4822f8dc4c424b0a", + "dbt_det/models/staging/stg_customers.yml": "55ce32de721f6a366ad5c8c2d3036f7eab2719b6ebe9d7b0bf381afadf58fc6b", + "dbt_det/models/staging/stg_order_items.sql": "d9ffb6428433e8ad821637b07a40f88037201b1ea3ad4d05c06ace185f496b39", + "dbt_det/models/staging/stg_order_items.yml": "2d90adc3a2e3beeddab1b0b8a0140309a1bb5b667337d1e187b2ab40333cd966", + "dbt_det/models/staging/stg_orders.sql": "ce9432edd2e51bd8d6caa6e887268ae57ebad6041881548e140e172fc4420c91", + "dbt_det/models/staging/stg_orders.yml": "fe63c9f2fea411ec373b19a500e90c4e0c0b7352ba2d4591a33cbcf4a138ed7c", + "dbt_det/models/staging/stg_orders_history.sql": "937bea699b5df5da6808580fe3963f432163f3741afc0f71275e5261d81f655b", + "dbt_det/models/staging/stg_orders_history.yml": "b6bc4e53dc5078994b19f94b38371ea27308a452cccf351bfd43b6c489e2152a", + "dbt_det/models/staging/stg_product_reviews.sql": "939f77269e38a2b8265ce001e62a8194fd5335d340f97168a67f6915c8881f75", + "dbt_det/models/staging/stg_product_reviews.yml": "e7692c4d586af9ea57c9afbb421d5424be907fb28f4464bf82f75033b402ef48", + "dbt_det/models/staging/stg_products.sql": "7455128b10e0357213f26c209913a73df3f417be28ddd9d4dd8681390e7dcdab", + "dbt_det/models/staging/stg_products.yml": "56af441c9582505bf71fe86b1a6e79fc83db957b945db4f1a5ab1775aecb4e90", + "dbt_det/osi/det.yml": "638f0692719b23c6fb6d7e55a6cc09661bd3fda8c77eecd1a73c008058c58a31", + "dbt_det/profiles.yml": "c40b04badc35b941a8de1079c8d0a298f6446f0d5caf5bd50dbe3797ab2a2255", + "dbt_det/seeds/raw/__seed_config.yml": "03fbba2c4026678a9eef7cfe48aeaf3391a2919b05b1f67b2744ee0891ff199f", + "dbt_det/seeds/raw/customers.csv": "3443d991581aefaabc9648fd96302c948db616761e0b4137401beae0f7add2db", + "dbt_det/seeds/raw/order_items.csv": "81bcd163cc42c3474dfcf5201634164189bd3ba19c7cb05fce833c80b2d81a55", + "dbt_det/seeds/raw/orders.csv": "0c7f5319bb652bd73a6b89649825377e875c8c166086c299b971ee8d760e375f", + "dbt_det/seeds/raw/orders_history.csv": "755df44c63bbdfa922276db2382e5b9cfb9603e6fc02f5c72552b9aaa14e32e8", + "dbt_det/seeds/raw/product_reviews.csv": "f9f33d24a151c6ff680ce5c4521f87418284a4de045dcb0eacb11fba048b7324", + "dbt_det/seeds/raw/products.csv": "9274598e42761eea40a269c36524bc83eab320de4160c2fe0f0a5584fd8aebb3" + }, "preset-clean": { "dbt_det/EXPECTED_FAILURES.md": "14e2890fcde05b3d20274fb5a531e0f57b92864ccbd54c78ffbe1cea52c68942", "dbt_det/days/customers/day_000.csv": "69f9b44ab0cfdf2f6bca58f63f2be044b749c9f1cc80501a8355c4d3dc915104", diff --git a/tests/fixtures/metrics/ecommerce_training.metrics.yml b/tests/fixtures/metrics/ecommerce_training.metrics.yml new file mode 100644 index 0000000..d6bcedb --- /dev/null +++ b/tests/fixtures/metrics/ecommerce_training.metrics.yml @@ -0,0 +1,66 @@ +# Every aggregation and operator, over the training model: defects (orphans, nulls, +# invalid enum values, messy text, duplicate keys) and three days of changes. +model2data-metrics: 0.1.0 +model: ecommerce_training + +metrics: + delivered_revenue: + measure: orders.total_amount + agg: sum + where: + orders.status: delivered + open_orders: + count: orders + where: + orders.status: {in: [pending, processing], ne: cancelled} + average_order: + measure: orders.total_amount + agg: average + where: + orders.status: {not_in: [cancelled], is_null: false} + median_order: + measure: orders.total_amount + agg: median + smallest_order: + measure: orders.total_amount + agg: min + where: + orders.total_amount: {between: [100, 4000]} + largest_quantity: + measure: order_items.quantity + agg: max + rows_with_a_status: + measure: orders.status + agg: count + customers_ordering: + measure: orders.customer_id + agg: count_distinct + where: + customers.last_name: {ne: Smith} + items_of_recent_orders: + count: order_items + where: + orders.order_date: {gte: "2025-06-01", lt: "2026-01-01 00:00:00"} + products.price: {gt: 20.5} + any: + - customers.phone: {is_null: true} + - all: + - customers.country: {not_in: [United States]} + - order_items.quantity: {lte: 5} + item_value: + measure: order_items.unit_price + agg: sum + time: orders.updated_at + orders_count: + count: orders + order_items_quantity: + measure: order_items.quantity + agg: sum + per_order: + ratio: {numerator: item_value, denominator: orders_count} + odd_mix: + expression: (item_value - delivered_revenue) / (open_orders - open_orders) + -2 * 1.5 + units_per_order: + expression: order_items_quantity / orders_count + +infer: true diff --git a/tests/fixtures/metrics/shop.metrics.yml b/tests/fixtures/metrics/shop.metrics.yml new file mode 100644 index 0000000..37e5aca --- /dev/null +++ b/tests/fixtures/metrics/shop.metrics.yml @@ -0,0 +1,40 @@ +# Every kind of metric, column and join over shop.model2data.yml, for the dbt tests. +model2data-metrics: 0.1.0 +model: shop + +metrics: + paid_amount: + measure: orders.amount + where: + orders.status: paid + vip_lines: + count: lines + where: + customers.vip: true + customers.joined_on: {gte: "2020-01-01"} + tier_two_qty: + measure: lines.qty + where: + customers.tier: [2] + orders.placed_at: {lt: "2100-01-01 00:00:00"} + visitors_north: + measure: visits.people + agg: sum + where: + any: + - stores.region: {is_null: false} + - slots.capacity: {between: [0, 50]} + median_price: + measure: lines.price + agg: median + distinct_notes: + measure: orders.note + agg: count_distinct + log_max: + measure: logs.level + agg: max + paid_share: + ratio: {numerator: paid_amount, denominator: orders_amount} + format: percent + spread: + expression: (orders_amount - paid_amount) / orders_count diff --git a/tests/fixtures/metrics/shop.model2data.yml b/tests/fixtures/metrics/shop.model2data.yml new file mode 100644 index 0000000..3186834 --- /dev/null +++ b/tests/fixtures/metrics/shop.model2data.yml @@ -0,0 +1,75 @@ +# A small model for the metrics tests: every column kind, a two-step path +# (lines -> orders -> customers), two paths to one table (transfers), a +# composite foreign key (visits -> slots) and a table no metric can date (logs). +model2data: 0.4.0 +name: shop + +enums: + status: [open, paid, cancelled] + level: [1, 2] + +tables: + customers: + columns: + id: {type: bigint, pk: true} + email: {type: email, unique: true} + country: text + vip: boolean + tier: level + joined_on: date + + stores: + columns: + id: {type: bigint, pk: true} + region: text + + orders: + role: fact + columns: + id: {type: bigint, pk: true} + customer_id: {type: bigint, references: customers.id} + store_id: {type: bigint, references: stores.id} + status: status + amount: + type: numeric + not_null: true + measure: true + generate: {min: 10, max: 100} + placed_at: timestamp + note: text + + lines: + columns: + id: {type: bigint, pk: true} + order_id: {type: bigint, not_null: true, references: orders.id} + qty: {type: int, measure: sum} + price: {type: numeric, measure: average} + + transfers: + columns: + id: {type: bigint, pk: true} + sender_id: {type: bigint, references: customers.id} + receiver_id: {type: bigint, references: customers.id} + amount: numeric + + slots: + keys: + - {pk: [store_id, day]} + columns: + store_id: {type: bigint, references: stores.id} + day: date + capacity: int + + visits: + foreign_keys: + - {columns: [store_id, day], references: slots, to_columns: [store_id, day]} + columns: + id: {type: bigint, pk: true} + store_id: bigint + day: date + people: int + + logs: + columns: + id: int + level: int diff --git a/tests/fixtures/ossie/LICENSE b/tests/fixtures/ossie/LICENSE new file mode 100644 index 0000000..67db858 --- /dev/null +++ b/tests/fixtures/ossie/LICENSE @@ -0,0 +1,175 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. diff --git a/tests/fixtures/ossie/NOTICE b/tests/fixtures/ossie/NOTICE new file mode 100644 index 0000000..2deb079 --- /dev/null +++ b/tests/fixtures/ossie/NOTICE @@ -0,0 +1,2 @@ +Open Semantic Interchange +Copyright 2026 - Open Semantic Interchange diff --git a/tests/fixtures/ossie/VENDORED.md b/tests/fixtures/ossie/VENDORED.md new file mode 100644 index 0000000..949e49b --- /dev/null +++ b/tests/fixtures/ossie/VENDORED.md @@ -0,0 +1,10 @@ +# Vendored: the Apache Ossie 0.1.1 JSON Schema + +`osi-schema.json` is `core-spec/osi-schema.json` of [apache/ossie](https://github.com/apache/ossie) +(formerly Open Semantic Interchange) at tag `osi-0.1.1-rc1` (commit +`faf581054dcf7964d5fe0ceae7d6f415c8ce32a5`), whose `version.txt` and `core-spec/spec.md` say +0.1.1. It is copied unchanged. The tests validate every Ossie document model2data exports against +it (`tests/test_metrics_ossie.py`). + +It is licensed under the Apache License, Version 2.0: see `LICENSE` and `NOTICE` beside it, copied +from the same commit. It is test data only and is not shipped in the model2data package. diff --git a/tests/fixtures/ossie/osi-schema.json b/tests/fixtures/ossie/osi-schema.json new file mode 100644 index 0000000..a4a6f92 --- /dev/null +++ b/tests/fixtures/ossie/osi-schema.json @@ -0,0 +1,344 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/open-semantic-interchange/OSI/core-spec/osi-schema.json", + "title": "OSI Core Metadata Specification", + "description": "JSON Schema for validating OSI (Open Semantic Interoperability) semantic model definitions", + "type": "object", + "properties": { + "version": { + "type": "string", + "const": "0.1.1", + "description": "OSI specification version" + }, + "dialects": { + "type": "array", + "description": "Supported expression language dialects (enumeration definition)", + "items": { + "$ref": "#/$defs/Dialect" + } + }, + "vendors": { + "type": "array", + "description": "Supported vendors for custom extensions (enumeration definition)", + "items": { + "$ref": "#/$defs/Vendor" + } + }, + "semantic_model": { + "type": "array", + "description": "Collection of semantic model definitions", + "items": { + "$ref": "#/$defs/SemanticModel" + } + } + }, + "required": ["version", "semantic_model"], + "additionalProperties": false, + "$defs": { + "Dialect": { + "type": "string", + "enum": ["ANSI_SQL", "SNOWFLAKE", "MDX", "TABLEAU", "DATABRICKS", "MAQL"], + "description": "Supported SQL and expression language dialects" + }, + "Vendor": { + "type": "string", + "enum": ["COMMON", "SNOWFLAKE", "SALESFORCE", "DBT", "DATABRICKS", "GOODDATA"], + "description": "Supported vendors for custom extensions" + }, + "AIContext": { + "description": "Additional context for AI tools", + "oneOf": [ + { + "type": "string" + }, + { + "type": "object", + "properties": { + "instructions": { + "type": "string", + "description": "Instructions for AI on how to use this entity" + }, + "synonyms": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Alternative names and terms" + }, + "examples": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Sample questions or use cases" + } + }, + "additionalProperties": true + } + ] + }, + "CustomExtension": { + "type": "object", + "description": "Vendor-specific attributes for extensibility", + "properties": { + "vendor_name": { + "$ref": "#/$defs/Vendor" + }, + "data": { + "type": "string", + "description": "JSON string containing vendor-specific data" + } + }, + "required": ["vendor_name", "data"], + "additionalProperties": false + }, + "DialectExpression": { + "type": "object", + "description": "Expression in a specific dialect", + "properties": { + "dialect": { + "$ref": "#/$defs/Dialect" + }, + "expression": { + "type": "string", + "description": "SQL or dialect-specific expression" + } + }, + "required": ["dialect", "expression"], + "additionalProperties": false + }, + "Expression": { + "type": "object", + "description": "Expression definition with multi-dialect support", + "properties": { + "dialects": { + "type": "array", + "items": { + "$ref": "#/$defs/DialectExpression" + }, + "minItems": 1 + } + }, + "required": ["dialects"], + "additionalProperties": false + }, + "Dimension": { + "type": "object", + "description": "Dimension metadata", + "properties": { + "is_time": { + "type": "boolean", + "description": "Indicates if this is a time-based dimension for temporal filtering" + } + }, + "additionalProperties": false + }, + "Field": { + "type": "object", + "description": "Row-level attribute for grouping, filtering, and metric expressions", + "properties": { + "name": { + "type": "string", + "description": "Unique identifier for the field within the dataset" + }, + "expression": { + "$ref": "#/$defs/Expression" + }, + "dimension": { + "$ref": "#/$defs/Dimension" + }, + "label": { + "type": "string", + "description": "Label for categorization" + }, + "description": { + "type": "string", + "description": "Human-readable description" + }, + "ai_context": { + "$ref": "#/$defs/AIContext" + }, + "custom_extensions": { + "type": "array", + "items": { + "$ref": "#/$defs/CustomExtension" + } + } + }, + "required": ["name", "expression"], + "additionalProperties": false + }, + "Dataset": { + "type": "object", + "description": "Logical dataset representing a business entity (fact or dimension table)", + "properties": { + "name": { + "type": "string", + "description": "Unique identifier for the dataset" + }, + "source": { + "type": "string", + "description": "Reference to underlying physical table/view (database.schema.table) or query" + }, + "primary_key": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Primary key columns (single or composite)" + }, + "unique_keys": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "string" + } + }, + "description": "Array of unique key definitions (each can be single or composite)" + }, + "description": { + "type": "string", + "description": "Human-readable description" + }, + "ai_context": { + "$ref": "#/$defs/AIContext" + }, + "fields": { + "type": "array", + "items": { + "$ref": "#/$defs/Field" + } + }, + "custom_extensions": { + "type": "array", + "items": { + "$ref": "#/$defs/CustomExtension" + } + } + }, + "required": ["name", "source"], + "additionalProperties": false + }, + "Relationship": { + "type": "object", + "description": "Foreign key relationship between datasets", + "properties": { + "name": { + "type": "string", + "description": "Unique identifier for the relationship" + }, + "from": { + "type": "string", + "description": "Dataset on the many side of the relationship" + }, + "to": { + "type": "string", + "description": "Dataset on the one side of the relationship" + }, + "from_columns": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "description": "Foreign key columns in the 'from' dataset" + }, + "to_columns": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "description": "Primary/unique key columns in the 'to' dataset" + }, + "ai_context": { + "$ref": "#/$defs/AIContext" + }, + "custom_extensions": { + "type": "array", + "items": { + "$ref": "#/$defs/CustomExtension" + } + } + }, + "required": ["name", "from", "to", "from_columns", "to_columns"], + "additionalProperties": false + }, + "Metric": { + "type": "object", + "description": "Quantitative measure defined on business data", + "properties": { + "name": { + "type": "string", + "description": "Unique identifier for the metric" + }, + "expression": { + "$ref": "#/$defs/Expression" + }, + "description": { + "type": "string", + "description": "Human-readable description of what the metric measures" + }, + "ai_context": { + "$ref": "#/$defs/AIContext" + }, + "custom_extensions": { + "type": "array", + "items": { + "$ref": "#/$defs/CustomExtension" + } + } + }, + "required": ["name", "expression"], + "additionalProperties": false + }, + "SemanticModel": { + "type": "object", + "description": "Top-level container representing a complete semantic model", + "properties": { + "name": { + "type": "string", + "description": "Unique identifier for the semantic model" + }, + "description": { + "type": "string", + "description": "Human-readable description" + }, + "ai_context": { + "$ref": "#/$defs/AIContext" + }, + "datasets": { + "type": "array", + "items": { + "$ref": "#/$defs/Dataset" + }, + "minItems": 1, + "description": "Collection of logical datasets" + }, + "relationships": { + "type": "array", + "items": { + "$ref": "#/$defs/Relationship" + }, + "description": "Defines how datasets are connected" + }, + "metrics": { + "type": "array", + "items": { + "$ref": "#/$defs/Metric" + }, + "description": "Quantifiable measures spanning datasets" + }, + "custom_extensions": { + "type": "array", + "items": { + "$ref": "#/$defs/CustomExtension" + } + } + }, + "required": ["name", "datasets"], + "additionalProperties": false + } + } +} diff --git a/tests/test_determinism.py b/tests/test_determinism.py index 9cf26f3..5d067b6 100644 --- a/tests/test_determinism.py +++ b/tests/test_determinism.py @@ -62,6 +62,22 @@ def _case(case_id: str, model: Path, *args: str) -> None: _case("skew", EXAMPLES / "saas_platform.model2data.yml", "--skew", "0.8") _case("incremental-history", FIXTURES / "defects" / "hcol.model2data.yml", "--days", "3") _case("composite-keys", FIXTURES / "composite_only.dbml") +# Metrics add files (known values, Ossie, a dbt test per metric) and change none. +_SPEC = ROOT / "model2data" / "spec" / "examples" +_case( + "metrics", + _SPEC / "coffee_webshop.model2data.yml", + "--metrics", + str(_SPEC / "coffee_webshop.metrics.yml"), +) +_case( + "metrics-defects-days", + _TRAIN, + "--metrics", + str(FIXTURES / "metrics" / "ecommerce_training.metrics.yml"), + "--days", + "2", +) for _locale in ("en_US", "nl_BE", "de_DE", "fr_FR", "ja_JP"): _case(f"locale-{_locale}", _ECOM, "--locale", _locale) diff --git a/tests/test_guide.py b/tests/test_guide.py index f3a9d57..f61c6b6 100644 --- a/tests/test_guide.py +++ b/tests/test_guide.py @@ -101,9 +101,14 @@ def test_every_command_a_page_names_exists(topic): @pytest.mark.parametrize("topic", TOPICS) def test_every_option_a_page_names_exists(topic): + commands = list(_commands().values()) + # `metrics` is a group: its subcommands' options count too. + for command in list(commands): + if isinstance(command, click.Group): + commands.extend(command.commands.values()) options = { opt - for command in _commands().values() + for command in commands for param in command.params for opt in [*param.opts, *param.secondary_opts] } | {"--help"} diff --git a/tests/test_metrics_cli.py b/tests/test_metrics_cli.py new file mode 100644 index 0000000..ab9eed9 --- /dev/null +++ b/tests/test_metrics_cli.py @@ -0,0 +1,316 @@ +"""The CLI's metrics: `validate` on a metrics file, `generate --metrics`, `metrics list|export`.""" + +import hashlib +import json +import os +import shutil +from pathlib import Path + +import pytest +import yaml +from typer.testing import CliRunner + +from model2data.cli import app +from model2data.cli import main as generate_cli + +ROOT = Path(__file__).resolve().parent.parent +SPEC = ROOT / "model2data" / "spec" / "examples" +FIXTURES = ROOT / "tests" / "fixtures" / "metrics" +COFFEE = SPEC / "coffee_webshop.model2data.yml" +COFFEE_METRICS = SPEC / "coffee_webshop.metrics.yml" +TRAINING = ROOT / "examples" / "ecommerce_training.model2data.yml" +TRAINING_METRICS = FIXTURES / "ecommerce_training.metrics.yml" + +runner = CliRunner() + + +def _invoke(args, cwd: Path): + previous = Path.cwd() + os.chdir(cwd) + try: + return runner.invoke(app, [str(arg) for arg in args]) + finally: + os.chdir(previous) + + +def _hashes(directory: Path) -> dict[str, str]: + return { + path.relative_to(directory).as_posix(): hashlib.sha256(path.read_bytes()).hexdigest() + for path in sorted(directory.rglob("*")) + if path.is_file() + } + + +# --------------------------------------------------------------------------- +# validate +# --------------------------------------------------------------------------- +def test_validate_checks_a_metrics_file_against_the_model_beside_it(tmp_path): + result = _invoke(["validate", COFFEE_METRICS], tmp_path) + assert result.exit_code == 0, result.output + assert ( + "✅ coffee_webshop.metrics.yml conforms to metrics spec 0.1.0, checked against " + "coffee_webshop.model2data.yml." in result.output + ) + + +def test_validate_takes_the_model_from_an_option(tmp_path): + metrics = tmp_path / "mine.metrics.yml" + shutil.copy(COFFEE_METRICS, metrics) + result = _invoke(["validate", metrics, "--model", COFFEE], tmp_path) + assert result.exit_code == 0, result.output + assert "checked against coffee_webshop.model2data.yml" in result.output + + +def test_validate_without_a_model_checks_the_file_alone_and_says_so(tmp_path): + metrics = tmp_path / "lonely.metrics.yml" + metrics.write_text("model2data-metrics: 0.1.0\nmodel: lonely\nmetrics:\n n: {count: t}\n") + result = _invoke(["validate", metrics], tmp_path) + assert result.exit_code == 0, result.output + assert ( + "no model to check it against: there is no lonely.model2data.yml beside it" in result.output + ) + assert "✅ lonely.metrics.yml conforms to metrics spec 0.1.0." in result.output + + +def test_validate_reports_a_metrics_files_errors(tmp_path): + metrics = tmp_path / "coffee_webshop.metrics.yml" + metrics.write_text( + "model2data-metrics: 0.1.0\nmodel: coffee_webshop\nmetrics:\n" + " m: {measure: orders.status, agg: sum}\n" + ) + shutil.copy(COFFEE, tmp_path / COFFEE.name) + result = _invoke(["validate", metrics], tmp_path) + assert result.exit_code == 1 + assert "❌ coffee_webshop.metrics.yml: 1 error" in result.output + assert "metrics.m.agg (line 4): sum needs a numeric column" in result.output + github = _invoke(["validate", metrics, "--format", "github"], tmp_path) + assert "::error file=" in github.output and "line=4" in github.output + + +def test_validate_refuses_a_metrics_file_whose_model_does_not_conform(tmp_path): + (tmp_path / "bad.model2data.yml").write_text("model2data: 0.4.0\ntables: {}\n") + metrics = tmp_path / "bad.metrics.yml" + metrics.write_text("model2data-metrics: 0.1.0\nmodel: bad\n") + result = _invoke(["validate", metrics], tmp_path) + assert result.exit_code == 1 + assert "its model, bad.model2data.yml, does not conform" in result.output + + +def test_validate_mixes_models_and_metrics_files(tmp_path): + result = _invoke(["validate", COFFEE, COFFEE_METRICS], tmp_path) + assert result.exit_code == 0, result.output + assert "conforms to spec 0.2.0" in result.output + assert "conforms to metrics spec 0.1.0" in result.output + assert "✅ 2 model files conform." in result.output + + +# --------------------------------------------------------------------------- +# generate --metrics +# --------------------------------------------------------------------------- +COMMON = ["--seed", "7", "--as-of", "2026-03-15", "--rows", "60", "--name", "shop"] + + +def test_metrics_only_add_files(tmp_path): + plain, with_metrics = tmp_path / "plain", tmp_path / "metrics" + plain.mkdir() + with_metrics.mkdir() + first = _invoke(["generate", "-f", COFFEE, *COMMON], plain) + assert first.exit_code == 0, first.output + second = _invoke(["generate", "-f", COFFEE, *COMMON, "--metrics", COFFEE_METRICS], with_metrics) + assert second.exit_code == 0, second.output + before, after = _hashes(plain), _hashes(with_metrics) + assert {path: after[path] for path in before} == before + added = sorted(set(after) - set(before)) + assert added == sorted( + [ + "dbt_shop/coffee_webshop.metrics.yml", + "dbt_shop/metric_values.json", + "dbt_shop/osi/shop.yml", + *( + f"dbt_shop/data-tests/metrics/metric_{name}.sql" + for name in [ + "revenue", + "orders", + "order_lines", + "average_order_value", + "web_revenue", + "web_share", + "coffee_units_sold", + "orders_total_amount", + "orders_count", + "order_items_quantity", + "order_items_count", + ] + ), + ] + ) + assert "📏 Computing each metric's known value and its dbt test..." in second.output + assert ( + "Metrics: 11 (metric_values.json, osi/shop.yml, data-tests/metrics/)" + in (second.output) + ) + values = json.loads((with_metrics / "dbt_shop" / "metric_values.json").read_text()) + assert values["run"] == {"seed": 7, "as_of": "2026-03-15"} + assert list(values["metrics"])[:2] == ["revenue", "orders"] + ossie = yaml.safe_load((with_metrics / "dbt_shop" / "osi" / "shop.yml").read_text()) + assert ossie["version"] == "0.1.1" + + +def test_generate_with_metrics_twice_writes_the_same_bytes(tmp_path): + runs = [] + for name in ("a", "b"): + (tmp_path / name).mkdir() + result = _invoke( + ["generate", "-f", COFFEE, *COMMON, "--metrics", COFFEE_METRICS], tmp_path / name + ) + assert result.exit_code == 0, result.output + runs.append(_hashes(tmp_path / name)) + assert runs[0] == runs[1] + + +def test_generate_refuses_a_metrics_file_with_errors_before_writing_anything(tmp_path): + metrics = tmp_path / "broken.metrics.yml" + metrics.write_text("model2data-metrics: 0.1.0\nmodel: someone_else\n") + result = _invoke(["generate", "-f", COFFEE, *COMMON, "--metrics", metrics], tmp_path) + assert result.exit_code == 1 + assert "the metrics are for model 'someone_else'" in result.output + assert not (tmp_path / "dbt_shop").exists() + + +def test_generate_prints_a_metrics_files_warnings(tmp_path): + metrics = tmp_path / "w.metrics.yml" + metrics.write_text( + "model2data-metrics: 0.1.0\nmodel: coffee_webshop\n" + "metrics:\n big: {count: orders, where: {orders.total_amount: {gt: 9000}}}\n" + ) + result = _invoke(["generate", "-f", COFFEE, *COMMON, "--metrics", metrics], tmp_path) + assert result.exit_code == 0, result.output + assert "⚠️ w.metrics.yml: 1 warning" in result.output + assert "can never match" in result.output + + +def test_generate_as_a_function_with_days_and_defects(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + generate_cli( + file=TRAINING, + rows=40, + seed=5, + name="training", + force=True, + days=2, + metrics_file=TRAINING_METRICS, + ) + project = tmp_path / "dbt_training" + values = json.loads((project / "metric_values.json").read_text()) + assert values["run"] == {"seed": 5, "as_of": "2026-01-01"} + assert (project / "data-tests" / "metrics" / "metric_odd_mix.sql").exists() + assert values["metrics"]["odd_mix"]["value"] is None + + +def test_generate_with_no_metrics_in_the_file_still_says_so(tmp_path): + metrics = tmp_path / "none.metrics.yml" + metrics.write_text("model2data-metrics: 0.1.0\nmodel: coffee_webshop\ninfer: false\n") + result = _invoke(["generate", "-f", COFFEE, *COMMON, "--metrics", metrics], tmp_path) + assert result.exit_code == 0, result.output + assert "Metrics: 0" in result.output + + +# --------------------------------------------------------------------------- +# metrics list / export +# --------------------------------------------------------------------------- +def test_metrics_list(tmp_path): + result = _invoke(["metrics", "list", "-f", COFFEE, "--metrics", COFFEE_METRICS], tmp_path) + assert result.exit_code == 0, result.output + lines = result.output.splitlines() + assert lines[0].split() == [ + "revenue", + "simple", + "file", + "sum", + "of", + "orders.total_amount,", + "filtered,", + "by", + "orders.order_date", + ] + assert any(line.split()[:3] == ["order_lines", "count", "file"] for line in lines) + assert any( + line.split()[:4] == ["average_order_value", "ratio", "file", "revenue"] for line in lines + ) + assert any(line.split()[:3] == ["web_share", "derived", "file"] for line in lines) + assert any(line.split()[:3] == ["orders_count", "count", "inferred"] for line in lines) + + +def test_metrics_list_without_a_file_lists_the_inferred_ones(tmp_path): + result = _invoke(["metrics", "list", "-f", COFFEE], tmp_path) + assert [line.split()[0] for line in result.output.splitlines()] == [ + "orders_total_amount", + "orders_count", + "order_items_quantity", + "order_items_count", + ] + bare = _invoke( + ["metrics", "list", "-f", ROOT / "examples" / "ecommerce.model2data.yml"], tmp_path + ) + assert bare.exit_code == 0 + assert "No metrics" in bare.output + + +def test_metrics_export_prints_ossie(tmp_path): + result = _invoke(["metrics", "export", "-f", COFFEE, "--metrics", COFFEE_METRICS], tmp_path) + assert result.exit_code == 0, result.output + assert result.output.startswith("# Apache Ossie (Open Semantic Interchange) 0.1.1") + assert yaml.safe_load(result.output)["semantic_model"][0]["name"] == "coffee_webshop" + + +def test_metrics_export_writes_a_file(tmp_path): + out = tmp_path / "coffee.osi.yml" + args = [ + "metrics", + "export", + "-f", + COFFEE, + "--metrics", + COFFEE_METRICS, + "--to", + "OSSIE", + "-o", + out, + ] + result = _invoke(args, tmp_path) + assert result.exit_code == 0, result.output + assert f"✅ Wrote {out} (Apache Ossie 0.1.1, 11 metrics)" in result.output + assert "ℹ️ Not expressible in Ossie 0.1.1 itself" in result.output + again = _invoke(args, tmp_path) + assert again.exit_code == 1 and "already exists" in again.output + assert _invoke([*args, "--force"], tmp_path).exit_code == 0 + + +def test_metrics_export_without_losses_prints_none(tmp_path): + model = tmp_path / "one.model2data.yml" + model.write_text("model2data: 0.4.0\nname: one\ntables:\n t:\n columns:\n id: int\n") + result = _invoke(["metrics", "export", "-f", model, "-o", tmp_path / "o.yml"], tmp_path) + assert result.exit_code == 0, result.output + assert "Not expressible" not in result.output + + +def test_metrics_export_refuses_an_unknown_format(tmp_path): + result = _invoke(["metrics", "export", "-f", COFFEE, "--to", "metricflow"], tmp_path) + assert result.exit_code == 2 + assert "Choose one of: ossie" in result.output + + +def test_metrics_with_no_command_shows_help(tmp_path): + result = _invoke(["metrics"], tmp_path) + assert "list" in result.output and "export" in result.output + + +@pytest.mark.parametrize("command", ["list", "export"]) +def test_metrics_commands_refuse_a_broken_metrics_file(tmp_path, command): + metrics = tmp_path / "x.metrics.yml" + metrics.write_text( + "model2data-metrics: 0.1.0\nmodel: coffee_webshop\nmetrics:\n m: {count: nope}\n" + ) + result = _invoke(["metrics", command, "-f", COFFEE, "--metrics", metrics], tmp_path) + assert result.exit_code == 1 + assert "metrics.m.count" in result.output diff --git a/tests/test_metrics_dbt.py b/tests/test_metrics_dbt.py new file mode 100644 index 0000000..7c547e3 --- /dev/null +++ b/tests/test_metrics_dbt.py @@ -0,0 +1,123 @@ +"""The metric tests against dbt itself: `dbt build` computes every metric and gets its known value. + +`metric_values.json` is computed in Python, on DuckDB, from the frames the run +wrote. Each generated `data-tests/metrics/metric_.sql` computes the same +metric over the project's staging models, as the warehouse loaded them from +the seeds. These cases run a real `dbt build` and require every metric test to +pass: on the reference example, on a model with every kind of column and join, +and on the training model with its defects (orphans, nulls, invalid values, +messy text, duplicate keys) over several days. One more edits a known value +and requires its test to fail, so a passing test is known to check something. + +Run with the project's own dbt first on PATH (`PATH="$PWD/.venv/bin:$PATH"`). +""" + +from __future__ import annotations + +import json +import re +import shutil +import subprocess +from pathlib import Path + +import pytest + +from model2data.cli import main as generate_cli + +DBT = shutil.which("dbt") +ROOT = Path(__file__).resolve().parent.parent +SPEC = ROOT / "model2data" / "spec" / "examples" +FIXTURES = ROOT / "tests" / "fixtures" / "metrics" + +pytestmark = pytest.mark.skipif(DBT is None, reason="dbt CLI not found on PATH") + + +def _build(project: Path, *args: str) -> dict[str, str]: + """Each test's status after `dbt build`; a model or seed that does not succeed fails here.""" + result = subprocess.run( + [str(DBT), "build", "--profiles-dir", ".", *args], + cwd=project, + capture_output=True, + text=True, + timeout=300, + ) + run_results = project / "target" / "run_results.json" + assert run_results.exists(), result.stdout + result.stderr + statuses = {} + for node in json.loads(run_results.read_text())["results"]: + unique_id, status = node["unique_id"], node["status"] + if unique_id.startswith("test."): + statuses[unique_id.split(".")[2]] = status + else: + assert status == "success", f"{unique_id} {status}:\n{result.stdout}" + return statuses + + +def _metric_tests(statuses: dict[str, str]) -> dict[str, str]: + return {name: status for name, status in statuses.items() if name.startswith("metric_")} + + +@pytest.mark.parametrize( + "model, metrics, options", + [ + pytest.param( + SPEC / "coffee_webshop.model2data.yml", + SPEC / "coffee_webshop.metrics.yml", + {"rows": 300, "seed": 7}, + id="coffee-example", + ), + pytest.param( + FIXTURES / "shop.model2data.yml", + FIXTURES / "shop.metrics.yml", + {"rows": 80, "seed": 3}, + id="every-kind-and-join", + ), + pytest.param( + ROOT / "examples" / "ecommerce_training.model2data.yml", + FIXTURES / "ecommerce_training.metrics.yml", + {"rows": 60, "days": 3}, + id="training-defects-and-days", + ), + pytest.param( + ROOT / "examples" / "ecommerce_training.model2data.yml", + FIXTURES / "ecommerce_training.metrics.yml", + {"rows": 60, "days": 2, "defects_preset": "messy"}, + id="messy-preset", + ), + ], +) +def test_every_metric_test_passes_in_dbt_build(tmp_path, monkeypatch, model, metrics, options): + monkeypatch.chdir(tmp_path) + generate_cli( + file=model, force=True, adapter="duckdb", name="metrics", metrics_file=metrics, **options + ) + project = tmp_path / "dbt_metrics" + values = json.loads((project / "metric_values.json").read_text()) + statuses = _metric_tests(_build(project)) + assert set(statuses) == {f"metric_{name}" for name in values["metrics"]} + assert all(status == "pass" for status in statuses.values()), statuses + + +def test_a_wrong_known_value_fails_its_test(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + generate_cli( + file=SPEC / "coffee_webshop.model2data.yml", + rows=50, + seed=1, + force=True, + name="metrics", + metrics_file=SPEC / "coffee_webshop.metrics.yml", + ) + project = tmp_path / "dbt_metrics" + test = project / "data-tests" / "metrics" / "metric_revenue.sql" + text = test.read_text() + # Move the known value by twice the tolerance. + text = re.sub( + r"abs\(value - ([0-9.]+)\) > ([0-9.]+)", + lambda m: f"abs(value - ({m.group(1)} + 2 * {m.group(2)})) > {m.group(2)}", + text, + ) + test.write_text(text) + statuses = _metric_tests(_build(project)) + assert statuses.pop("metric_revenue") == "fail" + assert set(statuses.values()) == {"pass"} diff --git a/tests/test_metrics_expression.py b/tests/test_metrics_expression.py new file mode 100644 index 0000000..a82a118 --- /dev/null +++ b/tests/test_metrics_expression.py @@ -0,0 +1,76 @@ +"""A derived metric's expression: the grammar, the arithmetic, the SQL (metrics spec, "Metrics").""" + +from decimal import Decimal + +import pytest + +from model2data.metrics.expression import ( + BinaryOp, + ExpressionError, + Name, + Negate, + Number, + evaluate, + names, + parse, + to_sql, +) + + +def test_precedence_and_left_association(): + assert parse("a - b - c") == BinaryOp("-", BinaryOp("-", Name("a"), Name("b")), Name("c")) + assert parse("a + b * c") == BinaryOp("+", Name("a"), BinaryOp("*", Name("b"), Name("c"))) + assert parse("(a + b) / 2.5") == BinaryOp( + "/", BinaryOp("+", Name("a"), Name("b")), Number(Decimal("2.5")) + ) + + +def test_unary_signs(): + assert parse("-a") == Negate(Name("a")) + assert parse("+a") == Name("a") + assert parse("--a") == Negate(Negate(Name("a"))) + assert parse("a * -2") == BinaryOp("*", Name("a"), Negate(Number(Decimal(2)))) + + +def test_whitespace_is_ignored(): + assert parse(" a\t+\nb ") == parse("a+b") + + +@pytest.mark.parametrize( + "text, message", + [ + ("", "the expression is empty"), + ("a +", "ends where a metric name or number should be"), + ("a b", "unexpected 'b' at character 3"), + ("a)", "unexpected ')' at character 2"), + ("(a", "the parenthesis at character 1 is never closed"), + ("a * )", "unexpected ')' at character 5: a metric name or number goes here"), + ("Revenue", "unexpected 'R' at character 1"), + ("a % b", "unexpected '%' at character 3"), + ("1.", "unexpected '.' at character 2"), + ("a.b", "unexpected '.' at character 2"), + ], +) +def test_what_does_not_parse(text, message): + with pytest.raises(ExpressionError) as raised: + parse(text) + assert message in str(raised.value) + + +def test_names_in_order_once_each(): + assert names(parse("b / (a + b) - -c * 2")) == ["b", "a", "c"] + + +def test_evaluate_in_decimals_with_sql_nulls(): + node = parse("(a - b) / c * 100 + -1") + values = {"a": Decimal(30), "b": Decimal(10), "c": Decimal(40)} + assert evaluate(node, values) == Decimal(49) + assert evaluate(node, {**values, "c": Decimal(0)}) is None + assert evaluate(node, {**values, "a": None}) is None + assert evaluate(parse("-a"), {"a": None}) is None + assert evaluate(parse("a / 3"), {"a": Decimal(1)}) == Decimal(1) / Decimal(3) + + +def test_to_sql_parenthesises_and_never_divides_by_zero(): + sql = to_sql(parse("a - b / c * 2.50 + -d"), lambda name: f"[{name}]") + assert sql == "(([a] - (([b] / nullif([c], 0)) * 2.50)) + (-[d]))" diff --git a/tests/test_metrics_ossie.py b/tests/test_metrics_ossie.py new file mode 100644 index 0000000..41d8a7e --- /dev/null +++ b/tests/test_metrics_ossie.py @@ -0,0 +1,235 @@ +"""The Ossie 0.1.1 export: the mapping, the lossiness report, and the vendored schema. + +`fixtures/ossie/osi-schema.json` is Apache Ossie's own JSON Schema for 0.1.1 +(see `fixtures/ossie/VENDORED.md`); every document exported here validates +against it. +""" + +import json +from pathlib import Path + +import jsonschema +import pytest +import yaml + +from model2data.metrics import OSSIE_VERSION, load, resolve, to_ossie +from model2data.model import load as load_model + +ROOT = Path(__file__).resolve().parent.parent +FIXTURES = ROOT / "tests" / "fixtures" +SPEC = ROOT / "model2data" / "spec" / "examples" +SHOP = load_model(FIXTURES / "metrics" / "shop.model2data.yml") +OSSIE_SCHEMA = json.loads((FIXTURES / "ossie" / "osi-schema.json").read_text(encoding="utf-8")) + + +def _valid(document: dict) -> None: + jsonschema.Draft202012Validator(OSSIE_SCHEMA).validate(document) + + +def _coffee(): + model = load_model(SPEC / "coffee_webshop.model2data.yml") + return resolve(model, load(SPEC / "coffee_webshop.metrics.yml", model)) + + +def test_the_vendored_schema_is_ossie_0_1_1(): + assert OSSIE_SCHEMA["properties"]["version"]["const"] == OSSIE_VERSION == "0.1.1" + jsonschema.Draft202012Validator.check_schema(OSSIE_SCHEMA) + + +@pytest.mark.parametrize( + "model_path", + sorted((ROOT / "examples").glob("*.model2data.yml")) + [SPEC / "coffee_webshop.model2data.yml"], + ids=lambda path: path.name, +) +def test_every_example_model_exports_a_valid_document(model_path): + export = to_ossie(resolve(load_model(model_path))) + _valid(export.document) + # And the YAML reads back as the same document. + assert yaml.safe_load(export.to_yaml()) == json.loads(json.dumps(export.document)) + + +def test_the_coffee_example_exports_a_valid_document(): + export = to_ossie(_coffee()) + _valid(export.document) + _valid(yaml.safe_load(export.to_yaml())) + + +def test_the_semantic_model_and_its_datasets(): + document = to_ossie(_coffee()).document + assert document["version"] == "0.1.1" + (model,) = document["semantic_model"] + assert model["name"] == "coffee_webshop" + assert model["description"].startswith("A Belgian specialty-coffee roaster") + products = next(d for d in model["datasets"] if d["name"] == "products") + assert products["source"] == "staging.stg_products" + assert products["primary_key"] == ["id"] + assert products["unique_keys"] == [["sku"]] + assert products["description"] == "The catalogue: one row per sellable product" + fields = {f["name"]: f for f in products["fields"]} + assert list(fields)[:3] == ["id", "name", "sku"] + assert fields["id"] == { + "name": "id", + "expression": {"dialects": [{"dialect": "ANSI_SQL", "expression": "id"}]}, + } + assert fields["name"]["dimension"] == {"is_time": False} + assert fields["name"]["label"] == "Product" # from the metrics file's dimensions + assert fields["created_at"]["dimension"] == {"is_time": True} + assert json.loads(fields["roast"]["custom_extensions"][0]["data"]) == { + "model2data": {"values": ["light", "medium", "dark"]} + } + customers = next(d for d in model["datasets"] if d["name"] == "customers") + assert "dimension" not in {f["name"]: f for f in customers["fields"]}["email"] + + +def test_relationships_run_from_the_many_side(): + model = to_ossie(_coffee()).document["semantic_model"][0] + assert model["relationships"][0] == { + "name": "orders_customer_id_to_customers", + "from": "orders", + "to": "customers", + "from_columns": ["customer_id"], + "to_columns": ["id"], + } + + +def test_metric_expressions(): + metrics = {m["name"]: m for m in to_ossie(_coffee()).document["semantic_model"][0]["metrics"]} + + def sql(name: str) -> str: + (dialect,) = metrics[name]["expression"]["dialects"] + assert dialect["dialect"] == "ANSI_SQL" + return dialect["expression"] + + revenue = "SUM(CASE WHEN orders.status IN ('paid', 'shipped', 'delivered') THEN orders.total_amount END)" + assert sql("revenue") == revenue + assert sql("orders") == "COUNT(DISTINCT orders.id)" + assert sql("order_lines") == "COUNT(order_items.id)" + assert sql("average_order_value") == f"({revenue}) / NULLIF(COUNT(DISTINCT orders.id), 0)" + assert sql("web_share").startswith("(SUM(CASE WHEN (orders.status IN") + assert "/ NULLIF(SUM(" in sql("web_share") and "100" not in sql("web_share") + assert sql("coffee_units_sold") == ( + "SUM(CASE WHEN (orders.status <> 'cancelled' AND products.category IN ('beans', 'ground', " + "'capsules') AND customers.phone IS NOT NULL AND (orders.channel = 'web' OR " + "order_items.quantity >= 3)) THEN order_items.quantity END)" + ) + assert metrics["revenue"]["description"].startswith("Order value") + assert metrics["revenue"]["ai_context"].startswith("What customers paid for.") + assert json.loads(metrics["revenue"]["custom_extensions"][0]["data"]) == { + "model2data": {"label": "Revenue", "format": "currency", "time": "orders.order_date"} + } + assert metrics["revenue"]["custom_extensions"][0]["vendor_name"] == "COMMON" + + +def test_the_lossiness_report(): + export = to_ossie(_coffee()) + report = {(loss.subject, loss.item) for loss in export.lossiness} + assert ("metric coffee_units_sold", "join path") in report + assert any( + subject.startswith("metrics revenue, orders") and item == "label" + for subject, item in report + ) + assert ("metrics revenue, average_order_value, web_share", "format") in report + web_share = next( + m for m in export.document["semantic_model"][0]["metrics"] if m["name"] == "web_share" + ) + # percent is passed on as the fraction's format, the value never multiplied + assert ( + json.loads(web_share["custom_extensions"][0]["data"])["model2data"]["format"] == "percent" + ) + assert any(item == "time" for _, item in report) + assert any(item == "values" and "products.category" in subject for subject, item in report) + text = export.to_yaml() + assert text.startswith("# Apache Ossie (Open Semantic Interchange) 0.1.1") + assert "# - metric coffee_units_sold: join path: reaches customers through orders" in text + assert 'version: "0.1.1"' in text + + +def test_roles_grain_one_to_one_and_awkward_names(): + model = load_model( + """ +model2data: 0.4.0 +name: odd +tables: + user accounts: + role: dimension + columns: + id: {type: int, pk: true} + Display Name: text + profiles: + columns: + user_id: {type: int, references: {to: user accounts.id, one_to_one: true}} + is_public: boolean + events: + role: fact + grain: [user_id, at] + columns: + user_id: {type: int, references: user accounts.id} + at: timestamp + amount: {type: numeric, measure: true} +""" + ) + export = to_ossie(resolve(model), source=lambda entity: f"db.analytics.{entity.name}") + _valid(export.document) + datasets = {d["name"]: d for d in export.document["semantic_model"][0]["datasets"]} + assert datasets["user_accounts"]["source"] == "db.analytics.user_accounts" + assert datasets["user_accounts"]["fields"][1]["expression"]["dialects"][0]["expression"] == ( + '"Display Name"' + ) + assert json.loads(datasets["events"]["custom_extensions"][0]["data"]) == { + "model2data": {"role": "fact", "grain": ["user_id", "at"]} + } + relationship = export.document["semantic_model"][0]["relationships"][0] + assert json.loads(relationship["custom_extensions"][0]["data"]) == { + "model2data": {"one_to_one": True} + } + subjects = [(loss.subject, loss.item) for loss in export.lossiness] + assert ("dataset user_accounts", "role") in subjects + assert ("dataset events", "role and grain") in subjects + assert ("relationship profiles_user_id_to_user_accounts", "one_to_one") in subjects + # events has no primary key: its row count cannot name whose rows it counts + metrics_out = {m["name"]: m for m in export.document["semantic_model"][0]["metrics"]} + assert metrics_out["events_count"]["expression"]["dialects"][0]["expression"] == "COUNT(*)" + assert ("metric events_count", "row count") in subjects + assert metrics_out["events_amount"]["expression"]["dialects"][0]["expression"] == ( + "SUM(events.amount)" + ) + + +def test_a_filtered_row_count_and_literals_of_every_kind(): + metrics = load( + """ +model2data-metrics: 0.1.0 +model: shop +infer: false +metrics: + m: + count: lines + where: + customers.vip: true + customers.joined_on: {between: ["2025-01-01", "2025-12-31"]} + orders.placed_at: {lt: "2026-01-01 00:00:00"} + orders.note: {in: ["it's"]} + lines.price: {gte: 1.5, lte: 2} + orders.status: {not_in: [cancelled], is_null: false} +""", + SHOP, + ) + export = to_ossie(resolve(SHOP, metrics)) + _valid(export.document) + (metric,) = export.document["semantic_model"][0]["metrics"] + assert metric["expression"]["dialects"][0]["expression"] == ( + "COUNT(CASE WHEN (customers.vip = TRUE AND customers.joined_on BETWEEN DATE '2025-01-01' " + "AND DATE '2025-12-31' AND orders.placed_at < TIMESTAMP '2026-01-01 00:00:00' AND " + "orders.note IN ('it''s') AND (lines.price >= 1.5 AND lines.price <= 2) AND " + "(orders.status NOT IN ('cancelled') AND orders.status IS NOT NULL)) THEN lines.id END)" + ) + + +def test_a_model_without_relationships_or_metrics(): + model = load_model("model2data: 0.4.0\nname: one\ntables:\n t:\n columns:\n id: int\n") + document = to_ossie(resolve(model)).document + _valid(document) + assert "relationships" not in document["semantic_model"][0] + assert "metrics" not in document["semantic_model"][0] + assert to_ossie(resolve(model)).lossiness == [] + assert "Not expressible" not in to_ossie(resolve(model)).to_yaml() diff --git a/tests/test_metrics_reader.py b/tests/test_metrics_reader.py new file mode 100644 index 0000000..7a581b0 --- /dev/null +++ b/tests/test_metrics_reader.py @@ -0,0 +1,753 @@ +"""Reading a metrics file: the YAML profile, the schema, and every check, each with its path. + +Each test names the rule of `model2data/spec/metrics/README.md` it pins. The +model is `fixtures/metrics/shop.model2data.yml`. +""" + +from pathlib import Path +from textwrap import dedent + +import pytest + +from model2data.metrics import ( + Condition, + MetricsError, + Where, + from_dict, + is_metrics_file, + load, + sibling_model, + validate, +) +from model2data.metrics.reader import metrics_stem +from model2data.model import load as load_model + +FIXTURES = Path(__file__).resolve().parent / "fixtures" / "metrics" +SHOP = load_model(FIXTURES / "shop.model2data.yml") +HEAD = "model2data-metrics: 0.1.0\nmodel: shop\n" + + +def _issues(body: str, model=SHOP): + """`(path, message, severity)` of every issue in a document of `HEAD` + `body`.""" + return [(i.path, i.message, i.severity) for i in validate(HEAD + dedent(body), model)] + + +def _errors(body: str, model=SHOP): + return [ + (path, message) for path, message, severity in _issues(body, model) if severity == "error" + ] + + +def _warnings(body: str, model=SHOP): + return [ + (path, message) for path, message, severity in _issues(body, model) if severity == "warning" + ] + + +def _one_error(body: str, model=SHOP) -> tuple[str, str]: + errors = _errors(body, model) + assert len(errors) == 1, errors + return errors[0] + + +# --------------------------------------------------------------------------- +# A file that conforms +# --------------------------------------------------------------------------- +def test_a_file_with_every_kind_reads_into_typed_metrics(): + metrics = load( + HEAD + + dedent( + """ + metrics: + revenue: + label: Revenue + description: Paid orders. + ai_context: Money in. + measure: orders.amount + where: + orders.status: paid + time: orders.placed_at + format: currency + x-owner: finance + orders_n: + count: orders + share: + ratio: {numerator: revenue, denominator: orders_amount, x-note: kept} + format: percent + net: + expression: revenue - 2 + dimensions: + customers.country: {label: Country, description: Where they live} + customers.vip: false + stores.region: true + infer: true + x-team: data + """ + ), + SHOP, + ) + assert list(metrics.metrics) == ["revenue", "orders_n", "share", "net"] + revenue = metrics.metrics["revenue"] + assert revenue.kind == "simple" and revenue.measure == "orders.amount" and revenue.agg is None + assert revenue.where == Where([Condition("orders.status", {"eq": "paid"}, written="value")]) + assert revenue.extensions == {"x-owner": "finance"} + assert metrics.metrics["orders_n"].kind == "count" + share = metrics.metrics["share"] + assert share.kind == "ratio" and share.ratio is not None + assert share.ratio.extensions == {"x-note": "kept"} + assert metrics.metrics["net"].kind == "derived" + assert metrics.dimensions["customers.country"].label == "Country" + assert metrics.dimensions["customers.vip"].offered is False + assert metrics.dimensions["stores.region"].offered is True + assert metrics.extensions == {"x-team": "data"} + assert metrics.warnings == [] + + +def test_the_reference_example_conforms_against_its_model(): + spec = Path(__file__).resolve().parent.parent / "model2data" / "spec" / "examples" + model = load_model(spec / "coffee_webshop.model2data.yml") + metrics = load(spec / "coffee_webshop.metrics.yml", model) + assert metrics.model == "coffee_webshop" + assert metrics.warnings == [] + + +def test_json_reads_as_json(): + text = '{"model2data-metrics": "0.1.0", "model": "shop", "metrics": {"n": {"count": "orders"}}}' + assert load(text, SHOP).metrics["n"].count == "orders" + + +def test_a_file_path_is_read(tmp_path): + path = tmp_path / "shop.metrics.json" + path.write_text('{"model2data-metrics": "0.1.0", "model": "shop"}', encoding="utf-8") + assert load(path).model == "shop" + assert load(str(path)).model == "shop" + + +def test_from_dict_takes_a_parsed_document(): + metrics = from_dict({"model2data-metrics": "0.1.0", "model": "shop", "infer": False}, SHOP) + assert metrics.infer is False and metrics.metrics == {} + + +def test_the_conditions_keep_how_they_were_written(): + metrics = load( + HEAD + + dedent( + """ + metrics: + m: + count: orders + where: + orders.status: [open, paid] + orders.amount: {gte: 20, lt: 50, x-why: range} + any: + - orders.note: {is_null: true} + - all: + - orders.status: paid + - customers.vip: true + """ + ), + SHOP, + ) + where = metrics.metrics["m"].where + assert where is not None + first, second, group = where.terms + assert first == Condition("orders.status", {"in": ["open", "paid"]}, written="list") + assert second == Condition( + "orders.amount", {"gte": 20, "lt": 50}, extensions={"x-why": "range"} + ) + assert isinstance(group, Where) and group.any + assert [c.column for c in where.conditions()] == [ + "orders.status", + "orders.amount", + "orders.note", + "orders.status", + "customers.vip", + ] + + +# --------------------------------------------------------------------------- +# The YAML profile and the version +# --------------------------------------------------------------------------- +@pytest.mark.parametrize( + "text, message", + [ + ("a: &x 1\nb: *x\n", "anchors"), + ("model2data-metrics: 0.1.0\nmodel: shop\nmodel: shop\n", "duplicate key"), + ("{not json", "not valid JSON"), + ("model: [\n", "not valid YAML"), + ("", "empty"), + ], +) +def test_the_yaml_profile_applies(text, message): + with pytest.raises(MetricsError) as raised: + load(text) + assert message in str(raised.value) + assert str(raised.value).startswith("The metrics file has 1 issue") + + +def test_a_list_at_the_top_is_refused(): + assert validate("- a\n") == [validate("- a\n")[0]] + assert "mapping at the top level" in validate("- a\n")[0].message + + +def test_a_minor_version_this_reader_does_not_implement_is_refused(): + issues = validate("model2data-metrics: 0.2.0\nmodel: shop\n") + assert [(i.path, i.is_error) for i in issues] == [("model2data-metrics", True)] + assert "metrics spec 0.2.0" in issues[0].message + + +def test_a_malformed_version_is_the_schemas_to_report(): + issues = validate("model2data-metrics: '1'\nmodel: shop\n") + assert [i.path for i in issues] == ["model2data-metrics"] + issues = validate("model2data-metrics: 1.0\nmodel: shop\n") + assert [i.path for i in issues] == ["model2data-metrics"] + + +def test_a_later_patch_version_reads(): + assert validate("model2data-metrics: 0.1.7\nmodel: shop\n", SHOP) == [] + + +def test_issues_carry_their_line(): + issues = validate(HEAD + "metrics:\n m:\n count: nowhere\n", SHOP) + assert [(i.path, i.line) for i in issues] == [("metrics.m.count", 5)] + + +# --------------------------------------------------------------------------- +# The schema +# --------------------------------------------------------------------------- +def test_required_keys(): + issues = validate("model2data-metrics: 0.1.0\n") + assert [(i.path, i.message) for i in issues] == [("", "`model` is required")] + + +def test_an_unknown_key_is_an_error_and_an_extension_is_not(): + assert _one_error("metrics:\n m: {count: orders, colour: red, x-colour: red}\n") == ( + "metrics.m.colour", + 'unknown key "colour": the keys allowed here are label, description, ai_context, ' + "format, measure, agg, count, ratio, expression, time, where; an extension key " + "starts with x-", + ) + + +@pytest.mark.parametrize( + "body, path, words", + [ + ("metrics:\n Revenue: {count: orders}\n", "metrics", "must be a metric name"), + ("metrics:\n m: {measure: amount, agg: sum}\n", "metrics.m.measure", "must name a column"), + ("metrics:\n m: {count: a.b.c}\n", "metrics.m.count", "must name a table"), + ("metrics:\n m: {count: orders, format: euro}\n", "metrics.m.format", "must be one of"), + ( + "metrics:\n m: {measure: orders.amount, agg: total}\n", + "metrics.m.agg", + "must be one of", + ), + ( + "metrics:\n m: {ratio: {numerator: a}}\n", + "metrics.m.ratio", + "`denominator` is required", + ), + ("metrics:\n m: {count: orders, where: {}}\n", "metrics.m.where", "must not be empty"), + ( + "metrics:\n m: {count: orders, where: {status: paid}}\n", + "metrics.m.where.status", + "unknown key", + ), + ( + "metrics:\n m: {count: orders, where: {orders.status: {like: p}}}\n", + "metrics.m.where.orders.status.like", + "unknown key", + ), + ( + "metrics:\n m: {count: orders, where: {orders.status: null}}\n", + "metrics.m.where.orders.status", + "must be", + ), + ( + "metrics:\n m: {count: orders, where: {orders.amount: {between: [1]}}}\n", + "metrics.m.where.orders.amount.between", + "at least 2", + ), + ( + "metrics:\n m: {count: orders, where: {orders.status: []}}\n", + "metrics.m.where.orders.status", + "", + ), + ( + "metrics:\n m: {count: orders, where: {any: []}}\n", + "metrics.m.where.any", + "must not be empty", + ), + ("dimensions:\n customers.vip: maybe\n", "dimensions.customers.vip", "must be"), + ("dimensions:\n vip: true\n", "dimensions", "must name a column"), + ("infer: yes\n", "infer", "true or false"), + ], +) +def test_schema_errors_say_what_is_wrong_where(body, path, words): + errors = _errors(body) + assert any(p == path and words in m for p, m in errors), errors + + +# --------------------------------------------------------------------------- +# Checks beyond the schema +# --------------------------------------------------------------------------- +def test_a_metric_needs_a_kind(): + assert _one_error("metrics:\n m: {label: M}\n") == ( + "metrics.m", + "a metric needs one of `measure` (a column), `count` (a table's rows), `ratio` or " + "`expression`", + ) + + +def test_a_metric_has_one_kind(): + assert _one_error("metrics:\n m: {count: orders, measure: orders.amount}\n") == ( + "metrics.m", + "a metric is of one kind, but `measure` and `count` are both given: keep one", + ) + + +def test_agg_only_with_measure(): + assert _one_error("metrics:\n m: {count: orders, agg: sum}\n") == ( + "metrics.m.agg", + "`agg` says how a `measure` aggregates; this metric has none", + ) + + +@pytest.mark.parametrize( + "key, value", [("time", "orders.placed_at"), ("where", "{orders.status: paid}")] +) +def test_time_and_where_are_not_for_a_ratio_or_derived_metric(key, value): + path, message = _one_error( + f"metrics:\n m: {{ratio: {{numerator: orders_amount, denominator: orders_count}}, {key}: {value}}}\n" + ) + assert path == f"metrics.m.{key}" and "a ratio is" in message + path, message = _one_error( + f"metrics:\n m: {{expression: orders_amount * 2, {key}: {value}}}\n" + ) + assert path == f"metrics.m.{key}" and "a derived metric is" in message + + +@pytest.mark.parametrize( + "expression, message", + [ + ("revenue +", "the expression ends where a metric name or number should be"), + ("revenue $ 2", "unexpected '$' at character 9"), + ("(revenue", "the parenthesis at character 1 is never closed"), + ("revenue orders", "unexpected 'orders' at character 9"), + ("2 * 3", "the expression uses no metric"), + (" ", "the expression is empty"), + ], +) +def test_an_expression_that_does_not_parse_is_an_error(expression, message): + path, found = _one_error(f'metrics:\n m: {{expression: "{expression}"}}\n', model=None) + assert path == "metrics.m.expression" + assert message in found + + +def test_a_cycle_between_metrics_is_an_error_naming_it(): + errors = _errors( + """ + metrics: + a: {expression: b + 1} + b: {ratio: {numerator: c, denominator: orders_count}} + c: {expression: a * 2} + d: {expression: d} + """, + model=None, + ) + assert errors == [ + ("metrics.a", "metrics depend on each other in a cycle: a -> b -> c -> a"), + ("metrics.d", "metrics depend on each other in a cycle: d -> d"), + ] + + +def test_without_a_model_unknown_metrics_are_reported_only_when_nothing_is_inferred(): + body = "metrics:\n r: {ratio: {numerator: nope, denominator: also_nope}}\n" + assert _errors(body, model=None) == [] + errors = _errors("infer: false\n" + body, model=None) + assert [path for path, _ in errors] == [ + "metrics.r.ratio.numerator", + "metrics.r.ratio.denominator", + ] + + +# --------------------------------------------------------------------------- +# Checks against the model +# --------------------------------------------------------------------------- +def test_the_model_name_must_match(): + issues = validate("model2data-metrics: 0.1.0\nmodel: other\n", SHOP) + assert [(i.path, i.message) for i in issues] == [ + ("model", "the metrics are for model 'other', and the model is 'shop'") + ] + + +def test_a_model_without_a_name_goes_by_the_name_it_is_given(): + nameless = load_model( + (FIXTURES / "shop.model2data.yml").read_text().replace("name: shop\n", "") + ) + assert validate("model2data-metrics: 0.1.0\nmodel: shop\n", nameless, model_name="shop") == [] + assert validate("model2data-metrics: 0.1.0\nmodel: shop\n", nameless, model_name="x") + assert validate("model2data-metrics: 0.1.0\nmodel: shop\n", nameless) == [] + + +@pytest.mark.parametrize( + "body, path, message", + [ + ( + "metrics:\n m: {measure: order.amount, agg: sum}\n", + "metrics.m.measure", + "the model has no table 'order' (did you mean 'orders'?)", + ), + ( + "metrics:\n m: {measure: orders.amout, agg: sum}\n", + "metrics.m.measure", + "table 'orders' has no column 'amout' (did you mean 'amount'?)", + ), + ( + "metrics:\n m: {count: order}\n", + "metrics.m.count", + "the model has no table 'order' (did you mean 'orders'?)", + ), + ( + "metrics:\n m: {count: orders, time: orders.when}\n", + "metrics.m.time", + "table 'orders' has no column 'when'", + ), + ( + "metrics:\n m: {count: orders, where: {orders.state: paid}}\n", + "metrics.m.where.orders.state", + "table 'orders' has no column 'state' (did you mean 'status'?)", + ), + ( + "dimensions:\n customers.nope: true\n", + "dimensions.customers.nope", + "table 'customers' has no column 'nope'", + ), + ( + "dimensions:\n people.vip: true\n", + "dimensions.people.vip", + "the model has no table 'people'", + ), + ], +) +def test_unknown_tables_and_columns(body, path, message): + assert _one_error(body) == (path, message) + + +def test_a_column_without_a_measure_needs_an_agg(): + assert _one_error("metrics:\n m: {measure: orders.status}\n") == ( + "metrics.m.measure", + "orders.status has no `measure` in the model, so say how it aggregates with `agg` " + "(sum, average, min, max, median, count, count_distinct)", + ) + + +@pytest.mark.parametrize("agg", ["sum", "average", "min", "max", "median"]) +def test_a_numeric_aggregation_needs_a_numeric_column(agg): + assert _one_error(f"metrics:\n m: {{measure: orders.status, agg: {agg}}}\n") == ( + "metrics.m.agg", + f"{agg} needs a numeric column, and orders.status is status: count and count_distinct " + "apply to any column", + ) + + +@pytest.mark.parametrize("agg", ["count", "count_distinct"]) +def test_counting_applies_to_any_column(agg): + assert _errors(f"metrics:\n m: {{measure: orders.status, agg: {agg}}}\n") == [] + + +def test_an_enum_named_like_a_number_is_not_numeric(): + assert _one_error("metrics:\n m: {measure: customers.tier, agg: sum}\n")[0] == "metrics.m.agg" + + +def test_unknown_metrics_in_a_ratio_or_expression(): + errors = _errors( + """ + metrics: + r: {ratio: {numerator: revenu, denominator: orders_count}} + d: {expression: r + orders_amont} + """ + ) + assert errors == [ + ("metrics.r.ratio.numerator", "there is no metric 'revenu'"), + ( + "metrics.d.expression", + "there is no metric 'orders_amont' (did you mean 'orders_amount'?)", + ), + ] + + +def test_inferred_metrics_are_known_to_ratios_unless_inference_is_off(): + body = "metrics:\n r: {ratio: {numerator: orders_amount, denominator: orders_count}}\n" + assert _errors(body) == [] + assert [p for p, _ in _errors("infer: false\n" + body)] == [ + "metrics.r.ratio.numerator", + "metrics.r.ratio.denominator", + ] + + +def test_time_must_be_a_date_or_timestamp(): + assert _one_error("metrics:\n m: {count: orders, time: orders.note}\n") == ( + "metrics.m.time", + "orders.note is text, not a date or timestamp column", + ) + + +def test_time_on_a_table_reached_through_a_foreign_key(): + assert _errors("metrics:\n m: {count: lines, time: customers.joined_on}\n") == [] + + +def test_a_column_no_path_reaches_is_an_error(): + assert _one_error("metrics:\n m: {count: customers, where: {orders.status: paid}}\n") == ( + "metrics.m.where.orders.status", + "filter column orders.status cannot be reached from customers: a column has to be on " + "the metric's table, or on one its foreign keys lead to (many-to-one)", + ) + assert _one_error("metrics:\n m: {count: customers, time: orders.placed_at}\n")[0] == ( + "metrics.m.time" + ) + + +def test_two_paths_to_a_table_is_an_error_naming_both(): + assert _one_error("metrics:\n m: {count: transfers, where: {customers.vip: true}}\n") == ( + "metrics.m.where.customers.vip", + "filter column customers.vip is reached from transfers along two paths, so which " + "customers it means is unclear: transfers.sender_id -> customers.id; or " + "transfers.receiver_id -> customers.id", + ) + + +def test_a_composite_foreign_key_is_a_path(): + assert _errors("metrics:\n m: {count: visits, where: {stores.region: north}}\n") == [] + + +@pytest.mark.parametrize( + "column, value, message", + [ + ( + "orders.status", + "payed", + "\"payed\" is not a member of status (open, paid, cancelled) (did you mean 'paid'?)", + ), + ("orders.status", 3, "3 is not a member of status (open, paid, cancelled)"), + ("customers.vip", "yes", 'must be true or false, not "yes"'), + ("orders.amount", "'12'", 'must be a number, as numeric is numeric, not the string "12"'), + ("lines.qty", 1.5, "must be a whole number, as int is an integer type (got 1.5)"), + ( + "customers.joined_on", + "'2026-01-01 08:00'", + 'must be an ISO 8601 date as a string, YYYY-MM-DD (got "2026-01-01 08:00")', + ), + ("customers.joined_on", "'2026-13-01'", "must be an ISO 8601 date"), + ( + "orders.placed_at", + "'yesterday'", + "must be an ISO 8601 timestamp as a string, YYYY-MM-DD or YYYY-MM-DD HH:MM:SS", + ), + ("orders.placed_at", "'2026-01-01T08:00:00+01:00'", "must be an ISO 8601 timestamp"), + ("orders.note", 12, "must be text, as text is a text column: quote it (got 12)"), + ], +) +def test_a_value_must_fit_its_column(column, value, message): + table = column.split(".")[0] + reach = "lines" if table in ("lines", "orders", "customers") else table + path, found = _one_error(f"metrics:\n m: {{count: {reach}, where: {{{column}: {value}}}}}\n") + assert path == f"metrics.m.where.{column}" + assert found.startswith(message) + + +def test_values_that_fit(): + assert ( + _errors( + """ + metrics: + m: + count: lines + where: + customers.tier: [1, "2"] + customers.vip: false + lines.qty: {in: [1, 2.0]} + customers.joined_on: {between: ["2025-01-01", "2025-12-31"]} + orders.placed_at: {gte: "2025-01-01", lt: "2026-01-01 00:00:00"} + orders.note: ["12", "on hold"] + """ + ) + == [] + ) + + +def test_a_bad_value_in_a_list_has_its_index(): + assert _one_error("metrics:\n m: {count: orders, where: {orders.status: [open, gone]}}\n")[ + 0 + ] == ("metrics.m.where.orders.status.1") + assert _one_error( + "metrics:\n m: {count: orders, where: {orders.amount: {between: [1, x]}}}\n" + )[0] == ("metrics.m.where.orders.amount.between.1") + + +@pytest.mark.parametrize( + "column, kind", + [("orders.status", "an enum"), ("customers.vip", "boolean"), ("orders.note", "text")], +) +@pytest.mark.parametrize("operator", ["gt", "gte", "lt", "lte", "between"]) +def test_order_operators_only_on_numbers_and_times(column, kind, operator): + operand = "[a, b]" if operator == "between" else "x" + path, message = _one_error( + f"metrics:\n m: {{count: lines, where: {{{column}: {{{operator}: {operand}}}}}}}\n" + ) + assert path == f"metrics.m.where.{column}.{operator}" + assert message == ( + f"`{operator}` compares by order, and {column} is {kind}: list the values it may hold " + "with `in` instead" + ) + + +def test_nested_filters_report_their_path(): + errors = _errors( + """ + metrics: + m: + count: lines + where: + any: + - orders.status: paid + - all: + - orders.status: gone + - customers.vip: maybe + """ + ) + assert [path for path, _ in errors] == [ + "metrics.m.where.any.1.all.0.orders.status", + "metrics.m.where.any.1.all.1.customers.vip", + ] + + +# --------------------------------------------------------------------------- +# Warnings +# --------------------------------------------------------------------------- +def test_a_metric_with_no_time_warns(): + assert _warnings("metrics:\n m: {measure: logs.level, agg: max}\n") == [ + ( + "metrics.m", + "no date or timestamp column on logs, or on a table it reaches, dates this metric: " + "it has a total but no value per month. Set `time` to date it", + ) + ] + + +def test_a_metric_dated_through_its_parents_does_not_warn(): + assert _warnings("metrics:\n m: {count: lines}\n") == [] + + +@pytest.mark.parametrize( + "condition, operator", + [ + ("{gt: 100}", "gt"), + ("{gte: 100.5}", "gte"), + ("{lt: 10}", "lt"), + ("{lte: 9}", "lte"), + ("{eq: 200}", "eq"), + ("{in: [1, 2]}", "in"), + ("{between: [101, 200]}", "between"), + ("{between: [1, 9]}", "between"), + ], +) +def test_a_comparison_outside_the_generated_range_never_matches(condition, operator): + assert _warnings( + f"metrics:\n m: {{count: orders, where: {{orders.amount: {condition}}}}}\n" + ) == [ + ( + f"metrics.m.where.orders.amount.{operator}", + "can never match: orders.amount is generated between 10 and 100", + ) + ] + + +@pytest.mark.parametrize( + "condition", ["{gte: 100}", "{lte: 10}", "{in: [5, 50]}", "{ne: 500}", "{between: [5, 10]}"] +) +def test_a_comparison_that_can_match_does_not_warn(condition): + assert ( + _warnings(f"metrics:\n m: {{count: orders, where: {{orders.amount: {condition}}}}}\n") + == [] + ) + + +def test_a_one_sided_range_warns_with_its_bound(tmp_path): + text = ( + (FIXTURES / "shop.model2data.yml") + .read_text() + .replace("generate: {min: 10, max: 100}", "generate: {min: 10}") + ) + model = load_model(text) + assert _warnings( + "metrics:\n m: {count: orders, where: {orders.amount: {lt: 5}}}\n", model + ) == [ + ( + "metrics.m.where.orders.amount.lt", + "can never match: orders.amount is generated at least 10", + ) + ] + text = text.replace("generate: {min: 10}", "generate: {max: 100}") + model = load_model(text) + assert _warnings("metrics:\n m: {count: orders, where: {orders.amount: 500}}\n", model) == [ + ("metrics.m.where.orders.amount", "can never match: orders.amount is generated at most 100") + ] + assert _warnings("metrics:\n m: {count: lines, where: {lines.qty: {gt: 1000}}}\n") == [] + + +def test_is_null_on_a_column_that_is_never_null_never_matches(): + assert _warnings( + "metrics:\n m: {count: orders, where: {orders.amount: {is_null: true}, orders.id: {is_null: false}}}\n" + ) == [ + ( + "metrics.m.where.orders.amount.is_null", + "can never match: orders.amount is never null in the generated data", + ) + ] + + +def test_a_between_upside_down_never_matches(): + assert _warnings( + "metrics:\n m: {count: orders, where: {orders.placed_at: {between: ['2026-02-01', '2026-01-01']}}}\n" + ) == [ + ( + "metrics.m.where.orders.placed_at.between", + 'can never match: "2026-02-01" is above "2026-01-01"', + ) + ] + + +def test_load_raises_on_errors_and_keeps_warnings(): + with pytest.raises(MetricsError) as raised: + load(HEAD + "metrics:\n m: {count: logs, where: {logs.id: x}}\n", SHOP) + assert [i.path for i in raised.value.issues] == ["metrics.m.where.logs.id"] + assert [i.path for i in raised.value.warnings] == ["metrics.m"] + metrics = load(HEAD + "metrics:\n m: {count: logs}\n", SHOP) + assert [i.path for i in metrics.warnings] == ["metrics.m"] + + +# --------------------------------------------------------------------------- +# Files +# --------------------------------------------------------------------------- +def test_metrics_files_are_told_by_their_suffix(tmp_path): + assert is_metrics_file("coffee.metrics.yml") and is_metrics_file(Path("a/B.METRICS.JSON")) + assert not is_metrics_file("coffee.model2data.yml") + assert metrics_stem(Path("coffee.metrics.yaml")) == "coffee" + assert metrics_stem(Path("odd.yml")) == "odd" + + +@pytest.mark.parametrize( + "model_name", + ["shop.model2data.yml", "shop.model2data.yaml", "shop.model2data.json", "shop.dbml"], +) +def test_the_sibling_model_is_found_by_stem(tmp_path, model_name): + (tmp_path / model_name).write_text("x") + assert sibling_model(tmp_path / "shop.metrics.yml") == tmp_path / model_name + assert sibling_model(tmp_path / "other.metrics.yml") is None + + +def test_a_file_that_is_not_utf8_is_refused(tmp_path): + path = tmp_path / "shop.metrics.yml" + path.write_bytes(b"model: \xff\n") + with pytest.raises(MetricsError): + load(path) diff --git a/tests/test_metrics_semantic.py b/tests/test_metrics_semantic.py new file mode 100644 index 0000000..48522a9 --- /dev/null +++ b/tests/test_metrics_semantic.py @@ -0,0 +1,441 @@ +"""The semantic model: paths, inferred metrics, dimensions, entities and resolved metrics.""" + +from pathlib import Path + +import pytest + +from model2data.metrics import ( + Graph, + Join, + Metric, + Metrics, + MetricsError, + inferred_metrics, + load, + resolve, +) +from model2data.metrics.graph import describe +from model2data.metrics.infer import metric_name +from model2data.model import load as load_model + +FIXTURES = Path(__file__).resolve().parent / "fixtures" / "metrics" +SHOP = load_model(FIXTURES / "shop.model2data.yml") + + +def _metrics(body: str) -> Metrics: + return load("model2data-metrics: 0.1.0\nmodel: shop\n" + body, SHOP) + + +# --------------------------------------------------------------------------- +# Paths +# --------------------------------------------------------------------------- +def test_paths_follow_foreign_keys_from_child_to_parent(): + graph = Graph(SHOP) + to_orders = Join("lines", ("order_id",), "orders", ("id",)) + to_customers = Join("orders", ("customer_id",), "customers", ("id",)) + assert graph.paths("lines", "customers") == [(to_orders, to_customers)] + assert graph.paths("lines", "lines") == [()] + assert graph.paths("customers", "orders") == [] + assert graph.paths("logs", "orders") == [] + assert describe((to_orders, to_customers)) == ( + "lines.order_id -> orders.id, then orders.customer_id -> customers.id" + ) + assert describe(()) == "the table itself" + + +def test_two_paths_are_found_and_the_search_stops_there(): + graph = Graph(SHOP) + assert len(graph.paths("transfers", "customers")) == 2 + assert len(graph.paths("transfers", "customers", limit=1)) == 1 + + +def test_a_composite_foreign_key_is_one_step(): + assert Graph(SHOP).paths("visits", "stores") == [ + ( + Join("visits", ("store_id", "day"), "slots", ("store_id", "day")), + Join("slots", ("store_id",), "stores", ("id",)), + ) + ] + + +def test_a_cycle_of_references_is_walked_once(): + model = load_model( + """ +model2data: 0.4.0 +tables: + a: + columns: + id: {type: int, pk: true} + b_id: {type: int, references: b.id} + b: + columns: + id: {type: int, pk: true} + a_id: {type: int, references: a.id} + c_id: {type: int, references: c.id} + c: + columns: + id: {type: int, pk: true} + at: date +""" + ) + graph = Graph(model) + assert len(graph.paths("a", "c")) == 1 + assert graph.first_time_column("a") == ("c.at", graph.paths("a", "c")[0]) + + +def test_the_default_time_column(): + graph = Graph(SHOP) + assert graph.first_time_column("orders") == ("orders.placed_at", ()) + # lines has no date: its order's, one step away, before the customer's, two steps away + found = graph.first_time_column("lines") + assert found is not None and found[0] == "orders.placed_at" + # stores, and anything it reaches, has none + assert graph.first_time_column("stores") is None + # transfers reaches customers along two paths, so customers cannot date it + assert graph.first_time_column("transfers") is None + assert graph.temporal_columns("customers") == ["joined_on"] + + +def test_an_enum_named_like_a_date_is_not_a_time(): + model = load_model( + """ +model2data: 0.4.0 +enums: + update_kind: [a, b] +tables: + t: + columns: + id: {type: int, pk: true} + kind: update_kind +""" + ) + assert Graph(model).temporal_columns("t") == [] + + +# --------------------------------------------------------------------------- +# Inferred metrics +# --------------------------------------------------------------------------- +def test_inferred_metrics_are_named_as_the_studio_names_them(): + inferred = inferred_metrics(SHOP) + assert list(inferred) == [ + "orders_amount", + "orders_count", + "lines_qty", + "lines_price_average", + "lines_count", + ] + amount = inferred["orders_amount"] + assert (amount.kind, amount.measure, amount.agg, amount.inferred) == ( + "simple", + "orders.amount", + "sum", + True, + ) + assert amount.label == "Orders amount" + assert amount.description == "Sum of orders.amount." + average = inferred["lines_price_average"] + assert average.label == "Lines price (average)" + assert average.description == "Average of lines.price." + assert inferred["orders_count"] == Metric( + kind="count", + count="orders", + label="Orders count", + description="Rows of orders.", + inferred=True, + ) + + +def test_a_row_count_for_facts_and_tables_with_a_measure_only(): + model = load_model( + """ +model2data: 0.4.0 +tables: + events: + role: fact + columns: + id: {type: int, pk: true} + prices: + role: dimension + columns: + id: {type: int, pk: true} + amount: {type: numeric, measure: max} + plain: + columns: + id: {type: int, pk: true} + off: {type: int, measure: false} +""" + ) + assert list(inferred_metrics(model)) == ["events_count", "prices_amount_max"] + + +@pytest.mark.parametrize( + "text, name", + [ + ("Order Items__Total", "order_items_total"), + ("raw.orders_amount", "raw_orders_amount"), + ("2024_sales", "c_2024_sales"), + ("x", "x_x"), + ("___", "unnamed"), + ("Coût €", "co_t"), + ], +) +def test_metric_names(text, name): + assert metric_name(text) == name + + +def test_a_name_taken_twice_gets_a_number(): + model = load_model( + """ +model2data: 0.4.0 +tables: + a b: + columns: + c: {type: int, measure: true} + a_b: + columns: + c: {type: int, measure: true} +""" + ) + assert list(inferred_metrics(model)) == ["a_b_c", "a_b_count", "a_b_c_2", "a_b_count_2"] + + +def test_an_explicit_metric_replaces_the_inferred_one_and_comes_first(): + semantic = resolve( + SHOP, + _metrics( + "metrics:\n lines_qty: {measure: lines.qty, agg: max}\n extra: {count: stores}\n" + ), + ) + assert list(semantic.metrics) == [ + "lines_qty", + "extra", + "orders_amount", + "orders_count", + "lines_price_average", + "lines_count", + ] + assert semantic.metrics["lines_qty"].agg == "max" + assert semantic.metrics["lines_qty"].inferred is False + + +def test_infer_false_keeps_only_the_files_metrics(): + semantic = resolve(SHOP, _metrics("infer: false\nmetrics:\n n: {count: stores}\n")) + assert list(semantic.metrics) == ["n"] + + +def test_no_metrics_file_means_the_inferred_metrics(): + semantic = resolve(SHOP) + assert list(semantic.metrics) == list(inferred_metrics(SHOP)) + assert semantic.name == "shop" + nameless = load_model("model2data: 0.4.0\ntables:\n t:\n columns:\n id: int\n") + assert resolve(nameless, model_name="stem").name == "stem" + assert resolve(nameless).name == "model" + + +def test_resolve_refuses_metrics_that_do_not_fit_the_model(): + metrics = load("model2data-metrics: 0.1.0\nmodel: shop\nmetrics:\n m: {count: nowhere}\n") + with pytest.raises(MetricsError) as raised: + resolve(SHOP, metrics) + assert [i.path for i in raised.value.issues] == ["metrics.m.count"] + + +# --------------------------------------------------------------------------- +# Resolved metrics +# --------------------------------------------------------------------------- +def test_a_simple_metric_resolves_its_table_aggregation_time_and_joins(): + semantic = resolve( + SHOP, + _metrics( + """ +metrics: + m: + measure: lines.price + where: + customers.vip: true + any: + - orders.status: paid + - lines.qty: {gt: 2} +""" + ), + ) + metric = semantic.metrics["m"] + assert (metric.table, metric.column, metric.column_kind, metric.agg) == ( + "lines", + "price", + "decimal", + "average", + ) + assert metric.time == "orders.placed_at" and metric.time_kind == "timestamp" + assert list(metric.joins) == ["orders", "customers"] + assert len(metric.joins["customers"]) == 2 + assert metric.label == "M" + + +def test_an_explicit_time_and_no_time(): + semantic = resolve( + SHOP, + _metrics( + "metrics:\n a: {count: lines, time: customers.joined_on}\n" + " b: {measure: logs.level, agg: max}\n" + ), + ) + assert semantic.metrics["a"].time == "customers.joined_on" + assert semantic.metrics["a"].time_kind == "date" + assert list(semantic.metrics["a"].joins) == ["customers"] + assert semantic.metrics["b"].time is None and semantic.metrics["b"].joins == {} + + +def test_ratio_and_derived_metrics_know_their_inputs(): + semantic = resolve( + SHOP, + _metrics( + """ +metrics: + r: {ratio: {numerator: orders_amount, denominator: orders_amount}} + d: {expression: r * 100 - lines_qty / r} +""" + ), + ) + assert semantic.metrics["r"].inputs == ("orders_amount",) + assert semantic.metrics["d"].inputs == ("r", "lines_qty") + assert semantic.simple_inputs("d") == ["orders_amount", "lines_qty"] + assert semantic.simple_inputs("orders_amount") == ["orders_amount"] + + +# --------------------------------------------------------------------------- +# Entities, relationships, dimensions, measures +# --------------------------------------------------------------------------- +def test_entities_carry_keys_grain_and_dbt_names(): + model = load_model( + """ +model2data: 0.4.0 +tables: + Order Lines: + role: fact + grain: [order_id, n] + description: One per line + keys: + - {unique: [order_id, n]} + columns: + id: {type: int, pk: true} + order_id: int + n: int + code: {type: text, unique: true} +""" + ) + entity = resolve(model).entities["Order Lines"] + assert entity.name == "order_lines" + assert entity.primary_key == ("id",) + assert entity.unique_keys == (("code",), ("order_id", "n")) + assert entity.grain == ("order_id", "n") + assert (entity.role, entity.description) == ("fact", "One per line") + + +def test_relationships_run_from_the_many_side(): + model = load_model( + """ +model2data: 0.4.0 +tables: + users: + columns: + id: {type: int, pk: true} + profiles: + columns: + user_id: {type: int, references: {to: users.id, one_to_one: true}} + pairs: + keys: + - {pk: [a, b]} + columns: + a: int + b: int + links: + foreign_keys: + - {columns: [x, y], references: pairs, to_columns: [a, b], one_to_one: true} + columns: + x: int + y: int +""" + ) + relationships = resolve(model).relationships + assert [(r.name, r.from_table, r.to_table, r.one_to_one) for r in relationships] == [ + ("profiles_user_id_to_users", "profiles", "users", True), + ("links_x_y_to_pairs", "links", "pairs", True), + ] + assert relationships[1].from_columns == ("x", "y") + assert relationships[1].to_columns == ("a", "b") + + +def test_default_dimensions_and_overrides(): + semantic = resolve( + SHOP, + _metrics( + """ +dimensions: + customers.vip: false + customers.country: {label: Country, description: Where they live} + orders.note: true + orders.status: {label: Order status} +""" + ), + ) + found = {d.path: d for d in semantic.dimensions} + assert list(found) == [ + "customers.country", + "customers.tier", + "customers.joined_on", + "orders.status", + "orders.placed_at", + "orders.note", + "slots.day", + "visits.day", + ] + assert found["customers.country"].label == "Country" + assert found["customers.country"].description == "Where they live" + assert found["customers.country"].kind == "categorical" + assert found["customers.tier"].members == ("1", "2") + assert found["orders.status"].label == "Order status" + assert found["orders.placed_at"].kind == "time" + assert found["customers.joined_on"].label == "Joined on" + + +def test_keys_foreign_keys_and_measures_are_not_categorical_dimensions(): + model = load_model( + """ +model2data: 0.4.0 +enums: + e: [a, b] +tables: + p: + keys: + - {pk: [k, k2]} + columns: + k: e + k2: int + u: {type: e, unique: true} + r: {type: e, references: q.id} + m: {type: boolean, measure: count} + ok: e + q: + keys: + - {pk: [id, id2]} + columns: + id: e + id2: int + w: + foreign_keys: + - {columns: [f, g], references: q, to_columns: [id, id2]} + columns: + f: e + g: int + flag: bool +""" + ) + assert [d.path for d in resolve(model).dimensions] == ["p.ok", "w.flag"] + + +def test_measures_list_every_measure_column(): + assert [(m.table, m.column, m.agg) for m in resolve(SHOP).measures] == [ + ("orders", "amount", "sum"), + ("lines", "qty", "sum"), + ("lines", "price", "average"), + ] diff --git a/tests/test_metrics_spec.py b/tests/test_metrics_spec.py new file mode 100644 index 0000000..f49bc70 --- /dev/null +++ b/tests/test_metrics_spec.py @@ -0,0 +1,165 @@ +"""The metrics spec, its schema and the code that reads it say the same thing. + +`model2data/spec/metrics/metrics.schema.json` and `README.md` are normative; +the reader in `model2data.metrics` implements them. Every list the schema +enumerates (aggregations, formats, operators, kinds) is also a constant in the +code, so this module checks the two against each other rather than trusting +both to be edited together. +""" + +import dataclasses +import json +import re +from pathlib import Path + +import jsonschema +import pytest +import yaml + +from model2data.metrics import ( + AGGREGATIONS, + FORMATS, + OPERATORS, + SCHEMA_URL, + SPEC_VERSION, + Metric, + load, +) +from model2data.metrics.check import schema +from model2data.metrics.infer import metric_name +from model2data.metrics.types import KIND_KEYS, ORDER_OPERATORS +from model2data.model import load as load_model + +SPEC = Path(__file__).resolve().parent.parent / "model2data" / "spec" +SCHEMA = json.loads((SPEC / "metrics" / "metrics.schema.json").read_text(encoding="utf-8")) +README = (SPEC / "metrics" / "README.md").read_text(encoding="utf-8") + + +def test_the_schema_is_a_valid_draft_2020_12_schema(): + jsonschema.Draft202012Validator.check_schema(SCHEMA) + assert schema() == SCHEMA + + +def test_the_schema_id_is_the_url_documents_point_at(): + assert SCHEMA["$id"] == SCHEMA_URL + assert ( + SCHEMA_URL + == f"https://www.jbanalytica.com/model2data/spec/metrics/{SPEC_VERSION}/metrics.schema.json" + ) + example = (SPEC / "examples" / "coffee_webshop.metrics.yml").read_text(encoding="utf-8") + assert example.splitlines()[0] == f"# yaml-language-server: $schema={SCHEMA_URL}" + assert f"model2data-metrics: {SPEC_VERSION}" in example + assert README.startswith(f"# model2data metrics — spec {SPEC_VERSION}") + + +def test_the_example_validates_against_the_schema(): + document = yaml.safe_load((SPEC / "examples" / "coffee_webshop.metrics.yml").read_text()) + jsonschema.Draft202012Validator(SCHEMA).validate(document) + + +def test_the_readme_example_conforms_against_the_example_model(): + block = re.search(r"```yaml\n(.*?)```", README, re.S) + assert block is not None + jsonschema.Draft202012Validator(SCHEMA).validate(yaml.safe_load(block.group(1))) + model = load_model(SPEC / "examples" / "coffee_webshop.model2data.yml") + assert load(block.group(1), model).warnings == [] + + +@pytest.mark.parametrize( + "broken", + [ + {"model": "m"}, + {"model2data-metrics": "0.1", "model": "m"}, + {"model2data-metrics": "0.1.0", "model": "m", "metrics": {"Bad": {"count": "t"}}}, + {"model2data-metrics": "0.1.0", "model": "m", "metrics": {"m": {}}}, + { + "model2data-metrics": "0.1.0", + "model": "m", + "metrics": {"m": {"count": "t", "measure": "t.c"}}, + }, + { + "model2data-metrics": "0.1.0", + "model": "m", + "metrics": {"m": {"count": "t", "agg": "sum"}}, + }, + { + "model2data-metrics": "0.1.0", + "model": "m", + "metrics": {"m": {"expression": "a", "time": "t.c"}}, + }, + { + "model2data-metrics": "0.1.0", + "model": "m", + "metrics": { + "m": {"ratio": {"numerator": "a", "denominator": "b"}, "where": {"t.c": 1}} + }, + }, + { + "model2data-metrics": "0.1.0", + "model": "m", + "metrics": {"m": {"count": "t", "where": {"t.c": {"not": 1}}}}, + }, + {"model2data-metrics": "0.1.0", "model": "m", "dimensions": {"t.c": "yes"}}, + ], +) +def test_the_schema_refuses(broken): + assert list(jsonschema.Draft202012Validator(SCHEMA).iter_errors(broken)) + + +def test_the_aggregations_are_the_model_specs_seven(): + model_schema = json.loads((SPEC / "model.schema.json").read_text(encoding="utf-8")) + measure = model_schema["$defs"]["column"]["properties"]["measure"]["oneOf"][1]["enum"] + assert tuple(measure) == AGGREGATIONS + assert tuple(SCHEMA["$defs"]["metric"]["properties"]["agg"]["enum"]) == AGGREGATIONS + + +def test_formats_operators_and_kinds_match_the_code(): + metric = SCHEMA["$defs"]["metric"] + assert tuple(metric["properties"]["format"]["enum"]) == FORMATS + operators = SCHEMA["$defs"]["condition"]["oneOf"][2]["properties"] + assert tuple(operators) == OPERATORS + assert set(ORDER_OPERATORS) <= set(OPERATORS) + assert [branch["required"][0] for branch in metric["oneOf"]] == list(KIND_KEYS) + for key in ("measure", "count", "ratio", "expression", "time", "where", "agg"): + assert f"`{key}`" in README + + +def test_every_metric_key_is_a_field_of_the_typed_metric(): + keys = set(SCHEMA["$defs"]["metric"]["properties"]) + fields = {f.name for f in dataclasses.fields(Metric)} - {"kind", "inferred", "extensions"} + assert keys == fields + + +def test_inferred_names_and_expression_names_are_metric_names(): + pattern = re.compile(SCHEMA["$defs"]["metricName"]["pattern"]) + for text in ("Order Items", "2024", "x", "__", "raw.orders_total", "Ça va"): + assert pattern.match(metric_name(text)) + assert "[a-z][a-z0-9_]*" in README # the grammar's NAME is the same pattern + + +@pytest.mark.parametrize( + "path, ok", + [ + ("t.c", True), + ("s.t.c", True), + ("c", False), + ("a.b.c.d", False), + (".c", False), + ("t.", False), + ], +) +def test_the_column_path_pattern(path, ok): + assert bool(re.match(SCHEMA["$defs"]["columnPath"]["pattern"], path)) is ok + + +def test_publishing_lists_the_metrics_schema(): + publishing = (SPEC / "PUBLISHING.md").read_text(encoding="utf-8") + assert ( + "https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json" + in publishing + ) + assert "model2data/spec/metrics/metrics.schema.json` of engine 1.13.0" in publishing + + +def test_the_model_spec_links_the_metrics_spec(): + assert "metrics/README.md" in (SPEC / "README.md").read_text(encoding="utf-8") diff --git a/tests/test_metrics_values.py b/tests/test_metrics_values.py new file mode 100644 index 0000000..9bc9223 --- /dev/null +++ b/tests/test_metrics_values.py @@ -0,0 +1,286 @@ +"""Known values: every metric over a run's tables, with SQL's nulls, by hand on a tiny shop. + +The frames below are small enough to check each number on paper; the comment +beside each assertion does. +""" + +from datetime import date +from decimal import Decimal +from pathlib import Path + +import pandas as pd +import pytest + +from model2data.generate.core import generate_data_from_dbml +from model2data.metrics import known_values, load, resolve +from model2data.metrics.values import _rounded +from model2data.model import load as load_model +from model2data.model import to_engine + +FIXTURES = Path(__file__).resolve().parent / "fixtures" / "metrics" +SPEC = Path(__file__).resolve().parent.parent / "model2data" / "spec" / "examples" +SHOP = load_model(FIXTURES / "shop.model2data.yml") + + +def _frames() -> dict[str, pd.DataFrame]: + customers = pd.DataFrame( + { + "id": [1, 2, 3, 4], + "email": ["a@x", "b@x", "c@x", "d@x"], + "country": ["BE", "NL", "", "null"], # the last two read as null, as dbt reads them + "vip": [True, False, None, None], + "tier": ["1", "2", None, "1"], + "joined_on": ["2024-12-03", "2025-01-09", "2025-01-30", None], + } + ) + orders = pd.DataFrame( + { + "id": pd.array([1, 2, 3, 4], dtype="Int64"), + "customer_id": pd.array([1, 2, None, 9], dtype="Int64"), # 9 is an orphan + "store_id": pd.array([None] * 4, dtype="Int64"), + "status": ["paid", "cancelled", None, "open"], + "amount": [10.5, 20.0, 30.0, 40.25], + "placed_at": [ + "2025-01-05 10:00:00", + "2025-01-20 23:59:59", + "2025-02-01 00:00:00", + None, + ], + "note": ["", "rush", None, "null"], + } + ) + lines = pd.DataFrame( + { + "id": pd.array([1, 2, 3, 4, 5], dtype="Int64"), + "order_id": pd.array([1, 1, 2, 3, 4], dtype="Int64"), + "qty": pd.array([2, 3, 1, None, 5], dtype="Int64"), + "price": [1.1, 2.2, None, 4.0, 5.5], + } + ) + logs = pd.DataFrame({"id": [1, 2], "level": [3, 7]}) + empty = { + "stores": pd.DataFrame({"id": [], "region": []}), + "transfers": pd.DataFrame({"id": [], "sender_id": [], "receiver_id": [], "amount": []}), + "slots": pd.DataFrame({"store_id": [], "day": [], "capacity": []}), + "visits": pd.DataFrame({"id": [], "store_id": [], "day": [], "people": []}), + } + return {"customers": customers, "orders": orders, "lines": lines, "logs": logs, **empty} + + +METRICS = """ +model2data-metrics: 0.1.0 +model: shop +metrics: + not_cancelled: {count: orders, where: {orders.status: {ne: cancelled}}} + not_in_cancelled: {count: orders, where: {orders.status: {not_in: [cancelled]}}} + no_status: {count: orders, where: {orders.status: {is_null: true}}} + some_status: {count: orders, where: {orders.status: {is_null: false}}} + vip_qty: {measure: lines.qty, agg: sum, where: {customers.vip: true}} + not_vip_qty: {measure: lines.qty, agg: sum, where: {customers.vip: {ne: true}}} + be_lines: {count: lines, where: {customers.country: [BE]}} + no_country_lines: {count: lines, where: {customers.country: {is_null: true}}} + paid_or_big: {count: lines, where: {any: [{orders.status: paid}, {lines.qty: {gt: 4}}]}} + both: {count: lines, where: {any: [{all: [{orders.status: paid}, {lines.qty: {gte: 3}}]}, {lines.qty: 1}]}} + january: {count: orders, where: {orders.placed_at: {between: ["2025-01-01", "2025-01-20 23:59:59"]}}} + joined_2025: {count: customers, where: {customers.joined_on: {gte: "2025-01-01"}}} + tier_one: {count: customers, where: {customers.tier: 1}} + noted: {count: orders, where: {orders.note: {is_null: false}}} + price_avg: {measure: lines.price, agg: average} + price_median: {measure: lines.price, agg: median} + amount_min: {measure: orders.amount, agg: min} + amount_max: {measure: orders.amount, agg: max} + buyers: {measure: orders.customer_id, agg: count_distinct} + statuses: {measure: orders.status, agg: count} + nothing: {measure: orders.amount, agg: sum, where: {orders.amount: {gt: 99}}} + per_order: {ratio: {numerator: orders_amount, denominator: orders_count}} + zero_division: {expression: orders_amount / (lines_qty - 11)} + third: {expression: no_status / 3} + mixed: {expression: lines_qty + customers_n} + customers_n: {count: customers} + avg_twice: {expression: price_avg * 2} + undated: {measure: logs.level, agg: sum} + with_undated: {expression: undated + orders_count} +""" + + +@pytest.fixture(scope="module") +def values(): + semantic = resolve(SHOP, load(METRICS, SHOP)) + return known_values(semantic, _frames(), seed=3, as_of="2025-03-01").values + + +@pytest.mark.parametrize( + "name, expected", + [ + ("orders_amount", Decimal("100.75")), # 10.5 + 20 + 30 + 40.25 + ("orders_count", 4), + ("lines_qty", 11), # 2 + 3 + 1 + 5, the null skipped + ("lines_count", 5), + ("not_cancelled", 2), # paid and open: the null status fails `ne` + ("not_in_cancelled", 2), + ("no_status", 1), + ("some_status", 3), + ("vip_qty", 5), # lines 1 and 2, of order 1, of customer 1 + ("not_vip_qty", 1), # line 3 (customer 2); a null vip, or no customer, fails `ne` + ("be_lines", 2), + ("no_country_lines", 2), # line 4's order has no customer, line 5's is an orphan + ("paid_or_big", 3), # lines 1 and 2 (paid), line 5 (qty 5) + ("both", 2), # line 2 (paid, 3), line 3 (qty 1) + ("january", 2), # both bounds inclusive + ("joined_2025", 2), + ("tier_one", 2), # an integer member written as a number + ("noted", 1), # "" and "null" read as null, as dbt reads them + ("price_avg", Decimal("3.2")), # (1.1 + 2.2 + 4.0 + 5.5) / 4 + ("price_median", Decimal("3.1")), # between 2.2 and 4.0 + ("amount_min", Decimal("10.5")), + ("amount_max", Decimal("40.25")), + ("buyers", 3), # 1, 2 and the orphan 9 + ("statuses", 3), + ("nothing", None), # a sum over no rows is null + ("per_order", Decimal("25.1875")), # 100.75 / 4 + ("zero_division", None), # 11 - 11 is zero + ("third", Decimal("0.333333")), + ("mixed", 15), # 11 + 4 + ("avg_twice", Decimal("6.4")), + ("undated", 10), + ("with_undated", 14), + ], +) +def test_known_values(values, name, expected): + assert values[name].value == expected + + +def test_values_by_month(values): + # Order 4 has no time: in the total, in no month. + assert values["orders_amount"].by_month == {"2025-01": Decimal("30.5"), "2025-02": 30} + # Lines are dated by their order; February's only line has no qty, so its sum is null. + assert values["lines_qty"].by_month == {"2025-01": 6, "2025-02": None} + assert values["lines_qty"].time == "orders.placed_at" + # A filter that keeps no row in a month: a count is 0. + assert values["not_cancelled"].by_month == {"2025-01": 1, "2025-02": 0} + # A ratio per month is its inputs' per month. + assert values["per_order"].by_month == {"2025-01": Decimal("15.25"), "2025-02": 30} + # Inputs dated differently: each in its own months, a missing count 0, a missing sum null. + assert values["customers_n"].by_month == {"2024-12": 1, "2025-01": 2} + assert values["mixed"].by_month == {"2024-12": None, "2025-01": 8, "2025-02": None} + assert values["third"].by_month == {"2025-01": 0, "2025-02": Decimal("0.333333")} + + +def test_a_metric_without_time_has_no_months(values): + assert values["undated"].by_month is None and values["undated"].time is None + assert values["with_undated"].by_month is None + + +def test_the_json_is_stable_text(): + metrics = load( + "model2data-metrics: 0.1.0\nmodel: shop\ninfer: false\nmetrics:\n" + " revenue: {label: Revenue, measure: orders.amount, agg: sum}\n" + " share: {expression: revenue / 3}\n" + " levels: {measure: logs.level, agg: max}\n", + SHOP, + ) + values = known_values(resolve(SHOP, metrics), _frames(), seed=3, as_of="2025-03-01") + assert values.to_json() == ( + """{ + "model2data-metrics": "0.1.0", + "model": "shop", + "run": { + "seed": 3, + "as_of": "2025-03-01" + }, + "rounding": "counts, and sums, minimums and maximums of integer columns, are exact; every other value is rounded to 6 decimal places, half to even", + "metrics": { + "revenue": { + "kind": "simple", + "label": "Revenue", + "value": 100.75, + "time": "orders.placed_at", + "by_month": { + "2025-01": 30.5, + "2025-02": 30 + } + }, + "share": { + "kind": "derived", + "label": "Share", + "value": 33.583333, + "by_month": { + "2025-01": 10.166667, + "2025-02": 10 + } + }, + "levels": { + "kind": "simple", + "label": "Levels", + "value": 7 + } + } +} +""" + ) + values.seed = values.as_of = None + assert '"run"' not in values.to_json() + + +def test_an_empty_mapping_is_written_as_such(): + from model2data.metrics.values import _dump + + assert _dump({"a": {}}) == '{\n "a": {}\n}' + + +@pytest.mark.parametrize( + "raw, exact, written", + [ + (Decimal("1200.000000000"), False, "1200"), + (Decimal("0.0000004"), False, "0"), + (Decimal("0.0000005"), False, "0"), # half to even + (Decimal("0.0000015"), False, "0.000002"), + (2.675, False, "2.675"), + (1 / 3, False, "0.333333"), + (Decimal("-1.50"), False, "-1.5"), + (7, False, "7"), + (Decimal("12"), True, "12"), + (None, True, "None"), + ], +) +def test_rounding(raw, exact, written): + value = _rounded(raw, exact) + assert (format(value, "f") if isinstance(value, Decimal) else str(value)) == written + + +def test_a_table_a_metric_reads_must_have_rows(): + frames = _frames() + del frames["logs"] + semantic = resolve( + SHOP, load("model2data-metrics: 0.1.0\nmodel: shop\nmetrics:\n l: {count: logs}\n", SHOP) + ) + with pytest.raises(ValueError, match="no generated rows for table 'logs'"): + known_values(semantic, frames) + + +def test_the_reference_example_over_generated_data(): + """Recomputes three of the coffee example's metrics with pandas, by their definitions.""" + model = load_model(SPEC / "coffee_webshop.model2data.yml") + metrics = load(SPEC / "coffee_webshop.metrics.yml", model) + inputs = to_engine(model) + frames = generate_data_from_dbml( + tables=inputs.tables, refs=inputs.refs, base_rows=150, seed=11, as_of=date(2026, 3, 15) + ) + values = known_values(resolve(model, metrics), frames).values + orders = frames["orders"] + kept = orders[orders["status"].isin(["paid", "shipped", "delivered"])] + assert values["revenue"].value == _rounded( + Decimal(str(round(kept["total_amount"].sum(), 6))), False + ) + assert values["orders"].value == orders["id"].nunique() + assert values["order_lines"].value == len(frames["order_items"]) + lines = frames["order_items"].merge( + orders, left_on="order_id", right_on="id", suffixes=("", "_o") + ) + months = pd.to_datetime(lines["order_date"]).dt.strftime("%Y-%m").value_counts().to_dict() + assert values["order_lines"].by_month == dict(sorted(months.items())) + orders_count = values["orders"].value + assert isinstance(orders_count, int) + assert values["average_order_value"].value == _rounded( + Decimal(str(values["revenue"].value)) / orders_count, False + ) diff --git a/uv.lock b/uv.lock index af4dadd..62b65b7 100644 --- a/uv.lock +++ b/uv.lock @@ -785,7 +785,7 @@ wheels = [ [[package]] name = "model2data" -version = "1.12.0" +version = "1.13.0" source = { editable = "." } dependencies = [ { name = "dbt-core" },