diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ca2e27..00b1a00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [1.14.0] - 2026-10-07 + +### Added +- **Lightdash metrics in the generated dbt project.** `generate --lightdash` writes the model's + metrics, dimensions and joins where [Lightdash](https://docs.lightdash.com/references/metrics) + reads them: under `config.meta` of each staging model's own `models/staging/stg_.yml`, + next to its tests (dbt reads a model's properties from one file only, so the meta is merged in + rather than written beside it). It takes the `--metrics` file's metrics, or without one the + metrics the model implies. Each table is joined many-to-one to every table it reaches along + exactly one path; a dimension gets its label and a column that is not one is hidden; a simple + metric sits on its column, a row count is a `count_distinct` of the one-column primary key, and + a ratio or expression is a `type: number` metric over its inputs; `ai_context` becomes + `ai_hint`, `percent` and `currency` the formats `'0.00%'` and `'#,##0.00'`. A filter becomes + the metric's `filters` wherever Lightdash's filter grammar means exactly the same (with + `'!null'` beside a not-equal, since Lightdash's lets nulls through), and a `CASE WHEN` in its + `sql` where it does not (an `any` group, a date comparison), so the metric counts what its + known value counts. What Lightdash has no place for (an enum's members, a table's role and + grain, a currency, a metric dated by another table's column, a ratio whose inputs are on two + tables, which is left out) is listed in the run's summary. Every example's output validates + against Lightdash's published JSON Schema for dbt YAML, parses with dbt, and compiles with + `lightdash compile --no-partial-compilation`. +- **`model2data metrics export --to lightdash`** writes the same meta as one dbt properties file + for the staging models, without generating, the losses listed at its top. +- **`model2data.metrics.to_lightdash`** and **`write_lightdash`** in the Python API, beside + `to_ossie`. + +Without `--lightdash`, a run writes byte for byte what 1.13.0 wrote, with or without +`--metrics`. + ## [1.13.0] - 2026-10-06 ### Added diff --git a/LLMS.md b/LLMS.md index ad7cce3..d5fd084 100644 --- a/LLMS.md +++ b/LLMS.md @@ -296,11 +296,13 @@ model2data --file SCHEMA.dbml [OPTIONS] --metrics PATH The model's metrics file: adds metric_values.json (known values), osi/NAME.yml (Apache Ossie 0.1.1) and data-tests/metrics/ (a dbt test per metric). Changes no other file +--lightdash Add Lightdash metrics and dimensions (config.meta) to the staging + models' YAML, from --metrics or the inferred metrics model2data validate FILE... [--model MODEL] a metrics file (*.metrics.yml) is checked against --model, or the model beside it with its stem model2data metrics list -f MODEL [--metrics FILE] every metric, inferred too -model2data metrics export -f MODEL [--metrics FILE] --to ossie [-o FILE] without generating +model2data metrics export -f MODEL [--metrics FILE] --to ossie|lightdash [-o FILE] without generating ``` For anything not covered here, see [README.md](README.md) and [docs/](docs/README.md) — this file exists to make a schema diff --git a/README.md b/README.md index 8e72b27..c880923 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,9 @@ orphans, nulls, invalid values, messy text, late-arriving rows. Every run with d orders, average order value. The run writes each metric's value over the generated data to `metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`. +New in 1.14.0: `--lightdash` writes the same metrics into the staging models' YAML as +[Lightdash](https://www.lightdash.com/) reads them, filters, joins and ratios included, so the +dashboard's revenue can be checked against the number the run already knows. → [Metrics with known values](docs/metrics.md) **Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub @@ -225,7 +228,7 @@ definition goes in; nothing but synthetic data comes out. `accepted_values` fail for the right reason. - **A team defining metrics once.** Write revenue in one metrics file, check every number the semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL, - and hand the definitions on as Apache Ossie. + and hand the definitions on as Apache Ossie or straight to Lightdash. ## How it works @@ -281,7 +284,8 @@ flowchart LR model's hints, table and column `description:` fields from the model, and a profile for DuckDB (zero-config, file-based) or Postgres. 4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes - its known value, a dbt test that checks it, and the Apache Ossie file. + its known value, a dbt test that checks it, and the Apache Ossie file; with `--lightdash`, + Lightdash metrics and dimensions in the staging models' YAML. --- diff --git a/README_PYPI.md b/README_PYPI.md index 9304bf2..b03d4a9 100644 --- a/README_PYPI.md +++ b/README_PYPI.md @@ -84,6 +84,9 @@ orphans, nulls, invalid values, messy text, late-arriving rows. Every run with d orders, average order value. The run writes each metric's value over the generated data to `metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`. +New in 1.14.0: `--lightdash` writes the same metrics into the staging models' YAML as +[Lightdash](https://www.lightdash.com/) reads them, filters, joins and ratios included, so the +dashboard's revenue can be checked against the number the run already knows. → [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md) **Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub @@ -225,7 +228,7 @@ definition goes in; nothing but synthetic data comes out. `accepted_values` fail for the right reason. - **A team defining metrics once.** Write revenue in one metrics file, check every number the semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL, - and hand the definitions on as Apache Ossie. + and hand the definitions on as Apache Ossie or straight to Lightdash. ## How it works @@ -251,7 +254,8 @@ model (.model2data.yml or DBML) + metrics file (optional) model's hints, table and column `description:` fields from the model, and a profile for DuckDB (zero-config, file-based) or Postgres. 4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes - its known value, a dbt test that checks it, and the Apache Ossie file. + its known value, a dbt test that checks it, and the Apache Ossie file; with `--lightdash`, + Lightdash metrics and dimensions in the staging models' YAML. --- diff --git a/docs/README.md b/docs/README.md index e72f6a5..c8cc7e0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,7 +16,7 @@ The [README](../README.md) shows what model2data does; these pages say how each - [Defects on purpose](defects.md) — `--defects messy|training`, defects per table, the expected-failures report. - [Metrics with known values](metrics.md) — the metrics file, `metric_values.json`, a dbt test - per metric, the Apache Ossie export. + per metric, the Apache Ossie export, Lightdash metrics in the dbt YAML. - [Validate models in CI](ci.md) — the GitHub Action, pre-commit, GitLab CI. - [Using model2data from a coding agent or an LLM](agents.md) — `model2data guide`, LLMS.md. diff --git a/docs/dbt-project.md b/docs/dbt-project.md index 42862b1..61f9ff8 100644 --- a/docs/dbt-project.md +++ b/docs/dbt-project.md @@ -76,7 +76,7 @@ dbt_{project_name}/ ├── models/ │ └── staging/ │ ├── stg_table1.sql -│ ├── stg_table1.yml +│ ├── stg_table1.yml # with --lightdash, also its Lightdash meta │ ├── ut_stg_table1.yml # only with --unit-tests │ └── ... ├── data-tests/ diff --git a/docs/metrics.md b/docs/metrics.md index cfa2a7b..244e52a 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -65,13 +65,88 @@ With `--metrics`, the run also writes, and changes nothing else: as ANSI SQL expressions. What Ossie 0.1.1 has no field for (a metric's label, format and time, enum members) is carried in `custom_extensions` and listed at the top of the file. +## Lightdash + +`--lightdash` puts the metrics where [Lightdash](https://docs.lightdash.com/references/metrics) +reads them: in the dbt project's own YAML, under `config.meta` of each staging model +(`models/staging/stg_
.yml`), next to the tests already there. dbt reads a model's +properties from one file only, so the meta is merged into those files rather than written beside +them; with or without `--metrics`, and without `--lightdash` they are byte for byte what they were. + +```bash +model2data --file coffee_webshop.model2data.yml --metrics coffee_webshop.metrics.yml --lightdash +cd dbt_coffee_webshop && dbt parse +lightdash compile --no-warehouse-credentials --no-partial-compilation # every explore compiles +``` + +Lightdash queries a warehouse it can reach, Postgres for one (`--adapter postgres`), or +MotherDuck; it does not open a local DuckDB file, so `lightdash deploy` needs one of those. + +```yaml +- name: stg_orders + config: + meta: + label: Orders + primary_key: id + joins: + - join: stg_customers + sql_on: ${stg_orders.customer_id} = ${stg_customers.id} + relationship: many-to-one + metrics: + average_order_value: + type: number + sql: ${revenue} / NULLIF(${orders}, 0) + label: Average order value + format: '#,##0.00' + columns: + - name: total_amount + config: + meta: + dimension: + hidden: true + metrics: + revenue: + type: sum + label: Revenue + ai_hint: What customers paid for. ... + format: '#,##0.00' + filters: + - status: [paid, shipped, delivered] + default_time_dimension: {field: order_date, interval: MONTH} +``` + +Each table becomes a Lightdash table, joined many-to-one to every table it reaches along exactly +one path, as a metric's filter reaches them. A dimension gets its label; a column that is not one +(a key, an amount) is hidden, as `dimensions: {customers.email: false}` asks. A simple metric sits +on its column with its aggregation as its `type`; a row count is a `count_distinct` of the +one-column primary key (the same number, and still right where another table joins this one); a +ratio or an expression is a `type: number` metric over its inputs. `ai_context` becomes +Lightdash's `ai_hint`, `percent` and `currency` its `'0.00%'` and `'#,##0.00'`. + +A filter becomes the metric's `filters` where Lightdash's filter grammar says exactly the same: +values and lists, `ne` and `not_in` (with `'!null'`, since Lightdash's not-equal lets nulls +through and model2data's does not), `is_null`, and comparisons and `between` on numbers. +Anything else, an `any` group or a date comparison, goes into the metric's `sql` as a +`CASE WHEN`, so the metric still counts exactly the rows its known value counts. + +What Lightdash has no place for is listed in the summary and at the top of the exported file: an +enum's members, a table's role and grain, a currency, a metric dated by a column of another table +(Lightdash's default time dimension is a field of the metric's own table), and a ratio whose +inputs are on two tables, which is left out: a Lightdash metric belongs to one table and sees +another only through a join from it, which would drop or repeat rows. + +## Without generating + `model2data metrics list -f MODEL [--metrics FILE]` shows every metric, and `model2data metrics -export -f MODEL [--metrics FILE] --to ossie [-o FILE]` writes the Ossie file without generating. +export -f MODEL [--metrics FILE] --to ossie|lightdash [-o FILE]` writes the Ossie file, or the +Lightdash meta as one dbt properties file for the staging models, without generating. In a +project `generate` wrote, use `--lightdash` instead: a second properties file for the same models +would not parse. Without `--metrics`, both use the model's inferred metrics. A sibling metrics file is not picked up by `generate` on its own: a file appearing beside a model must not change what an unchanged command writes. From Python, `model2data.metrics` loads and checks a file (`load`, `validate`), resolves it with the model into one semantic representation (`resolve`), and computes -(`known_values`) and exports (`to_ossie`) from it. +(`known_values`) and exports (`to_ossie`, `to_lightdash`, `write_lightdash`) from it. The e-commerce example has a metrics file too, [`examples/ecommerce.metrics.yml`](../examples/ecommerce.metrics.yml) (revenue, orders, average order value, units sold); the [README](../README.md#before-and-after) diff --git a/model2data/cli.py b/model2data/cli.py index 1f2045b..96f3bca 100644 --- a/model2data/cli.py +++ b/model2data/cli.py @@ -50,13 +50,16 @@ SPEC_VERSION as METRICS_SPEC_VERSION, ) from model2data.metrics import ( + LightdashExport, Metrics, MetricsError, is_metrics_file, known_values, metrics_stem, sibling_model, + to_lightdash, to_ossie, + write_lightdash, write_metric_tests, ) from model2data.metrics import load as load_metrics @@ -471,6 +474,16 @@ def main( "without it. Not picked up from beside the model: pass it." ), ), + lightdash: bool = typer.Option( + False, + "--lightdash", + help=( + "Add Lightdash metrics and dimensions to the staging models' YAML\n" + "(models/staging/stg_*.yml, under config.meta): the --metrics file's metrics,\n" + "or without one the metrics the model implies. Without it those files are\n" + "unchanged." + ), + ), ): """ Generate synthetic data and a dbt project from a model. @@ -726,6 +739,13 @@ def pick(option: Any, from_run: Any, default: Any) -> Any: # Metrics: only added files, computed from what the seeds hold # ------------------------- metrics_count: Optional[int] = None + lightdash_export = None + if _given(lightdash): + typer.echo("🔦 Adding Lightdash metrics and dimensions to the staging models...") + lightdash_export = to_lightdash( + resolve_metrics(model, metrics_doc, model_name=_model_stem(file)) + ) + write_lightdash(dest, lightdash_export) if metrics_doc is not None and metrics_file is not None: typer.echo("📏 Computing each metric's known value and its dbt test...") semantic = resolve_metrics(model, metrics_doc, model_name=_model_stem(file)) @@ -797,6 +817,13 @@ def pick(option: Any, from_run: Any, default: Any) -> Any: f"osi/{project_name}.yml, data-tests/metrics/)" ) + if lightdash_export is not None: + typer.echo( + f" Lightdash: {len(lightdash_export.metrics)} metrics " + "(models/staging/stg_*.yml, config.meta)" + ) + _print_losses("Not expressible in Lightdash, left out", lightdash_export.lossiness) + if report is not None: _print_defects(report) @@ -1074,7 +1101,14 @@ def guide_command( ) app.add_typer(metrics_app, name="metrics") -_METRICS_EXPORTS = ("ossie",) +_METRICS_EXPORTS = ("ossie", "lightdash") + + +def _print_losses(heading: str, losses: list) -> None: + if losses: + typer.echo(f"ℹ️ {heading}:") + for loss in losses: + typer.echo(f" - {loss}") def _metrics_inputs(file: Path, metrics_file: Optional[Path]) -> Any: @@ -1138,7 +1172,14 @@ def metrics_list_command( def metrics_export_command( file: Path = _MODEL_OPTION, metrics_file: Optional[Path] = _METRICS_OPTION, - to: str = typer.Option("ossie", "--to", help="The format: 'ossie' (Apache Ossie 0.1.1, YAML)."), + to: str = typer.Option( + "ossie", + "--to", + help=( + "The format: 'ossie' (Apache Ossie 0.1.1, YAML) or 'lightdash' (dbt model " + "properties with Lightdash's config.meta, YAML)." + ), + ), output: Optional[Path] = typer.Option( # noqa: B008 None, "--output", @@ -1153,6 +1194,11 @@ def metrics_export_command( Ossie datasets read the staging models of the dbt project `generate` writes (staging.stg_
). What Ossie 0.1.1 cannot say is listed at the top of the file, and carried in custom_extensions. + + Lightdash's file describes those staging models (stg_
) as a dbt properties + file. dbt reads a model's properties from one file only, so in a project `generate` + wrote, use `generate --lightdash` instead: it merges the same meta into the staging + models' own YAML. What Lightdash cannot say is listed at the top of the file. """ to = (_given(to) or "ossie").lower() if to not in _METRICS_EXPORTS: @@ -1160,7 +1206,7 @@ def metrics_export_command( f"Choose one of: {', '.join(_METRICS_EXPORTS)}.", param_hint="--to" ) semantic = _metrics_inputs(file, _given(metrics_file)) - export = to_ossie(semantic) + export = to_ossie(semantic) if to == "ossie" else to_lightdash(semantic) text = export.to_yaml() output = _given(output) if output is None: @@ -1170,8 +1216,11 @@ def metrics_export_command( typer.echo(f"❌ {output} already exists. Use --force to overwrite it.") raise typer.Exit(1) output.write_text(text, encoding="utf-8") + if isinstance(export, LightdashExport): + typer.echo(f"✅ Wrote {output} (Lightdash, dbt YAML, {len(export.metrics)} metrics)") + _print_losses("Not expressible in Lightdash, left out", export.lossiness) + return typer.echo(f"✅ Wrote {output} (Apache Ossie 0.1.1, {len(semantic.metrics)} metrics)") - if export.lossiness: - typer.echo("ℹ️ Not expressible in Ossie 0.1.1 itself, carried in custom_extensions:") - for loss in export.lossiness: - typer.echo(f" - {loss}") + _print_losses( + "Not expressible in Ossie 0.1.1 itself, carried in custom_extensions", export.lossiness + ) diff --git a/model2data/guide/tune.md b/model2data/guide/tune.md index 4003a11..ad2dd39 100644 --- a/model2data/guide/tune.md +++ b/model2data/guide/tune.md @@ -8,7 +8,7 @@ | `model2data validate M...` | check models against spec 0.4.0, and `*.metrics.yml` files against the metrics spec and their model (`--model M`, else the model beside it with the same stem) | 0 conforms (warnings allowed), 1 any error | | `model2data convert M` | print M as `.model2data.yml`; `-o FILE` writes it, `--force` overwrites | 0 ok, 1 unreadable model or output exists | | `model2data metrics list -f M [--metrics F]` | every metric: the file's, then the inferred ones | 0, 1 on a model or metrics error | -| `model2data metrics export -f M [--metrics F] --to ossie` | the model and its metrics as Apache Ossie 0.1.1; `-o FILE` writes it, `--force` overwrites | 0, 1 on an error or an existing output, 2 on another `--to` | +| `model2data metrics export -f M [--metrics F] --to ossie\|lightdash` | the model and its metrics as Apache Ossie 0.1.1, or as Lightdash meta in one dbt properties file for the staging models; `-o FILE` writes it, `--force` overwrites | 0, 1 on an error or an existing output, 2 on another `--to` | | `model2data guide [TOPIC]` | this page; topics `setup`, `triage`, `tune` | 0 | `generate` exits 0 on success; 1 on a model error, an existing `dbt_/` without @@ -142,7 +142,9 @@ Only tables with `incremental` change after the first day. With `--metrics FILE`, and nothing else changed: `metric_values.json` (each metric's known value over the seeds, overall and by month), `data-tests/metrics/metric_.sql` (a dbt test per metric that fails unless the warehouse gets that value) and `osi/.yml` -(Apache Ossie 0.1.1). +(Apache Ossie 0.1.1). With `--lightdash` (with or without `--metrics`), the staging models' +`stg_
.yml` also carry Lightdash metrics, dimensions and joins under `config.meta`; what +Lightdash cannot say is listed in the summary. ## Metrics file diff --git a/model2data/metrics/__init__.py b/model2data/metrics/__init__.py index 446f218..e9f7b0c 100644 --- a/model2data/metrics/__init__.py +++ b/model2data/metrics/__init__.py @@ -12,6 +12,7 @@ >>> values = metrics.known_values(semantic, frames) # over a run's generated tables >>> export = metrics.to_ossie(semantic) # Apache Ossie 0.1.1 >>> export.to_yaml(), export.lossiness + >>> metrics.to_lightdash(semantic).to_yaml() # Lightdash meta, dbt YAML `resolve(model)` with no metrics file gives the metrics the model implies on its own: one per `measure` column, and a row count per fact. @@ -23,6 +24,7 @@ from model2data.metrics.expression import parse as parse_expression from model2data.metrics.graph import Graph, Join from model2data.metrics.infer import inferred_metrics +from model2data.metrics.lightdash import LightdashExport, to_lightdash, write_lightdash from model2data.metrics.ossie import OSSIE_VERSION, Loss, OssieExport, to_ossie from model2data.metrics.reader import ( MetricsError, @@ -71,6 +73,7 @@ "Join", "KnownValue", "KnownValues", + "LightdashExport", "Loss", "Measure", "Metric", @@ -94,7 +97,9 @@ "parse_expression", "resolve", "sibling_model", + "to_lightdash", "to_ossie", "validate", + "write_lightdash", "write_metric_tests", ] diff --git a/model2data/metrics/lightdash.py b/model2data/metrics/lightdash.py new file mode 100644 index 0000000..318cb50 --- /dev/null +++ b/model2data/metrics/lightdash.py @@ -0,0 +1,597 @@ +"""A semantic model as Lightdash metrics and dimensions: the `meta` of the dbt project's YAML. + +Lightdash reads its semantic layer from a dbt project's model properties, under +`config.meta` (dbt 1.10 and later) at model and column level. The generated +project has one dbt model per table, the staging model `stg_`, so each +becomes a Lightdash table with its explore. The mapping: + +| model2data | Lightdash (dbt `config.meta`) | +|---|---| +| a table | its staging model: `label`, `primary_key`, and `joins` | +| a foreign key | a `many-to-one` (or `one-to-one`) join from the many side, `sql_on` its columns | +| a table reached along exactly one path | joined in the explore, a step at a time | +| a dimension | the column's `dimension: {label}`; a column that is not one is `hidden` | +| a simple metric | a metric on its column: `type` its aggregation | +| a row count | `count_distinct` on a one-column primary key (the same number, and safe in an explore that joins the table), else a model-level `count` of `1` | +| a filter | the metric's `filters`, or, where they cannot say it, a `CASE WHEN` in its `sql` | +| `time` on the metric's table | the metric's `default_time_dimension`, by `MONTH` | +| a ratio or derived metric | a model-level `type: number` metric over its inputs, `${name}` | +| `label`, `description`, `ai_context` | `label`, `description`, `ai_hint` | +| `format: percent`, `currency` | `format: '0.00%'`, `'#,##0.00'` (spreadsheet-style) | + +Lightdash's own filter grammar (`packages/common/src/types/filterGrammar.grammar.ts`) +decides what goes in `filters`. A list of values is passed through as an `IN` +and never parsed, so `eq` and `in` are written as lists. Lightdash's not-equal +lets nulls through, where model2data's lets them fail, so `ne` and `not_in` add +`'!null'`. Comparisons on numbers are `'> 5'` and `'between 0 and 50'`. +Anything else -- an `any` or `all` group, a comparison on a date or timestamp, +a value the grammar would read as something else -- is written into the +metric's `sql` as a `CASE WHEN`, which Lightdash ANDs with the filters, so the +metric still counts exactly what model2data counts. + +What Lightdash has no place for, and what this export leaves out, is listed in +the export's lossiness report: a table's role and grain, an enum's members, a +currency, a metric dated by a column of another table, a ratio or derived +metric whose inputs are on different tables (a Lightdash metric belongs to +one table and sees another only through a join from it), a table an explore +cannot join because two paths reach it, and a column name that is not a plain +lowercase identifier. +""" + +from __future__ import annotations + +import re +import textwrap +from collections.abc import Callable +from dataclasses import dataclass, field +from decimal import Decimal +from pathlib import Path +from typing import Any, Optional, Union + +import yaml + +from model2data.metrics import expression as expr +from model2data.metrics.columns import ColumnKind, column_kind, find_column, is_temporal +from model2data.metrics.graph import Graph +from model2data.metrics.graph import Path as JoinPath +from model2data.metrics.infer import _label +from model2data.metrics.ossie import Loss +from model2data.metrics.semantic import Entity, ResolvedMetric, SemanticModel +from model2data.metrics.sql import identifier, where_sql +from model2data.metrics.types import Condition, Where + +# The interval a metric's default time dimension is explored by: Lightdash's own default. +DEFAULT_INTERVAL = "MONTH" +# model2data's `format` as a Lightdash spreadsheet-style format; `number` is Lightdash's default. +FORMATS = {"percent": "0.00%", "currency": "#,##0.00"} +# The aggregation of a simple metric as a Lightdash metric type: the same words but one. +TYPES = { + "sum": "sum", + "average": "average", + "min": "min", + "max": "max", + "median": "median", + "count": "count", + "count_distinct": "count_distinct", +} + +_PLAIN = re.compile(r"^[a-z_][a-z0-9_]*$") +# A value Lightdash's filter grammar reads back as itself after a leading `!`: no +# symbol it treats specially (`%`, `,`, `^`), no quote, backslash or control +# character, no space at either end, and no keyword (`null`, `empty`) at the start. +_UNSAFE = re.compile(r'[%,^"\\\x00-\x1f]') +_KEYWORDS = ("null", "NULL", "empty", "EMPTY") + + +def default_source(entity: Entity) -> str: + """The dbt model a table is in the generated project: its staging model.""" + return f"stg_{entity.name}" + + +@dataclass +class LightdashExport: + """The dbt model properties that carry the Lightdash meta, and what they could not say. + + `models` holds one dbt properties entry per model (`name`, `description`, + `config.meta`, `columns`), `metrics` the names of the metrics exported, in + order. + """ + + models: list[dict[str, Any]] + lossiness: list[Loss] = field(default_factory=list) + metrics: list[str] = field(default_factory=list) + + @property + def document(self) -> dict[str, Any]: + """A dbt properties file (`version: 2`) describing every model.""" + return {"version": 2, "models": self.models} + + def to_yaml(self) -> str: + """The properties file as YAML, the lossiness report as comments above it.""" + lines = [ + "# Lightdash metrics and dimensions (dbt config.meta) for the staging models,", + "# written by model2data from the model and its metrics.", + ] + if self.lossiness: + lines.append("#") + lines.append("# Not expressible in Lightdash, left out:") + for loss in self.lossiness: + lines.extend( + textwrap.wrap( + str(loss), + width=100, + initial_indent="# - ", + subsequent_indent="# ", + break_on_hyphens=False, + ) + ) + return "\n".join(lines) + "\n" + _dump(self.document) + + +def _dump(data: dict[str, Any]) -> str: + # One line per SQL expression, however long: folded SQL is hard to read and to diff. + return yaml.safe_dump( + data, default_flow_style=False, sort_keys=False, allow_unicode=True, width=1_000_000 + ) + + +class _Export: + """The state of one export: the models' meta as it is built, and the losses.""" + + def __init__(self, semantic: SemanticModel, source: Callable[[Entity], str]): + self.semantic = semantic + self.model = semantic.model + self.graph = Graph(semantic.model) + self.names = {key: source(entity) for key, entity in semantic.entities.items()} + self.losses: list[Loss] = [] + self.model_metrics: dict[str, dict[str, Any]] = {key: {} for key in self.names} + self.column_metrics: dict[tuple[str, str], dict[str, Any]] = {} + # What a metric needs joined besides its own table, by metric. + self.needs: dict[str, set[str]] = {} + # The metrics dated by a column of another table, by that column. + self.dated_elsewhere: dict[str, list[str]] = {} + + # -- references ----------------------------------------------------------- + def ref(self, table: str, home: str, column: str) -> str: + """A column as Lightdash SQL names it, from a metric on table `home`.""" + if table == home: + return "${" + column + "}" + return "${" + f"{self.names[table]}.{column}" + "}" + + def filter_key(self, table: str, home: str, column: str) -> str: + return column if table == home else f"{self.names[table]}.{column}" + + def kind(self, path: str) -> ColumnKind: + _, column = find_column(self.model, path) + assert column is not None + return column_kind(self.model, column) + + +def to_lightdash( + semantic: SemanticModel, *, source: Optional[Callable[[Entity], str]] = None +) -> LightdashExport: + """The Lightdash meta of `semantic` as dbt model properties, and what it could not express. + + `source` names the dbt model each table is; by default the staging model of + the dbt project `model2data generate` writes, `stg_`. + """ + state = _Export(semantic, source or default_source) + # Simple metrics and row counts first: a ratio or derived metric reads what they need. + for name, metric in semantic.metrics.items(): + if metric.kind in ("simple", "count"): + _simple(state, name, metric) + for name, metric in semantic.metrics.items(): + if metric.kind not in ("simple", "count"): + _number(state, name, metric) + for time, names in state.dated_elsewhere.items(): + state.losses.append( + Loss( + f"{'metric' if len(names) == 1 else 'metrics'} {', '.join(names)}", + "time", + f"dated by {time}, a column of {state.names[time.rpartition('.')[0]]}; a " + "Lightdash metric's default time dimension is a field of its own table", + ) + ) + models = [_model(state, key) for key in semantic.entities] + _general_losses(state) + exported = [name for name in semantic.metrics if name in state.needs] + return LightdashExport(models, state.losses, exported) + + +# --------------------------------------------------------------------------- +# Metrics +# --------------------------------------------------------------------------- +def _common(metric: ResolvedMetric) -> dict[str, Any]: + entry: dict[str, Any] = {"label": metric.label} + if metric.description: + entry["description"] = metric.description.strip() + if metric.ai_context: + entry["ai_hint"] = metric.ai_context.strip() + if metric.format in FORMATS: + entry["format"] = FORMATS[metric.format] + return entry + + +def _simple(state: _Export, name: str, metric: ResolvedMetric) -> None: + assert metric.table is not None + home = metric.table + entity = state.semantic.entities[home] + filters, sql_where = _where(state, metric.where, home) if metric.where else ([], None) + entry: dict[str, Any] = {} + if metric.kind == "simple": + assert metric.column is not None and metric.agg is not None + column = metric.column + entry["type"] = TYPES[metric.agg] + value = state.ref(home, home, column) + elif len(entity.primary_key) == 1: + # A one-column key is never null and never repeats: counting it distinctly + # counts the rows, and stays right in an explore that joins this table. + column = entity.primary_key[0] + entry["type"] = "count_distinct" + value = state.ref(home, home, column) + else: + column = None + entry["type"] = "count" + value = "1" + if sql_where is not None: + entry["sql"] = f"CASE WHEN {sql_where} THEN {value} END" + elif column is None: + entry["sql"] = value + entry.update(_common(metric)) + if filters: + entry["filters"] = filters + if metric.time is not None: + time_table, _, time_column = metric.time.rpartition(".") + if time_table == home: + entry["default_time_dimension"] = {"field": time_column, "interval": DEFAULT_INTERVAL} + else: + state.dated_elsewhere.setdefault(metric.time, []).append(name) + state.needs[name] = set(metric.joins) + if column is None: + state.model_metrics[home][name] = entry + else: + state.column_metrics.setdefault((home, column), {})[name] = entry + + +def _number(state: _Export, name: str, metric: ResolvedMetric) -> None: + semantic = state.semantic + inputs = semantic.simple_inputs(name) + tables = list(dict.fromkeys(semantic.metrics[i].table for i in inputs)) + if len(tables) != 1: + named = ", ".join(state.names[t] for t in tables if t is not None) + state.losses.append( + Loss( + f"metric {name}", + "inputs", + f"its inputs are on {named}; a Lightdash metric belongs to one table and sees " + "another only through a join from it, which leaves out rows nothing joins to " + "and repeats rows joined more than once, so it is not exported", + ) + ) + return + home = tables[0] + assert home is not None + entry: dict[str, Any] = {"type": "number", "sql": _number_sql(state, name, top=True)} + entry.update(_common(metric)) + state.needs[name] = set().union(*(state.needs[i] for i in inputs)) + state.model_metrics[home][name] = entry + + +def _number_sql(state: _Export, name: str, top: bool = False) -> str: + """A ratio or derived metric over its simple inputs, `${input}`, inner ones written out.""" + metric = state.semantic.metrics[name] + if metric.kind in ("simple", "count"): + return "${" + name + "}" + if metric.kind == "ratio": + assert metric.numerator is not None and metric.denominator is not None + text = ( + f"{_number_sql(state, metric.numerator)} / " + f"NULLIF({_number_sql(state, metric.denominator)}, 0)" + ) + return text if top else f"({text})" + assert metric.node is not None + return expr.to_sql(metric.node, lambda dep: _number_sql(state, dep)).replace( + "nullif(", "NULLIF(" + ) + + +# --------------------------------------------------------------------------- +# Filters +# --------------------------------------------------------------------------- +def _where(state: _Export, where: Where, home: str) -> tuple[list[dict[str, Any]], Optional[str]]: + """A filter as Lightdash `filters`, and the SQL condition of the terms they cannot say.""" + filters: list[dict[str, Any]] = [] + rest: list[Union[Condition, Where]] = [] + for term in where.terms: + found = _condition_filters(state, term, home) if isinstance(term, Condition) else None + if found is None: + rest.append(term) + else: + filters.extend(found) + if not rest: + return filters, None + + def column_of(path: str) -> tuple[str, ColumnKind]: + table, _, column = path.rpartition(".") + return state.ref(table, home, column), state.kind(path) + + return filters, where_sql(Where(terms=rest), column_of) + + +def _text(value: Any, kind: ColumnKind) -> str: + """A value as Lightdash's filters hold it: always a string.""" + if isinstance(value, bool): + return "true" if value else "false" + if kind in ("integer", "decimal") and isinstance(value, float): + return format(Decimal(repr(value)), "f") + return str(value) + + +def _negatable(text: str) -> bool: + return ( + text != "" + and text == text.strip() + and not _UNSAFE.search(text) + and not text.startswith(_KEYWORDS) + ) + + +def _condition_filters( + state: _Export, condition: Condition, home: str +) -> Optional[list[dict[str, Any]]]: + """A condition as Lightdash filters (every one must hold), or None when they cannot say it.""" + table, _, column = condition.column.rpartition(".") + key = state.filter_key(table, home, column) + kind = state.kind(condition.column) + out: list[dict[str, Any]] = [] + for operator, operand in condition.operators.items(): + if operator == "is_null": + out.append({key: "null" if operand else "!null"}) + continue + if is_temporal(kind): + return None + if operator in ("eq", "in"): + values = [operand] if operator == "eq" else list(operand) + if kind == "boolean" and len(set(values)) > 1: + return None # Lightdash compares a boolean with the first value only + out.append({key: [_text(value, kind) for value in values]}) + elif operator in ("ne", "not_in"): + values = [operand] if operator == "ne" else list(operand) + texts = [_text(value, kind) for value in values] + if not all(_negatable(text) for text in texts): + return None + out.extend({key: f"!{text}"} for text in texts) + out.append({key: "!null"}) + elif operator == "between": + low, high = (_text(value, kind) for value in operand) + out.append({key: f"between {low} and {high}"}) + else: # gt, gte, lt, lte: on a number, as the spec allows no other kind here + sign = {"gt": ">", "gte": ">=", "lt": "<", "lte": "<="}[operator] + out.append({key: f"{sign} {_text(operand, kind)}"}) + # `ne` and `is_null: false` on one column both say `!null`: once is enough. + return [item for i, item in enumerate(out) if item not in out[:i]] + + +# --------------------------------------------------------------------------- +# Models +# --------------------------------------------------------------------------- +def _joins(state: _Export, key: str) -> list[tuple[str, JoinPath]]: + """Every table `key` reaches along exactly one path, nearest first, with the path.""" + found: list[tuple[str, JoinPath]] = [] + ambiguous: list[str] = [] + for other in state.semantic.entities: + if other == key: + continue + paths = state.graph.paths(key, other) + if len(paths) == 1: + found.append((other, paths[0])) + elif paths: + ambiguous.append(other) + if ambiguous: + state.losses.append( + Loss( + f"table {state.names[key]}", + "joins", + f"{', '.join(state.names[t] for t in ambiguous)} " + f"{'is' if len(ambiguous) == 1 else 'are'} reached along more than one path; " + "a Lightdash explore joins a table once, so its explore does not join " + f"{'it' if len(ambiguous) == 1 else 'them'}", + ) + ) + found.sort(key=lambda item: len(item[1])) + return found + + +def _model(state: _Export, key: str) -> dict[str, Any]: + semantic = state.semantic + entity = semantic.entities[key] + table = semantic.model.tables[key] + meta: dict[str, Any] = {"label": _label(key)} + if entity.primary_key: + meta["primary_key"] = ( + entity.primary_key[0] if len(entity.primary_key) == 1 else list(entity.primary_key) + ) + joins = _joins(state, key) + joined = {other for other, _ in joins} + entries = [] + for other, path in joins: + step = path[-1] + on = " AND ".join( + "${" + f"{state.names[step.child]}.{child}" + "} = ${" + f"{state.names[other]}.{parent}" + "}" + for child, parent in zip(step.child_columns, step.parent_columns, strict=True) + ) + join: dict[str, Any] = { + "join": state.names[other], + "sql_on": on, + "relationship": "one-to-one" if _one_to_one(semantic, step) else "many-to-one", + } + hidden = [ + name for name in _metrics_on(state, other) if not state.needs[name] <= joined | {key} + ] + if hidden: + join["fields"] = [ + *semantic.model.tables[other].columns, + *(name for name in _metrics_on(state, other) if name not in hidden), + ] + state.losses.append( + Loss( + f"table {state.names[key]}", + "joined metrics", + f"its explore joins {state.names[other]} without " + f"{', '.join(hidden)}: {'it filters' if len(hidden) == 1 else 'they filter'} " + "on a table the explore does not join", + ) + ) + entries.append(join) + if entries: + meta["joins"] = entries + if state.model_metrics[key]: + meta["metrics"] = state.model_metrics[key] + + out: dict[str, Any] = {"name": state.names[key]} + if entity.description: + out["description"] = entity.description.strip() + out["config"] = {"meta": meta} + dimensions = {d.column: d for d in semantic.dimensions if d.table == key} + columns = [] + for name, column in table.columns.items(): + item: dict[str, Any] = {"name": identifier(name)} + if column.description: + item["description"] = column.description.strip() + column_meta: dict[str, Any] = {} + dimension = dimensions.get(name) + if dimension is None: + column_meta["dimension"] = {"hidden": True} + else: + column_meta["dimension"] = {"label": dimension.label} + if dimension.description and dimension.description != column.description: + column_meta["dimension"]["description"] = dimension.description.strip() + if (key, name) in state.column_metrics: + column_meta["metrics"] = state.column_metrics[(key, name)] + item["config"] = {"meta": column_meta} + columns.append(item) + out["columns"] = columns + return out + + +def _metrics_on(state: _Export, key: str) -> list[str]: + names = [ + n for (table, _), metrics in state.column_metrics.items() if table == key for n in metrics + ] + return names + list(state.model_metrics[key]) + + +def _one_to_one(semantic: SemanticModel, step: Any) -> bool: + return any( + r.one_to_one + for r in semantic.relationships + if (r.from_table, r.from_columns, r.to_table) + == (step.child, step.child_columns, step.parent) + ) + + +def _general_losses(state: _Export) -> None: + semantic = state.semantic + for key, entity in semantic.entities.items(): + extra = [item for item, value in (("role", entity.role), ("grain", entity.grain)) if value] + if extra: + state.losses.append( + Loss( + f"table {state.names[key]}", + " and ".join(extra), + "Lightdash tables have no " + " or ".join(extra), + ) + ) + enums = [ + f"{state.names[d.table]}.{d.column}" for d in semantic.dimensions if d.members is not None + ] + if enums: + state.losses.append( + Loss( + f"{'dimension' if len(enums) == 1 else 'dimensions'} {', '.join(enums)}", + "values", + "Lightdash dimensions have no list of allowed values (the generated project's " + "accepted_values tests still check them)", + ) + ) + currency = [ + name + for name, metric in semantic.metrics.items() + if name in state.needs and metric.format == "currency" + ] + if currency: + state.losses.append( + Loss( + f"{'metric' if len(currency) == 1 else 'metrics'} {', '.join(currency)}", + "currency", + "the metrics file names no currency, so the format is '#,##0.00' with no " + "symbol; write one in, as '[$€]#,##0.00'", + ) + ) + awkward = [ + f"{state.names[key]}.{name}" + for key, table in semantic.model.tables.items() + for name in table.columns + if not _PLAIN.match(name) + ] + if awkward: + state.losses.append( + Loss( + f"{'column' if len(awkward) == 1 else 'columns'} {', '.join(awkward)}", + "name", + "Lightdash refers to a field by a plain lowercase name, in ${...} and in filters; " + "a reference to this one may not resolve", + ) + ) + + +# --------------------------------------------------------------------------- +# Into a generated dbt project +# --------------------------------------------------------------------------- +STAGING = Path("models") / "staging" + + +def _first(entry: dict[str, Any], keys: tuple[str, ...]) -> dict[str, Any]: + """`entry` with `keys` moved to the front, in that order, the rest as they were.""" + head = {k: entry[k] for k in keys if k in entry} + return {**head, **{k: v for k, v in entry.items() if k not in head}} + + +def _unquoted(name: str) -> str: + if len(name) >= 2 and name.startswith('"') and name.endswith('"'): + return name[1:-1].replace('""', '"') + return name + + +def write_lightdash(dest: Path, export: LightdashExport) -> list[Path]: + """Merge the export into the staging models' own YAML in the dbt project at `dest`. + + dbt reads a model's properties from one file only, so the meta goes into + `models/staging/.yml`, the file `generate` wrote for it: the model's + `description` and `config.meta`, and each column's `config.meta`. Its tests + and everything else are kept. The paths written, in model order. + """ + written = [] + for model in export.models: + path = dest / STAGING / f"{model['name']}.yml" + document = yaml.safe_load(path.read_text(encoding="utf-8")) + entries = document["models"] + index = next(i for i, e in enumerate(entries) if e["name"] == model["name"]) + entry = dict(entries[index]) + if "description" in model: + entry.setdefault("description", model["description"]) + entry["config"] = {**entry.get("config", {}), **model["config"]} + by_name = {_unquoted(c["name"]): c for c in model["columns"]} + columns = [] + for column in entry.get("columns", []): + extra = by_name.get(_unquoted(column["name"])) + if extra is not None: + column = {**column, "config": {**column.get("config", {}), **extra["config"]}} + column = _first(column, ("name", "description", "config")) + columns.append(column) + entry["columns"] = columns + entries[index] = _first(entry, ("name", "description", "config", "columns")) + path.write_text(_dump(document), encoding="utf-8") + written.append(path) + return written diff --git a/model2data/spec/metrics/README.md b/model2data/spec/metrics/README.md index 2f8e6f6..c2ae440 100644 --- a/model2data/spec/metrics/README.md +++ b/model2data/spec/metrics/README.md @@ -304,5 +304,6 @@ Read against its model, a conforming document also meets these: This spec fixes what a metric means. How a consumer computes and publishes metrics is its own business; the [model2data engine](https://github.com/JB-Analytica/model2data) computes each one's known value over a generated run (`metric_values.json`), writes a dbt test per metric that checks -the warehouse gets the same number, and exports Apache Ossie 0.1.1 as tagged `osi-0.1.1-rc1` in -apache/ossie (no final 0.1.1 tag exists yet). Its README says how. +the warehouse gets the same number, exports Apache Ossie 0.1.1 as tagged `osi-0.1.1-rc1` in +apache/ossie (no final 0.1.1 tag exists yet), and writes Lightdash metrics into the dbt project's +YAML. Its README says how. diff --git a/pyproject.toml b/pyproject.toml index 94e9ee4..fa033d5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "model2data" -version = "1.13.0" +version = "1.14.0" description = "Turn a data model into realistic synthetic data, a runnable dbt project and metrics with known values" readme = "README_PYPI.md" requires-python = ">=3.10" diff --git a/tests/fixtures/lightdash/LICENSE b/tests/fixtures/lightdash/LICENSE new file mode 100644 index 0000000..b428712 --- /dev/null +++ b/tests/fixtures/lightdash/LICENSE @@ -0,0 +1,25 @@ +Copyright (c) 2021-present Telescope Technology Limited (trading as “Lightdash”) + +Portions of this software are licensed as follows: + +* All content that resides under the "packages/backend/src/ee" directory of this repository, if that directory exists, is licensed under the license defined in "packages/backend/src/ee/LICENSE". +* All third party components incorporated into the Lightdash Software are licensed under the original license provided by the owner of the applicable component. +* Content outside of the above mentioned directories or restrictions above is available under the "MIT" license as defined below. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/tests/fixtures/lightdash/VENDORED.md b/tests/fixtures/lightdash/VENDORED.md new file mode 100644 index 0000000..9eceed7 --- /dev/null +++ b/tests/fixtures/lightdash/VENDORED.md @@ -0,0 +1,11 @@ +# Vendored: Lightdash's JSON Schema for dbt YAML + +`lightdash-dbt-2.0.json` is `packages/common/src/schemas/json/lightdash-dbt-2.0.json` of +[lightdash/lightdash](https://github.com/lightdash/lightdash) at commit +`1a72186cb4cbcd27ed81ff2dce248a88598e7727` (29 September 2026), the schema Lightdash publishes for +the `meta` of dbt model properties files. It is copied unchanged. The tests validate every +Lightdash export model2data writes against it (`tests/test_metrics_lightdash.py`). + +It is licensed under the MIT License: see `LICENSE` beside it, copied from the same commit (the +MIT terms cover everything outside `packages/backend/src/ee`). It is test data only and is not +shipped in the model2data package. diff --git a/tests/fixtures/lightdash/lightdash-dbt-2.0.json b/tests/fixtures/lightdash/lightdash-dbt-2.0.json new file mode 100644 index 0000000..8fe9e47 --- /dev/null +++ b/tests/fixtures/lightdash/lightdash-dbt-2.0.json @@ -0,0 +1,2128 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema", + "type": "object", + "properties": { + "version": { + "type": "number", + "enum": [2] + }, + "models": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "docs": { + "type": "object", + "properties": { + "show": { + "type": "boolean" + } + } + }, + "tests": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object" + }, + { + "type": "string" + } + ] + } + }, + "data_tests": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object" + }, + { + "type": "string" + } + ] + } + }, + "meta": { + "$ref": "#/$defs/modelMeta" + }, + "columns": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "quote": { + "type": "boolean" + }, + "tests": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object" + }, + { + "type": "string" + } + ] + } + }, + "data_tests": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object" + }, + { + "type": "string" + } + ] + } + }, + "tags": { + "type": "array", + "items": { + "type": "string" + } + }, + "meta": { + "$ref": "#/$defs/columnMeta" + }, + "config": { + "type": "object", + "properties": { + "meta": { + "$ref": "#/$defs/columnMeta" + } + } + } + } + } + }, + "config": { + "type": "object", + "properties": { + "meta": { + "$ref": "#/$defs/modelMeta" + } + } + } + }, + "required": ["name"] + } + }, + "metrics": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "model": { + "type": "string" + }, + "label": { + "type": "string" + }, + "description": { + "type": "string" + }, + "type": { + "type": "string" + }, + "sql": { + "type": "string" + }, + "timestamp": { + "type": "string" + }, + "time_grains": { + "type": "array", + "items": { + "type": "string" + } + }, + "dimensions": { + "type": "array", + "items": { + "type": "string" + } + }, + "filters": { + "type": "array", + "items": { + "type": "object" + } + }, + "meta": { + "type": "object", + "properties": { + "hidden": { + "type": "boolean" + }, + "round": { + "type": "number", + "minimum": 0 + }, + "format": { + "type": "string" + }, + "separator": { + "$ref": "#/$defs/separator" + }, + "compact": { + "type": "string", + "enum": [ + "auto", + "thousands", + "millions", + "billions", + "trillions", + "thousand", + "million", + "billion", + "trillion", + "K", + "M", + "B", + "T", + "kilobytes", + "megabytes", + "gigabytes", + "terabytes", + "petabytes", + "kilobyte", + "megabyte", + "gigabyte", + "terabyte", + "petabyte", + "KB", + "MB", + "GB", + "TB", + "PB", + "kibibytes", + "mebibytes", + "gibibytes", + "tebibytes", + "pebibytes", + "kibibyte", + "mebibyte", + "gibibyte", + "tebibyte", + "pebibyte", + "KiB", + "MiB", + "GiB", + "TiB", + "PiB" + ] + }, + "urls": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}" + }, + "label": { + "type": "string" + } + } + } + }, + "image": { + "type": "object", + "description": "Displays the dimension value as an image", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the image URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}. Only fields selected in the query are available in row context." + } + }, + "required": ["url"] + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "show_underlying_values": { + "type": "array", + "items": { + "type": "string" + } + }, + "default_time_dimension": { + "type": "object", + "description": "Specifies the default time dimension field and interval to use for time-based analysis on this metric. If specified, both field and interval are required. If there is already a default time dimension set in the model, this will override it.", + "properties": { + "field": { + "type": "string", + "description": "The name of the field to use as the default time dimension" + }, + "interval": { + "type": "string", + "enum": [ + "DAY", + "WEEK", + "MONTH", + "YEAR" + ], + "description": "The default time interval to use when analyzing this time dimension" + } + }, + "required": ["field", "interval"] + }, + "spotlight": { + "type": "object", + "description": "Set the visibility and/or categories of a metric in Spotlight", + "properties": { + "visibility": { + "type": "string", + "enum": ["show", "hide"] + }, + "categories": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of categories for the metric in Spotlight" + }, + "filter_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for filtering this metric in the Metrics Explorer" + }, + "segment_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for segmenting this metric in the Metrics Explorer" + }, + "default_segment": { + "type": "string", + "description": "A dimension name pre-selected in the Segment by picker when this metric is opened in the Metrics Explorer. Must be one of segment_by (if set) and a segmentable dimension." + }, + "default_filter": { + "type": "object", + "description": "A single filter pre-applied when this metric is opened in the Metrics Explorer, e.g. { status: active } or { status: [active, pending] }. Uses the metric filter syntax; only is / is not operators are supported here.", + "additionalProperties": true, + "maxProperties": 1 + }, + "owner": { + "type": "string", + "description": "Email of the metric owner" + } + }, + "anyOf": [ + { + "required": ["visibility"] + }, + { + "required": ["categories"] + }, + { + "required": ["filter_by"] + }, + { + "required": ["segment_by"] + }, + { + "required": ["owner"] + } + ] + } + } + } + }, + "required": ["name"] + } + }, + "seeds": { + "type": "array", + "items": { + "type": "object" + } + }, + "snapshots": { + "type": "array", + "items": { + "type": "object" + } + }, + "tests": { + "type": "array", + "items": { + "type": "object" + } + }, + "unit_tests": { + "type": "array", + "items": { + "type": "object" + } + }, + "sources": { + "type": "array", + "items": { + "type": "object" + } + }, + "analyses": { + "type": "array", + "items": { + "type": "object" + } + }, + "exposures": { + "type": "array", + "items": { + "type": "object" + } + }, + "macros": { + "type": "array", + "items": { + "type": "object" + } + } + }, + "additionalProperties": false, + "required": [], + "$defs": { + "separator": { + "type": "string", + "enum": [ + "default", + "commaPeriod", + "spacePeriod", + "periodComma", + "noSeparatorPeriod", + "apostrophePeriod" + ] + }, + "aiHint": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "filterItem": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { "type": "string" }, + { "type": "array", "items": { "type": "string" } } + ] + }, + "properties": { + "required": { + "type": "boolean", + "description": "If true, this filter cannot be removed by users" + } + } + }, + "modelMeta": { + "type": "object", + "properties": { + "joins": { + "type": "array", + "items": { + "type": "object", + "properties": { + "join": { + "type": "string" + }, + "sql_on": { + "type": "string" + }, + "always": { + "type": "boolean" + }, + "type": { + "type": "string", + "enum": ["inner", "left", "right", "full"] + }, + "hidden": { + "type": "boolean" + }, + "alias": { + "type": "string" + }, + "label": { + "type": "string" + }, + "fields": { + "type": "array", + "items": { + "type": "string" + } + }, + "relationship": { + "type": "string", + "enum": [ + "one-to-many", + "many-to-one", + "one-to-one", + "many-to-many" + ] + } + }, + "required": ["join", "sql_on"] + } + }, + "order_fields_by": { + "type": "string", + "enum": ["index", "label"] + }, + "group_label": { + "type": "string", + "deprecated": true, + "description": "Deprecated: use groups instead" + }, + "groups": { + "type": "array", + "description": "Groups are used to group tables in the sidebar. You can create nested groups up to 5 levels. Define labels & descriptions for the group keys in lightdash.config.yml under table_groups.", + "items": { + "type": "string", + "minLength": 1 + }, + "maxItems": 5 + }, + "group_details": { + "type": "object", + "description": "Set up group_details so you can group your dimensions and metrics in the sidebar using the groups parameter. You can create nested groups up to 5 levels", + "patternProperties": { + "^[a-zA-Z0-9_]+$": { + "type": "object", + "properties": { + "label": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string" + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints inherited by fields in this group for AI-powered query generation" + } + }, + "required": ["label"] + } + } + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints for AI-powered query generation" + }, + "metrics": { + "type": "object", + "patternProperties": { + "^[a-z0-9_]+$": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "percentile", + "median", + "average", + "boolean", + "count", + "count_distinct", + "date", + "max", + "min", + "number", + "string", + "sum", + "percent_of_previous", + "percent_of_total", + "running_total", + "sum_distinct", + "average_distinct" + ] + }, + "label": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string", + "minLength": 1 + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints for AI-powered query generation" + }, + "sql": { + "type": "string", + "minLength": 1 + }, + "hidden": { + "type": "boolean" + }, + "distinct_keys": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "round": { + "description": "Rounds the metric to the specified number of decimal places", + "type": "number", + "minimum": 0 + }, + "format": { + "type": "string" + }, + "separator": { + "$ref": "#/$defs/separator" + }, + "percentile": { + "type": "number" + }, + "compact": { + "type": "string", + "enum": [ + "auto", + "thousands", + "millions", + "billions", + "trillions", + "thousand", + "million", + "billion", + "trillion", + "K", + "M", + "B", + "T", + "kilobytes", + "megabytes", + "gigabytes", + "terabytes", + "petabytes", + "kilobyte", + "megabyte", + "gigabyte", + "terabyte", + "petabyte", + "KB", + "MB", + "GB", + "TB", + "PB", + "kibibytes", + "mebibytes", + "gibibytes", + "tebibytes", + "pebibytes", + "kibibyte", + "mebibyte", + "gibibyte", + "tebibyte", + "pebibyte", + "KiB", + "MiB", + "GiB", + "TiB", + "PiB" + ] + }, + "urls": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}" + }, + "label": { + "type": "string" + } + } + } + }, + "image": { + "type": "object", + "description": "Displays the dimension value as an image", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the image URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}. Only fields selected in the query are available in row context." + }, + "width": { + "type": "number" + }, + "height": { + "type": "number" + }, + "fit": { + "type": "string" + } + }, + "required": ["url"] + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "show_underlying_values": { + "type": "array", + "items": { + "type": "string" + } + }, + "groups": { + "type": "array", + "description": "Groups are used to group dimensions and metrics in the sidebar. You can create nested groups up to 5 levels", + "items": { + "type": "string", + "minLength": 1 + }, + "maxItems": 5 + }, + "default_time_dimension": { + "type": "object", + "description": "Specifies the default time dimension field and interval to use for time-based analysis on this metric. If specified, both field and interval are required. If there is already a default time dimension set in the model, this will override it.", + "properties": { + "field": { + "type": "string", + "description": "The name of the field to use as the default time dimension" + }, + "interval": { + "type": "string", + "enum": [ + "DAY", + "WEEK", + "MONTH", + "YEAR" + ], + "description": "The default time interval to use when analyzing this time dimension" + } + }, + "required": ["field", "interval"] + }, + "required_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "any_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "spotlight": { + "type": "object", + "description": "Set the visibility and/or categories of a metric in Spotlight", + "properties": { + "visibility": { + "type": "string", + "enum": ["show", "hide"] + }, + "categories": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of categories for the metric in Spotlight" + }, + "filter_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for filtering this metric in the Metrics Explorer" + }, + "segment_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for segmenting this metric in the Metrics Explorer" + }, + "default_segment": { + "type": "string", + "description": "A dimension name pre-selected in the Segment by picker when this metric is opened in the Metrics Explorer. Must be one of segment_by (if set) and a segmentable dimension." + }, + "default_filter": { + "type": "object", + "description": "A single filter pre-applied when this metric is opened in the Metrics Explorer, e.g. { status: active } or { status: [active, pending] }. Uses the metric filter syntax; only is / is not operators are supported here.", + "additionalProperties": true, + "maxProperties": 1 + }, + "owner": { + "type": "string", + "description": "Email of the metric owner" + } + }, + "anyOf": [ + { + "required": ["visibility"] + }, + { + "required": ["categories"] + }, + { + "required": ["filter_by"] + }, + { + "required": ["segment_by"] + }, + { + "required": ["owner"] + } + ] + }, + "drivers": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Array of metrics that drive this metric. Use 'metric_name' for same-table or 'table.metric_name' for cross-table." + } + }, + "required": ["type", "sql"] + } + } + }, + "default_time_dimension": { + "type": "object", + "description": "Specifies the default time dimension field and interval to use for time-based analysis (on any metric in the model). If specified, both field and interval are required.", + "properties": { + "field": { + "type": "string", + "description": "The name of the field to use as the default time dimension" + }, + "interval": { + "type": "string", + "enum": ["DAY", "WEEK", "MONTH", "YEAR"], + "description": "The default time interval to use when analyzing this time dimension" + } + }, + "required": ["field", "interval"] + }, + "default_show_underlying_values": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Default show underlying values configuration that will be applied to all metrics in this model that don't have their own show_underlying_values defined" + }, + "pre_aggregates": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "dimensions": { + "type": "array", + "items": { + "type": "string" + } + }, + "metrics": { + "type": "array", + "items": { + "type": "string" + } + }, + "table": { + "type": "string", + "minLength": 1, + "pattern": "^(?:[A-Za-z_][A-Za-z0-9_$]*|\"(?:\"\"|[^\"\\u0000-\\u001F])+\"|`[^`\\u0000-\\u001F]+`)(?:\\.(?:[A-Za-z_][A-Za-z0-9_$]*|\"(?:\"\"|[^\"\\u0000-\\u001F])+\"|`[^`\\u0000-\\u001F]+`))*$", + "description": "External table: a qualified identifier for a customer-managed warehouse table or view that Lightdash does not create or refresh. It must expose the pre-aggregate's generated column names. When set, Lightdash never materializes this pre-aggregate; materialization keys (refresh, sorts, max_rows, materialization_role) are ignored." + }, + "sorts": { + "description": "Materialization sort order. When omitted, Lightdash automatically sorts the time dimension first and descending, then all remaining materialized dimensions ascending. Set to false or [] to disable sorting, or provide canonical query sort entries with a compiled Lightdash fieldId and explicit descending value. Compiled time-dimension IDs include granularity (for example, orders_order_date_day) and must be updated when granularity changes. Sort order determines which rows are retained when max_rows truncates results.", + "oneOf": [ + { + "const": false + }, + { + "type": "array", + "items": { + "type": "object", + "properties": { + "fieldId": { + "type": "string", + "minLength": 1 + }, + "descending": { + "description": "Whether to sort descending. Must be an unquoted YAML boolean.", + "enum": [true, false] + } + }, + "required": [ + "fieldId", + "descending" + ], + "additionalProperties": false + } + } + ] + }, + "filters": { + "type": "array", + "items": { + "type": "object" + } + }, + "time_dimension": { + "type": "string" + }, + "granularity": { + "type": "string" + }, + "max_rows": { + "type": "integer" + }, + "refresh": { + "type": "object", + "properties": { + "cron": { + "type": "string" + } + } + }, + "materialization_role": { + "type": "object", + "properties": { + "email": { + "type": "string" + }, + "attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + } + }, + "required": ["email", "attributes"], + "additionalProperties": false + } + }, + "required": ["name", "dimensions", "metrics"] + } + }, + "spotlight": { + "type": "object", + "description": "Set the visibility and/or categories of a metric in Spotlight", + "properties": { + "visibility": { + "type": "string", + "enum": ["show", "hide"] + }, + "categories": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An optional array of categories for all metrics in this model in Spotlight" + }, + "owner": { + "type": "string", + "description": "Email of the model owner (inherited by all metrics)" + } + }, + "anyOf": [ + { + "required": ["visibility"] + }, + { + "required": ["categories"] + }, + { + "required": ["owner"] + } + ] + }, + "primary_key": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "owner": { + "type": "string", + "description": "Email of the model owner (inherited by all metrics). spotlight.owner takes precedence." + }, + "sql_where": { + "type": "string" + }, + "sql_filter": { + "type": "string" + }, + "sql_from": { + "type": "string" + }, + "required_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "any_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "parameters": { + "type": ["object", "null"], + "description": "Define parameters that can be used in SQL queries", + "patternProperties": { + "^[a-zA-Z0-9_-]+$": { + "type": "object", + "properties": { + "label": { + "description": "The label of the parameter as it will be displayed in the UI", + "type": "string" + }, + "type": { + "description": "The data type of the parameter (defaults to 'string' for backwards compatibility)", + "type": "string", + "enum": ["string", "number", "date"], + "default": "string" + }, + "description": { + "description": "A description of the parameter", + "type": "string" + }, + "options": { + "description": "A list of possible values for the parameter", + "type": "array", + "minItems": 1, + "items": { + "type": ["string", "number"] + } + }, + "default": { + "description": "The default value(s) for the parameter. For date parameters, `today` resolves to the current date at query time.", + "oneOf": [ + { + "type": ["string", "number"] + }, + { + "type": "array", + "items": { + "type": ["string", "number"] + } + } + ] + }, + "multiple": { + "description": "Whether the parameter input will be a multi select", + "type": "boolean" + }, + "allow_custom_values": { + "description": "Whether users can input custom values beyond predefined options", + "type": "boolean" + }, + "options_from_dimension": { + "description": "Get parameter options from a dimension in a model", + "type": "object", + "properties": { + "model": { + "description": "The model containing the dimension", + "type": "string" + }, + "dimension": { + "description": "The dimension to get options from", + "type": "string" + } + }, + "required": ["model", "dimension"] + } + }, + "required": ["label"], + "anyOf": [ + { + "required": ["options_from_dimension"], + "errorMessage": "Must have one of the following properties: allow_custom_values, options, options_from_dimension" + }, + { + "required": ["options"], + "errorMessage": "Must have one of the following properties: allow_custom_values, options, options_from_dimension" + }, + { + "properties": { + "allow_custom_values": { + "enum": [true] + } + }, + "required": ["allow_custom_values"], + "errorMessage": "Must have one of the following properties: allow_custom_values, options, options_from_dimension" + } + ] + } + } + }, + "default_filters": { + "type": "array", + "description": "Default filters that are automatically applied when exploring this model", + "items": { "$ref": "#/$defs/filterItem" } + }, + "required_filters": { + "type": "array", + "description": "Alias for default_filters (deprecated, use default_filters instead)", + "items": { "$ref": "#/$defs/filterItem" } + }, + "hidden": { + "type": "boolean", + "description": "When true, no explore is generated for the model itself. The model is still available as a join target and as the base table for explores defined under `explores`." + }, + "explores": { + "type": "object", + "description": "Defines custom explores for this model", + "patternProperties": { + "^[a-zA-Z0-9_]+$": { + "type": "object", + "properties": { + "default_filters": { + "type": "array", + "description": "Default filters that are automatically applied when exploring this explore", + "items": { "$ref": "#/$defs/filterItem" } + }, + "required_filters": { + "type": "array", + "description": "Alias for default_filters (deprecated, use default_filters instead)", + "items": { "$ref": "#/$defs/filterItem" } + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ], + "description": "Tags for organizing and filtering this explore. Overrides model tags when set." + }, + "sql_filter": { + "type": "string", + "description": "SQL filter applied to the WHERE clause of all queries using this explore" + }, + "sql_where": { + "type": "string", + "description": "Alias for sql_filter (deprecated, use sql_filter instead)" + }, + "group_label": { + "type": "string", + "deprecated": true, + "description": "Deprecated: use groups instead" + }, + "groups": { + "type": "array", + "description": "Groups are used to group tables in the sidebar. You can create nested groups up to 5 levels. Define labels & descriptions for the group keys in lightdash.config.yml under table_groups.", + "items": { + "type": "string", + "minLength": 1 + }, + "maxItems": 5 + } + } + } + } + } + } + }, + "columnMeta": { + "type": "object", + "properties": { + "metrics": { + "type": "object", + "patternProperties": { + "^[a-z0-9_]+$": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "percentile", + "median", + "average", + "boolean", + "count", + "count_distinct", + "date", + "max", + "min", + "number", + "string", + "sum", + "percent_of_previous", + "percent_of_total", + "running_total", + "sum_distinct", + "average_distinct" + ] + }, + "label": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string", + "minLength": 1 + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints for AI-powered query generation" + }, + "sql": { + "type": "string", + "minLength": 1 + }, + "hidden": { + "type": "boolean" + }, + "distinct_keys": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "round": { + "type": "number", + "minimum": 0 + }, + "format": { + "type": "string" + }, + "separator": { + "$ref": "#/$defs/separator" + }, + "percentile": { + "type": "number" + }, + "compact": { + "type": "string", + "enum": [ + "auto", + "thousands", + "millions", + "billions", + "trillions", + "thousand", + "million", + "billion", + "trillion", + "K", + "M", + "B", + "T", + "kilobytes", + "megabytes", + "gigabytes", + "terabytes", + "petabytes", + "kilobyte", + "megabyte", + "gigabyte", + "terabyte", + "petabyte", + "KB", + "MB", + "GB", + "TB", + "PB", + "kibibytes", + "mebibytes", + "gibibytes", + "tebibytes", + "pebibytes", + "kibibyte", + "mebibyte", + "gibibyte", + "tebibyte", + "pebibyte", + "KiB", + "MiB", + "GiB", + "TiB", + "PiB" + ] + }, + "urls": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}" + }, + "label": { + "type": "string" + } + } + } + }, + "image": { + "type": "object", + "description": "Displays the dimension value as an image", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the image URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}. Only fields selected in the query are available in row context." + } + }, + "required": ["url"] + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "show_underlying_values": { + "type": "array", + "items": { + "type": "string" + } + }, + "groups": { + "type": "array", + "description": "Groups are used to group dimensions and metrics in the sidebar. You can create nested groups up to 5 levels", + "items": { + "type": "string", + "minLength": 1 + }, + "maxItems": 5 + }, + "default_time_dimension": { + "type": "object", + "description": "Specifies the default time dimension field and interval to use for time-based analysis on this metric. If specified, both field and interval are required. If there is already a default time dimension set in the model, this will override it.", + "properties": { + "field": { + "type": "string", + "description": "The name of the field to use as the default time dimension" + }, + "interval": { + "type": "string", + "enum": [ + "DAY", + "WEEK", + "MONTH", + "YEAR" + ], + "description": "The default time interval to use when analyzing this time dimension" + } + }, + "required": ["field", "interval"] + }, + "required_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "any_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "spotlight": { + "type": "object", + "description": "Set the visibility and/or categories of a metric in Spotlight", + "properties": { + "visibility": { + "type": "string", + "enum": ["show", "hide"] + }, + "categories": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of categories for the metric in Spotlight" + }, + "filter_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for filtering this metric in the Metrics Explorer" + }, + "segment_by": { + "type": "array", + "items": { + "type": "string" + }, + "description": "An array of dimension names that are allowed for segmenting this metric in the Metrics Explorer" + }, + "default_segment": { + "type": "string", + "description": "A dimension name pre-selected in the Segment by picker when this metric is opened in the Metrics Explorer. Must be one of segment_by (if set) and a segmentable dimension." + }, + "default_filter": { + "type": "object", + "description": "A single filter pre-applied when this metric is opened in the Metrics Explorer, e.g. { status: active } or { status: [active, pending] }. Uses the metric filter syntax; only is / is not operators are supported here.", + "additionalProperties": true, + "maxProperties": 1 + }, + "owner": { + "type": "string", + "description": "Email of the metric owner" + } + }, + "anyOf": [ + { + "required": ["visibility"] + }, + { + "required": ["categories"] + }, + { + "required": ["filter_by"] + }, + { + "required": ["segment_by"] + }, + { + "required": ["owner"] + } + ] + }, + "drivers": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Array of metrics that drive this metric. Use 'metric_name' for same-table or 'table.metric_name' for cross-table." + } + }, + "required": ["type"] + } + } + }, + "dimension": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "string", + "number", + "timestamp", + "date", + "boolean" + ] + }, + "label": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string", + "minLength": 1 + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints for AI-powered query generation" + }, + "sql": { + "type": "string", + "minLength": 1 + }, + "convert_timezone": { + "type": "boolean", + "description": "When false, this dimension is rendered in its raw warehouse value and is not converted to the project timezone for display. Filter SQL still converts. Defaults to true." + }, + "timestamp_domain": { + "type": "string", + "enum": ["aware", "naive"], + "description": "Declares whether this timestamp column stores an instant ('aware') or a bare wall clock ('naive'); overrides the warehouse catalog." + }, + "hidden": { + "type": "boolean" + }, + "round": { + "type": "number", + "minimum": 0 + }, + "format": { + "type": "string" + }, + "separator": { + "$ref": "#/$defs/separator" + }, + "time_intervals": { + "anyOf": [ + { + "type": "array", + "items": { + "anyOf": [ + { + "type": "string", + "enum": [ + "RAW", + "DAY", + "WEEK", + "MONTH", + "QUARTER", + "YEAR", + "HOUR", + "MINUTE", + "SECOND", + "MILLISECOND", + "WEEK_NUM", + "MONTH_NUM", + "MONTH_NAME", + "DAY_OF_WEEK_NAME", + "QUARTER_NAME", + "DAY_OF_WEEK_INDEX", + "DAY_OF_MONTH_NUM", + "DAY_OF_YEAR_NUM", + "QUARTER_NUM", + "YEAR_NUM", + "HOUR_OF_DAY_NUM", + "MINUTE_OF_HOUR_NUM" + ] + }, + { + "type": "string" + } + ] + } + }, + { + "type": "string", + "enum": ["default", "OFF"] + } + ] + }, + "groups": { + "type": "array", + "description": "Groups are used to group dimensions and metrics in the sidebar. You can create nested groups up to 5 levels", + "items": { + "type": "string", + "minLength": 1 + }, + "maxItems": 5 + }, + "compact": { + "type": "string", + "enum": [ + "auto", + "thousands", + "millions", + "billions", + "trillions", + "thousand", + "million", + "billion", + "trillion", + "K", + "M", + "B", + "T", + "kilobytes", + "megabytes", + "gigabytes", + "terabytes", + "petabytes", + "kilobyte", + "megabyte", + "gigabyte", + "terabyte", + "petabyte", + "KB", + "MB", + "GB", + "TB", + "PB", + "kibibytes", + "mebibytes", + "gibibytes", + "tebibytes", + "pebibytes", + "kibibyte", + "mebibyte", + "gibibyte", + "tebibyte", + "pebibyte", + "KiB", + "MiB", + "GiB", + "TiB", + "PiB" + ] + }, + "urls": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}" + }, + "label": { + "type": "string" + } + } + } + }, + "image": { + "type": "object", + "description": "Displays the dimension value as an image", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the image URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}. Only fields selected in the query are available in row context." + } + }, + "required": ["url"] + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "required_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "any_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "spotlight": { + "type": "object", + "description": "Configure dimension visibility in the Metrics Explorer", + "properties": { + "filter_by": { + "type": "boolean", + "description": "Whether this dimension can be used for filtering in the Metrics Explorer. Defaults to true." + }, + "segment_by": { + "type": "boolean", + "description": "Whether this dimension can be used for segmenting in the Metrics Explorer. Defaults to true." + } + } + }, + "filter_autocomplete": { + "type": "object", + "description": "Configure static filter autocomplete suggestions and warehouse fetching for this dimension.", + "properties": { + "values": { + "type": "array", + "items": { + "type": "object", + "properties": { + "value": { + "type": "string" + }, + "label": { + "type": "string" + } + }, + "required": ["value"] + } + }, + "fetch_from_warehouse": { + "type": "boolean", + "description": "When false, filter autocomplete will not fetch values from the warehouse for this dimension. Defaults to true." + }, + "label_dimension": { + "type": "string", + "description": "Name of another dimension in the same table whose value is shown as the display label for each warehouse-fetched value." + }, + "options_from_dimension": { + "type": "object", + "description": "Fetch autocomplete values from a dimension in another model. Cascading filters are not applied.", + "properties": { + "model": { + "type": "string" + }, + "dimension": { + "type": "string" + }, + "label_dimension": { + "type": "string" + } + }, + "required": ["model", "dimension"] + } + } + } + } + }, + "additional_dimensions": { + "type": "object", + "patternProperties": { + "^[a-z0-9_]+$": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": [ + "string", + "number", + "timestamp", + "date", + "boolean" + ] + }, + "label": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string", + "minLength": 1 + }, + "ai_hint": { + "$ref": "#/$defs/aiHint", + "description": "Hints for AI-powered query generation" + }, + "sql": { + "type": "string", + "minLength": 1 + }, + "time_intervals": { + "anyOf": [ + { + "type": "array", + "items": { + "anyOf": [ + { + "type": "string", + "enum": [ + "RAW", + "DAY", + "WEEK", + "MONTH", + "QUARTER", + "YEAR", + "HOUR", + "MINUTE", + "SECOND", + "MILLISECOND", + "WEEK_NUM", + "MONTH_NUM", + "MONTH_NAME", + "DAY_OF_WEEK_NAME", + "QUARTER_NAME", + "DAY_OF_WEEK_INDEX", + "DAY_OF_MONTH_NUM", + "DAY_OF_YEAR_NUM", + "QUARTER_NUM", + "YEAR_NUM", + "HOUR_OF_DAY_NUM", + "MINUTE_OF_HOUR_NUM" + ] + }, + { + "type": "string" + } + ] + } + }, + { + "type": "string", + "enum": ["default", "OFF"] + } + ] + }, + "convert_timezone": { + "type": "boolean", + "description": "When false, this dimension is rendered in its raw warehouse value and is not converted to the project timezone for display. Filter SQL still converts. Defaults to true." + }, + "timestamp_domain": { + "type": "string", + "enum": ["aware", "naive"], + "description": "Declares whether this timestamp column stores an instant ('aware') or a bare wall clock ('naive'); overrides the warehouse catalog." + }, + "hidden": { + "type": "boolean" + }, + "round": { + "type": "number", + "minimum": 0 + }, + "format": { + "type": "string" + }, + "separator": { + "$ref": "#/$defs/separator" + }, + "compact": { + "type": "string", + "enum": [ + "auto", + "thousands", + "millions", + "billions", + "trillions", + "thousand", + "million", + "billion", + "trillion", + "K", + "M", + "B", + "T", + "kilobytes", + "megabytes", + "gigabytes", + "terabytes", + "petabytes", + "kilobyte", + "megabyte", + "gigabyte", + "terabyte", + "petabyte", + "KB", + "MB", + "GB", + "TB", + "PB", + "kibibytes", + "mebibytes", + "gibibytes", + "tebibytes", + "pebibytes", + "kibibyte", + "mebibyte", + "gibibyte", + "tebibyte", + "pebibyte", + "KiB", + "MiB", + "GiB", + "TiB", + "PiB" + ] + }, + "group_label": { + "type": "string", + "deprecated": true, + "description": "Deprecated: use groups instead" + }, + "urls": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}" + }, + "label": { + "type": "string" + } + } + } + }, + "image": { + "type": "object", + "description": "Displays the dimension value as an image", + "properties": { + "url": { + "type": "string", + "description": "Liquidjs template for the image URL. Available variables: ${value.raw}, ${value.formatted}, ${row.table_name.field_name.raw}, ${row.table_name.field_name.formatted}. Only fields selected in the query are available in row context." + } + }, + "required": ["url"] + }, + "tags": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + }, + "required_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "any_attributes": { + "type": "object", + "properties": {}, + "additionalProperties": { + "oneOf": [ + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "string" + } + ] + } + }, + "filter_autocomplete": { + "type": "object", + "description": "Configure static filter autocomplete suggestions and warehouse fetching for this dimension.", + "properties": { + "values": { + "type": "array", + "items": { + "type": "object", + "properties": { + "value": { + "type": "string" + }, + "label": { + "type": "string" + } + }, + "required": ["value"] + } + }, + "fetch_from_warehouse": { + "type": "boolean", + "description": "When false, filter autocomplete will not fetch values from the warehouse for this dimension. Defaults to true." + }, + "label_dimension": { + "type": "string", + "description": "Name of another dimension in the same table whose value is shown as the display label for each warehouse-fetched value." + }, + "options_from_dimension": { + "type": "object", + "description": "Fetch autocomplete values from a dimension in another model. Cascading filters are not applied.", + "properties": { + "model": { + "type": "string" + }, + "dimension": { + "type": "string" + }, + "label_dimension": { + "type": "string" + } + }, + "required": ["model", "dimension"] + } + } + } + } + } + } + } + } + } + } +} diff --git a/tests/test_metrics_cli.py b/tests/test_metrics_cli.py index ab9eed9..08ca607 100644 --- a/tests/test_metrics_cli.py +++ b/tests/test_metrics_cli.py @@ -294,6 +294,63 @@ def test_metrics_export_without_losses_prints_none(tmp_path): assert "Not expressible" not in result.output +def test_metrics_export_prints_lightdash(tmp_path): + args = ["metrics", "export", "-f", COFFEE, "--metrics", COFFEE_METRICS, "--to", "lightdash"] + result = _invoke(args, tmp_path) + assert result.exit_code == 0, result.output + assert result.output.startswith("# Lightdash metrics and dimensions (dbt config.meta)") + document = yaml.safe_load(result.output) + assert document["version"] == 2 + assert [m["name"] for m in document["models"]][:2] == ["stg_customers", "stg_products"] + + +def test_metrics_export_writes_a_lightdash_file(tmp_path): + out = tmp_path / "lightdash.yml" + args = ["metrics", "export", "-f", COFFEE, "--metrics", COFFEE_METRICS] + result = _invoke([*args, "--to", "Lightdash", "-o", out], tmp_path) + assert result.exit_code == 0, result.output + assert f"✅ Wrote {out} (Lightdash, dbt YAML, 11 metrics)" in result.output + assert "ℹ️ Not expressible in Lightdash, left out:" in result.output + assert " - metrics revenue, average_order_value: currency:" in result.output + assert yaml.safe_load(out.read_text())["models"][2]["name"] == "stg_orders" + + +def test_generate_with_lightdash_changes_only_the_staging_yaml(tmp_path): + plain, lit = tmp_path / "plain", tmp_path / "lightdash" + plain.mkdir() + lit.mkdir() + first = _invoke(["generate", "-f", COFFEE, *COMMON, "--metrics", COFFEE_METRICS], plain) + assert first.exit_code == 0, first.output + second = _invoke( + ["generate", "-f", COFFEE, *COMMON, "--metrics", COFFEE_METRICS, "--lightdash"], lit + ) + assert second.exit_code == 0, second.output + before, after = _hashes(plain), _hashes(lit) + assert set(before) == set(after) + changed = sorted(path for path in before if before[path] != after[path]) + assert changed == [ + f"dbt_shop/models/staging/stg_{name}.yml" + for name in sorted(["customers", "orders", "order_items", "products", "product_reviews"]) + ] + assert "🔦 Adding Lightdash metrics and dimensions to the staging models..." in second.output + assert "Lightdash: 11 metrics (models/staging/stg_*.yml, config.meta)" in ( + second.output + ) + assert "ℹ️ Not expressible in Lightdash, left out:" in second.output + orders = yaml.safe_load((lit / "dbt_shop/models/staging/stg_orders.yml").read_text()) + (entry,) = orders["models"] + assert entry["config"]["meta"]["metrics"]["average_order_value"]["type"] == "number" + # The tests generate wrote are still there. + assert entry["columns"][0]["tests"] == ["not_null", "unique"] + + +def test_generate_with_lightdash_and_no_metrics_file_uses_the_inferred_metrics(tmp_path): + result = _invoke(["generate", "-f", COFFEE, *COMMON, "--lightdash"], tmp_path) + assert result.exit_code == 0, result.output + assert "Lightdash: 4 metrics" in result.output + assert not (tmp_path / "dbt_shop" / "metric_values.json").exists() + + def test_metrics_export_refuses_an_unknown_format(tmp_path): result = _invoke(["metrics", "export", "-f", COFFEE, "--to", "metricflow"], tmp_path) assert result.exit_code == 2 diff --git a/tests/test_metrics_dbt.py b/tests/test_metrics_dbt.py index 7c547e3..5e0a069 100644 --- a/tests/test_metrics_dbt.py +++ b/tests/test_metrics_dbt.py @@ -8,6 +8,9 @@ and on the training model with its defects (orphans, nulls, invalid values, messy text, duplicate keys) over several days. One more edits a known value and requires its test to fail, so a passing test is known to check something. +The shipped examples with `--lightdash` must `dbt parse` cleanly, and, when +the Lightdash CLI is on PATH too, compile every explore with +`lightdash compile --no-partial-compilation`. Run with the project's own dbt first on PATH (`PATH="$PWD/.venv/bin:$PATH"`). """ @@ -25,6 +28,7 @@ from model2data.cli import main as generate_cli DBT = shutil.which("dbt") +LIGHTDASH = shutil.which("lightdash") ROOT = Path(__file__).resolve().parent.parent SPEC = ROOT / "model2data" / "spec" / "examples" FIXTURES = ROOT / "tests" / "fixtures" / "metrics" @@ -121,3 +125,68 @@ def test_a_wrong_known_value_fails_its_test(tmp_path, monkeypatch): statuses = _metric_tests(_build(project)) assert statuses.pop("metric_revenue") == "fail" assert set(statuses.values()) == {"pass"} + + +@pytest.mark.parametrize( + "model, metrics", + [ + pytest.param( + SPEC / "coffee_webshop.model2data.yml", + SPEC / "coffee_webshop.metrics.yml", + id="coffee-example", + ), + pytest.param( + ROOT / "examples" / "ecommerce.model2data.yml", + ROOT / "examples" / "ecommerce.metrics.yml", + id="ecommerce-example", + ), + pytest.param( + FIXTURES / "shop.model2data.yml", + FIXTURES / "shop.metrics.yml", + id="every-kind-and-join", + ), + ], +) +def test_the_lightdash_meta_parses_and_compiles(tmp_path, monkeypatch, model, metrics): + monkeypatch.chdir(tmp_path) + generate_cli( + file=model, rows=40, seed=2, force=True, name="lit", metrics_file=metrics, lightdash=True + ) + project = tmp_path / "dbt_lit" + parse = subprocess.run( + [str(DBT), "parse", "--profiles-dir", ".", "--no-partial-parse"], + cwd=project, + capture_output=True, + text=True, + timeout=300, + ) + assert parse.returncode == 0, parse.stdout + parse.stderr + assert "warn" not in parse.stdout.lower(), parse.stdout + manifest = json.loads((project / "target" / "manifest.json").read_text()) + metas = [ + node["config"]["meta"] + for node in manifest["nodes"].values() + if node["resource_type"] == "model" + ] + assert metas and all("label" in meta for meta in metas) + if LIGHTDASH is None: + return + compiled = subprocess.run( + [ + LIGHTDASH, + "compile", + "--project-dir", + ".", + "--profiles-dir", + ".", + "--no-warehouse-credentials", + "--no-partial-compilation", + "--no-version-check", + ], + cwd=project, + capture_output=True, + text=True, + timeout=300, + ) + assert compiled.returncode == 0, compiled.stdout + compiled.stderr + assert "ERRORS=0" in compiled.stdout + compiled.stderr diff --git a/tests/test_metrics_lightdash.py b/tests/test_metrics_lightdash.py new file mode 100644 index 0000000..a193c1c --- /dev/null +++ b/tests/test_metrics_lightdash.py @@ -0,0 +1,645 @@ +"""The Lightdash export: the mapping, the filters, the lossiness report, and the vendored schema. + +`fixtures/lightdash/lightdash-dbt-2.0.json` is Lightdash's own JSON Schema for +the meta of dbt YAML (see `fixtures/lightdash/VENDORED.md`); every document +exported here validates against it. +""" + +import json +from pathlib import Path + +import jsonschema +import pytest +import yaml + +from model2data.metrics import LightdashExport, load, resolve, to_lightdash, write_lightdash +from model2data.model import load as load_model + +ROOT = Path(__file__).resolve().parent.parent +FIXTURES = ROOT / "tests" / "fixtures" +SPEC = ROOT / "model2data" / "spec" / "examples" +SHOP = load_model(FIXTURES / "metrics" / "shop.model2data.yml") +SCHEMA = json.loads((FIXTURES / "lightdash" / "lightdash-dbt-2.0.json").read_text(encoding="utf-8")) + + +def _valid(document: dict) -> None: + jsonschema.Draft7Validator(SCHEMA).validate(document) + + +def _coffee() -> LightdashExport: + model = load_model(SPEC / "coffee_webshop.model2data.yml") + return to_lightdash(resolve(model, load(SPEC / "coffee_webshop.metrics.yml", model))) + + +def _shop(text: str) -> LightdashExport: + header = "model2data-metrics: 0.1.0\nmodel: shop\ninfer: false\nmetrics:\n" + return to_lightdash(resolve(SHOP, load(header + text, SHOP))) + + +def _models(export: LightdashExport) -> dict[str, dict]: + return {model["name"]: model for model in export.models} + + +def _meta(export: LightdashExport, model: str) -> dict: + return _models(export)[model]["config"]["meta"] + + +def _columns(export: LightdashExport, model: str) -> dict[str, dict]: + return {c["name"]: c["config"]["meta"] for c in _models(export)[model]["columns"]} + + +def _metric(export: LightdashExport, name: str) -> dict: + """A metric wherever it is: on a column or on its model.""" + for model in export.models: + if name in model["config"]["meta"].get("metrics", {}): + return model["config"]["meta"]["metrics"][name] + for column in model["columns"]: + if name in column["config"]["meta"].get("metrics", {}): + return column["config"]["meta"]["metrics"][name] + raise KeyError(name) + + +def _losses(export: LightdashExport) -> dict[tuple[str, str], str]: + return {(loss.subject, loss.item): loss.reason for loss in export.lossiness} + + +# --------------------------------------------------------------------------- +# The schema +# --------------------------------------------------------------------------- +def test_the_vendored_schema_is_lightdashs_dbt_schema(): + jsonschema.Draft7Validator.check_schema(SCHEMA) + assert "modelMeta" in SCHEMA["$defs"] and "columnMeta" in SCHEMA["$defs"] + + +@pytest.mark.parametrize( + "model_path, metrics_path", + [(path, None) for path in sorted((ROOT / "examples").glob("*.model2data.yml"))] + + [ + ( + ROOT / "examples" / "ecommerce.model2data.yml", + ROOT / "examples" / "ecommerce.metrics.yml", + ), + (SPEC / "coffee_webshop.model2data.yml", SPEC / "coffee_webshop.metrics.yml"), + (FIXTURES / "metrics" / "shop.model2data.yml", FIXTURES / "metrics" / "shop.metrics.yml"), + ( + ROOT / "examples" / "ecommerce_training.model2data.yml", + FIXTURES / "metrics" / "ecommerce_training.metrics.yml", + ), + ], + ids=lambda path: path.name if path else "inferred", +) +def test_every_example_exports_a_valid_document(model_path, metrics_path): + model = load_model(model_path) + metrics = load(metrics_path, model) if metrics_path else None + export = to_lightdash(resolve(model, metrics)) + _valid(export.document) + text = export.to_yaml() + assert yaml.safe_load(text) == export.document + assert text.startswith("# Lightdash metrics and dimensions (dbt config.meta)") + + +# --------------------------------------------------------------------------- +# Tables, joins, dimensions +# --------------------------------------------------------------------------- +def test_the_models_their_joins_and_dimensions(): + export = _coffee() + assert [m["name"] for m in export.models] == [ + "stg_customers", + "stg_products", + "stg_orders", + "stg_order_items", + "stg_product_reviews", + ] + orders = _models(export)["stg_orders"] + assert orders["description"].startswith("One row per order") + assert orders["config"]["meta"]["label"] == "Orders" + assert orders["config"]["meta"]["primary_key"] == "id" + # Every table reached along one path, nearest first, a step at a time. + assert _meta(export, "stg_order_items")["joins"] == [ + { + "join": "stg_products", + "sql_on": "${stg_order_items.product_id} = ${stg_products.id}", + "relationship": "many-to-one", + }, + { + "join": "stg_orders", + "sql_on": "${stg_order_items.order_id} = ${stg_orders.id}", + "relationship": "many-to-one", + }, + { + "join": "stg_customers", + "sql_on": "${stg_orders.customer_id} = ${stg_customers.id}", + "relationship": "many-to-one", + }, + ] + assert "joins" not in _meta(export, "stg_customers") + products = _columns(export, "stg_products") + assert products["name"]["dimension"] == {"label": "Product"} # the metrics file's label + assert products["category"]["dimension"] == {"label": "Category"} + assert products["id"]["dimension"] == {"hidden": True} # a key: not a dimension + assert products["price"]["dimension"] == {"hidden": True} + # `customers.email: false` in the metrics file: never offered. + assert _columns(export, "stg_customers")["email"]["dimension"] == {"hidden": True} + name = next(c for c in _models(export)["stg_products"]["columns"] if c["name"] == "name") + assert name["description"] == "The blend or single-origin name printed on the bag" + + +def test_composite_keys_one_to_one_and_a_dimensions_own_description(): + model = load_model( + """ +model2data: 0.4.0 +name: keys +tables: + slots: + keys: + - {pk: [store_id, day]} + columns: + store_id: int + day: date + visits: + foreign_keys: + - {columns: [store_id, day], references: slots, to_columns: [store_id, day]} + columns: + id: {type: int, pk: true} + store_id: int + day: date + badges: + columns: + visit_id: {type: int, references: {to: visits.id, one_to_one: true}} + shiny: {type: boolean, description: Whether it shines} +""" + ) + metrics = load( + "model2data-metrics: 0.1.0\nmodel: keys\ndimensions:\n" + " badges.shiny: {label: Shiny, description: Polished to a shine}\n", + model, + ) + export = to_lightdash(resolve(model, metrics)) + _valid(export.document) + assert _meta(export, "stg_slots")["primary_key"] == ["store_id", "day"] + assert "primary_key" not in _meta(export, "stg_badges") + assert _meta(export, "stg_visits")["joins"] == [ + { + "join": "stg_slots", + "sql_on": "${stg_visits.store_id} = ${stg_slots.store_id} AND " + "${stg_visits.day} = ${stg_slots.day}", + "relationship": "many-to-one", + } + ] + assert _meta(export, "stg_badges")["joins"][0]["relationship"] == "one-to-one" + assert _columns(export, "stg_badges")["shiny"]["dimension"] == { + "label": "Shiny", + "description": "Polished to a shine", + } + + +def test_a_custom_model_name(): + model = load_model(SPEC / "coffee_webshop.model2data.yml") + export = to_lightdash(resolve(model), source=lambda entity: f"dim_{entity.name}") + assert _models(export)["dim_orders"]["config"]["meta"]["joins"][0] == { + "join": "dim_customers", + "sql_on": "${dim_orders.customer_id} = ${dim_customers.id}", + "relationship": "many-to-one", + } + + +# --------------------------------------------------------------------------- +# Metrics +# --------------------------------------------------------------------------- +def test_simple_metrics_row_counts_ratios_and_derived_metrics(): + export = _coffee() + revenue = _columns(export, "stg_orders")["total_amount"]["metrics"]["revenue"] + assert revenue == { + "type": "sum", + "label": "Revenue", + "description": "Order value of the orders that were not cancelled or left unpaid.", + "ai_hint": "What customers paid for. Use it for sales and turnover questions; it leaves " + "out pending and cancelled orders.", + "format": "#,##0.00", + "filters": [{"status": ["paid", "shipped", "delivered"]}], + "default_time_dimension": {"field": "order_date", "interval": "MONTH"}, + } + assert _columns(export, "stg_orders")["id"]["metrics"]["orders"]["type"] == "count_distinct" + # A row count counts the one-column key distinctly: the same number, and safe when joined. + assert _columns(export, "stg_order_items")["id"]["metrics"]["order_lines"] == { + "type": "count_distinct", + "label": "Order lines", + } + assert _metric(export, "web_revenue")["filters"] == [ + {"status": ["paid", "shipped", "delivered"]}, + {"channel": ["web"]}, + ] + assert _meta(export, "stg_orders")["metrics"] == { + "average_order_value": { + "type": "number", + "sql": "${revenue} / NULLIF(${orders}, 0)", + "label": "Average order value", + "format": "#,##0.00", + }, + "web_share": { + "type": "number", + "sql": "(${web_revenue} / NULLIF(${revenue}, 0))", + "label": "Web share of revenue", + "format": "0.00%", + }, + } + assert export.metrics == [ + "revenue", + "orders", + "order_lines", + "average_order_value", + "web_revenue", + "web_share", + "coffee_units_sold", + "orders_total_amount", + "orders_count", + "order_items_quantity", + "order_items_count", + ] + + +def test_every_aggregation_and_a_ratio_inside_an_expression(): + export = _shop( + """ + a: {measure: lines.qty, agg: sum} + b: {measure: lines.price, agg: average} + c: {measure: lines.price, agg: min} + d: {measure: lines.price, agg: max} + e: {measure: lines.price, agg: median} + f: {measure: orders.note, agg: count} + g: {measure: orders.note, agg: count_distinct} + r: {ratio: {numerator: d, denominator: c}} + x: {expression: "r * 2 - a", format: number} +""" + ) + _valid(export.document) + types = {name: _metric(export, name)["type"] for name in "abcdefg"} + assert types == { + "a": "sum", + "b": "average", + "c": "min", + "d": "max", + "e": "median", + "f": "count", + "g": "count_distinct", + } + assert _metric(export, "x") == { + "type": "number", + "sql": "(((${d} / NULLIF(${c}, 0)) * 2) - ${a})", + "label": "X", + } + + +def test_a_row_count_without_a_one_column_key_is_a_model_metric(): + model = load_model( + "model2data: 0.4.0\nname: logs\ntables:\n logs:\n role: fact\n columns:\n" + " level: int\n at: timestamp\n" + ) + metrics = load( + "model2data-metrics: 0.1.0\nmodel: logs\nmetrics:\n" + " warnings: {count: logs, where: {logs.at: {gte: '2026-01-01 00:00:00'}}}\n", + model, + ) + export = to_lightdash(resolve(model, metrics)) + _valid(export.document) + assert _meta(export, "stg_logs")["metrics"] == { + "warnings": { + "type": "count", + "sql": "CASE WHEN ${at} >= timestamp '2026-01-01 00:00:00' THEN 1 END", + "label": "Warnings", + "default_time_dimension": {"field": "at", "interval": "MONTH"}, + }, + "logs_count": { + "type": "count", + "sql": "1", + "label": "Logs count", + "description": "Rows of logs.", + "default_time_dimension": {"field": "at", "interval": "MONTH"}, + }, + } + + +# --------------------------------------------------------------------------- +# Filters +# --------------------------------------------------------------------------- +def test_filters_lightdash_can_say(): + export = _shop( + """ + m: + measure: lines.price + agg: sum + where: + lines.qty: {gt: 1, lte: 9} + lines.price: {gte: 2.5, lt: 100, between: [0, 1000.25]} + orders.status: {ne: cancelled} + orders.note: {not_in: [a, b], is_null: false} + customers.vip: true + customers.tier: [1, 2] + customers.country: {is_null: true} +""" + ) + _valid(export.document) + assert _metric(export, "m")["filters"] == [ + {"qty": "> 1"}, + {"qty": "<= 9"}, + {"price": ">= 2.5"}, + {"price": "< 100"}, + {"price": "between 0 and 1000.25"}, + {"stg_orders.status": "!cancelled"}, + {"stg_orders.status": "!null"}, # Lightdash's not-equal lets nulls through + {"stg_orders.note": "!a"}, + {"stg_orders.note": "!b"}, + {"stg_orders.note": "!null"}, + {"stg_customers.vip": ["true"]}, + {"stg_customers.tier": ["1", "2"]}, + {"stg_customers.country": "null"}, + ] + assert "sql" not in _metric(export, "m") + + +def test_filters_lightdash_cannot_say_become_sql(): + export = _shop( + """ + m: + count: lines + where: + orders.placed_at: {lt: "2100-01-01 00:00:00"} + customers.vip: {in: [true, false]} + orders.note: {ne: "50%, or so"} + customers.country: {not_in: [nullland]} + orders.status: paid + any: + - lines.qty: {gte: 3} + - customers.joined_on: "2020-01-01" +""" + ) + _valid(export.document) + metric = _metric(export, "m") + assert metric["filters"] == [{"stg_orders.status": ["paid"]}] + assert metric["sql"] == ( + "CASE WHEN (${stg_orders.placed_at} < timestamp '2100-01-01 00:00:00' and " + "${stg_customers.vip} in (true, false) and ${stg_orders.note} <> '50%, or so' and " + "${stg_customers.country} not in ('nullland') and " + "(${qty} >= 3 or ${stg_customers.joined_on} = date '2020-01-01')) THEN ${id} END" + ) + + +@pytest.mark.parametrize( + "value, filtered", + [ + ("plain words-and_more", True), + (" padded", False), + ("a,b", False), + ("100%", False), + ("x^y", False), + ('say "hi"', False), + ("back\\slash", False), + ("emptyish", False), + ("NULL island", False), + ], +) +def test_a_negated_value_is_a_filter_only_when_lightdash_reads_it_back(value, filtered): + text = json.dumps(value) + export = _shop(f" m: {{count: orders, where: {{orders.note: {{ne: {text}}}}}}}\n") + metric = _metric(export, "m") + if filtered: + assert metric["filters"] == [{"note": f"!{value}"}, {"note": "!null"}] + assert "sql" not in metric + else: + assert "filters" not in metric + assert metric["sql"].startswith("CASE WHEN ${note} <> ") + + +# --------------------------------------------------------------------------- +# Lossiness +# --------------------------------------------------------------------------- +def test_the_lossiness_of_the_coffee_example(): + losses = _losses(_coffee()) + assert set(losses) == { + ("metrics order_lines, coffee_units_sold, order_items_quantity, order_items_count", "time"), + ( + "dimensions stg_products.category, stg_products.origin, stg_products.roast, " + "stg_orders.channel, stg_orders.status", + "values", + ), + ("metrics revenue, average_order_value", "currency"), + } + assert losses[ + ("metrics order_lines, coffee_units_sold, order_items_quantity, order_items_count", "time") + ] == ( + "dated by orders.order_date, a column of stg_orders; a Lightdash metric's default time " + "dimension is a field of its own table" + ) + + +def test_singular_losses_roles_grain_and_awkward_names(): + model = load_model( + """ +model2data: 0.4.0 +name: odd +enums: + kind: [a, b] +tables: + user accounts: + role: dimension + columns: + id: {type: int, pk: true} + Display Name: text + events: + role: fact + grain: [user_id, at] + columns: + user_id: {type: int, references: user accounts.id} + at: timestamp + kind: kind + amount: {type: numeric, measure: true} +""" + ) + metrics = load( + "model2data-metrics: 0.1.0\nmodel: odd\nmetrics:\n" + " spend: {measure: events.amount, format: currency}\n", + model, + ) + export = to_lightdash(resolve(model, metrics)) + _valid(export.document) + losses = _losses(export) + assert losses[("table stg_user_accounts", "role")] == "Lightdash tables have no role" + assert losses[("table stg_events", "role and grain")] == ( + "Lightdash tables have no role or grain" + ) + assert ("dimension stg_events.kind", "values") in losses + assert ("metric spend", "currency") in losses + assert ("column stg_user_accounts.Display Name", "name") in losses + columns = [c["name"] for c in _models(export)["stg_user_accounts"]["columns"]] + assert columns == ["id", '"Display Name"'] + + +def test_plural_awkward_names(): + model = load_model( + "model2data: 0.4.0\nname: odd\ntables:\n t:\n columns:\n" + " Big: int\n also odd: int\n" + ) + losses = _losses(to_lightdash(resolve(model))) + assert ("columns stg_t.Big, stg_t.also odd", "name") in losses + + +def test_a_ratio_across_tables_is_not_exported(): + export = _shop( + """ + qty: {measure: lines.qty, agg: sum} + amount: {measure: orders.amount} + per_euro: {ratio: {numerator: qty, denominator: amount}} + per_euro_twice: {expression: per_euro * 2} +""" + ) + _valid(export.document) + assert export.metrics == ["qty", "amount"] + losses = _losses(export) + assert losses[("metric per_euro", "inputs")].startswith( + "its inputs are on stg_lines, stg_orders; a Lightdash metric belongs to one table" + ) + assert ("metric per_euro_twice", "inputs") in losses + with pytest.raises(KeyError): + _metric(export, "per_euro") + + +def test_tables_reached_along_two_paths_are_not_joined(): + export = to_lightdash(resolve(SHOP)) + losses = _losses(export) + assert losses[("table stg_transfers", "joins")] == ( + "stg_customers is reached along more than one path; a Lightdash explore joins a table " + "once, so its explore does not join it" + ) + assert "joins" not in _meta(export, "stg_transfers") + + +AMBIGUOUS = """ +model2data: 0.4.0 +name: amb +enums: + colour: [red, blue] +tables: + c: + columns: + id: {type: bigint, pk: true} + x: colour + d: + columns: + id: {type: bigint, pk: true} + y: colour + b: + columns: + id: {type: bigint, pk: true} + c_id: {type: bigint, references: c.id} + d_id: {type: bigint, references: d.id} + v: {type: int, measure: sum} + a: + columns: + id: {type: bigint, pk: true} + b_id: {type: bigint, references: b.id} + c_id: {type: bigint, references: c.id} + d_id: {type: bigint, references: d.id} +""" + + +def test_a_joined_tables_metrics_that_filter_on_an_unjoined_table_are_left_out_of_the_join(): + model = load_model(AMBIGUOUS) + metrics = load( + "model2data-metrics: 0.1.0\nmodel: amb\nmetrics:\n" + " red_v: {measure: b.v, where: {c.x: red}}\n" + " red_share: {ratio: {numerator: red_v, denominator: b_v}}\n", + model, + ) + export = to_lightdash(resolve(model, metrics)) + _valid(export.document) + losses = _losses(export) + assert losses[("table stg_a", "joins")] == ( + "stg_c, stg_d are reached along more than one path; a Lightdash explore joins a table " + "once, so its explore does not join them" + ) + assert losses[("table stg_a", "joined metrics")] == ( + "its explore joins stg_b without red_v, red_share: they filter on a table the explore " + "does not join" + ) + (join,) = _meta(export, "stg_a")["joins"] + assert join["join"] == "stg_b" + assert join["fields"] == ["id", "c_id", "d_id", "v", "b_v", "b_count"] + + +def test_a_single_metric_left_out_of_a_join(): + model = load_model(AMBIGUOUS) + metrics = load( + "model2data-metrics: 0.1.0\nmodel: amb\ninfer: false\nmetrics:\n" + " red_v: {measure: b.v, where: {c.x: red}}\n", + model, + ) + losses = _losses(to_lightdash(resolve(model, metrics))) + assert losses[("table stg_a", "joined metrics")] == ( + "its explore joins stg_b without red_v: it filters on a table the explore does not join" + ) + + +def test_no_losses_and_no_metrics(): + model = load_model("model2data: 0.4.0\nname: one\ntables:\n t:\n columns:\n id: int\n") + export = to_lightdash(resolve(model)) + assert export.lossiness == [] and export.metrics == [] + assert "Not expressible" not in export.to_yaml() + assert export.document == { + "version": 2, + "models": [ + { + "name": "stg_t", + "config": {"meta": {"label": "T"}}, + "columns": [{"name": "id", "config": {"meta": {"dimension": {"hidden": True}}}}], + } + ], + } + + +# --------------------------------------------------------------------------- +# Into a generated project +# --------------------------------------------------------------------------- +def test_write_lightdash_merges_into_the_staging_yaml(tmp_path): + staging = tmp_path / "models" / "staging" + staging.mkdir(parents=True) + model = load_model( + "model2data: 0.4.0\nname: odd\ntables:\n t:\n description: A table\n columns:\n" + " id: {type: int, pk: true}\n Big Name: boolean\n gone: int\n" + ) + (staging / "stg_t.yml").write_text( + yaml.safe_dump( + { + "version": 2, + "models": [ + {"name": "other"}, + { + "name": "stg_t", + "config": {"tags": ["x"]}, + "columns": [ + {"name": "id", "tests": ["unique"]}, + {"name": '"Big Name"', "config": {"tags": ["y"]}}, + {"name": "extra"}, + ], + }, + ], + }, + sort_keys=False, + ) + ) + export = to_lightdash(resolve(model)) + written = write_lightdash(tmp_path, export) + assert written == [staging / "stg_t.yml"] + document = yaml.safe_load(written[0].read_text()) + other, entry = document["models"] + assert other == {"name": "other"} + assert list(entry) == ["name", "description", "config", "columns"] + assert entry["description"] == "A table" + assert entry["config"] == {"tags": ["x"], "meta": {"label": "T", "primary_key": "id"}} + columns = entry["columns"] + assert columns[0] == { + "name": "id", + "config": {"meta": {"dimension": {"hidden": True}}}, + "tests": ["unique"], + } + assert columns[1]["config"] == {"tags": ["y"], "meta": {"dimension": {"label": "Big Name"}}} + assert columns[2] == {"name": "extra"} diff --git a/uv.lock b/uv.lock index 62b65b7..1151899 100644 --- a/uv.lock +++ b/uv.lock @@ -785,7 +785,7 @@ wheels = [ [[package]] name = "model2data" -version = "1.13.0" +version = "1.14.0" source = { editable = "." } dependencies = [ { name = "dbt-core" },