diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b0d4cf..312f247 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Changed +- **The determinism promise is per version.** The same model, seed, `--as-of` and options still + give the same files, byte for byte, on any machine and any day, with the same model2data + version. A new release may change what a seed produces, and when it does this changelog says + which models are affected and why. Pin the version to keep fixtures the same across upgrades. + (The README promised the same bytes across every 1.x release.) + +### Fixed +- **A column placed after another keeps its time shape.** A date or timestamp column with an + `after`, or one a created/updated/closed name pair places after another, was set to that + column plus a random gap, which threw away the draw its `business_hours` had shaped: + appointments `after` a booking with `business_hours: true` came out at 3am as often as at + 10am. The gap still decides roughly when; `business_hours` (the column's own, or the run's) + now decides the weekday and hour. The share of such a column in weekday working hours goes + from about 29%, the rate of a uniform draw, to about 72%, the same as a column drawn on its + own. The column's growth and seasonality still come from the column it follows. +- **No pile-up at `--as-of` midnight.** A gap that ran past the end of the window set the value + to exactly `--as-of` midnight, so about 1% of a dependent column's rows shared that one + timestamp. Such a gap is now drawn again, and after 20 tries the value is drawn evenly + between the column it follows and the window's end. A `when` column filled in on day 0 is + placed the same way. +- **Output changes for the same seed** in a model with a column placed after another, wherever + either fix applies: a dependent column with business hours (its own or the run's), and every + table where a row's gap ran past `--as-of` (about 1 row in 100, so most such tables beyond a + few hundred rows). The values drawn after the first such row in a table change too. A table + with no column placed after another writes the same bytes as in 1.11.0. + ## [1.11.0] - 2026-10-02 ### Added diff --git a/README.md b/README.md index 21ab6bc..395aecb 100644 --- a/README.md +++ b/README.md @@ -57,8 +57,9 @@ access required. - **Relationship-preserving.** Foreign keys resolve to real parent rows; tables are generated in dependency order. - **Deterministic, byte for byte.** The same model, `--seed` and `--as-of` (and the same options) - give the same files, byte for byte, on any machine and in every 1.x release — safe to commit - fixtures, safe to diff across CI runs and upgrades. See [The determinism promise](#the-determinism-promise). + give the same files, byte for byte, on any machine and any day with the same model2data + version — safe to commit fixtures, safe to diff across CI runs. See + [The determinism promise](#the-determinism-promise). - **Re-rollable one table at a time.** `--table-seed orders=7` regenerates a single table and leaves every other table byte-identical, so you can keep the four tables that look right. - **A real dbt project, not just CSVs.** Seeds, staging models that `ref()` them, schema tests, @@ -323,13 +324,15 @@ The same model, seed and `--as-of` — with the same options (`--rows`, `--rows- - on any machine and operating system, and on every supported Python version; - with any Faker or pandas version model2data accepts; -- in every 1.x release. A release may make generation faster or add options, but never changes - what an existing model, seed and set of options produce. A fix that cannot keep that waits for - a new major version, and the changelog says so. +- with the same model2data version, whenever you run it. + +A new release may change what a seed produces, to fix a bug or make the data more realistic. +When it does, the changelog says which models are affected and why. Pin the version +(`model2data==1.11.0`) when fixtures must stay the same across upgrades. `tests/test_determinism.py` holds the file hashes of a spread of models, presets, multi-day runs and locales, and runs on every pull request — against the newest Faker and pandas too — so a -change to a single output byte fails the build rather than reaching a release. +change to a single output byte fails the build unless it is deliberate and in the changelog. If one table comes out wrong and the rest looks right, `--table-seed` re-rolls just that table. Every other table's seed CSV stays byte-identical, and children of the re-rolled table still diff --git a/model2data/generate/core.py b/model2data/generate/core.py index 5f17997..1830958 100644 --- a/model2data/generate/core.py +++ b/model2data/generate/core.py @@ -248,7 +248,7 @@ def generate_data_from_dbml( # composite-key retry regenerates a temporal column's value into a # frame that already respects created/updated/closed ordering, rather # than one where only the untouched columns do. - df = order_row_times(df, table_def, as_of=as_of) + df = order_row_times(df, table_def, as_of=as_of, time_profile=profile) df = _resolve_self_referencing_fks( df, table_def, diff --git a/model2data/generate/timeline.py b/model2data/generate/timeline.py index 2c709d8..c6b63d2 100644 --- a/model2data/generate/timeline.py +++ b/model2data/generate/timeline.py @@ -197,6 +197,60 @@ def weighted_timestamps(row_count: int, profile: TimeProfile, anchor: date) -> l _TIMESTAMP_GAP_SCALE_SECONDS = 3 * 86400 _DATE_GAP_SCALE_DAYS = 3 +# How many gaps `place_after` draws for a moment inside the window, in the +# column's shape, before it settles for one drawn evenly between the bounds. +_PLACE_TRIES = 20 + +# A weekend day's weight against a weekday's under `business_hours`, as +# `_day_weight` weighs it. +_WEEKEND_WEIGHT = 0.3 + + +def gap(kind: str) -> timedelta: + """How far after what it follows a dependent column lands: 3 days on average.""" + if kind == "timestamp": + return timedelta(seconds=round(random.expovariate(1 / _TIMESTAMP_GAP_SCALE_SECONDS))) + return timedelta(days=max(0, round(random.expovariate(1 / _DATE_GAP_SCALE_DAYS)))) + + +def place_after(lower: datetime, kind: str, upper: datetime, business_hours: bool) -> datetime: + """A moment a random gap after `lower`, in the column's shape, before `upper`. + + The gap decides roughly when; `business_hours` then decides the weekday + and the hour, as it does for a column drawn on its own: a weekend day is + kept 3 times in 10, and a timestamp's time of day is redrawn from + `HOUR_WEIGHTS` on the gap's day (a time before `lower` is drawn again). + A timestamp lands before `upper`, a date on or before it. A draw past + `upper` is drawn again, so a short window does not pile rows up on its + last moment; after `_PLACE_TRIES` the moment is drawn evenly between the + bounds, and a window with no room at all returns `lower`: the column + never lands before what it follows. + """ + + def inside(moment: datetime) -> bool: + return moment < upper if kind == "timestamp" else moment.date() <= upper.date() + + for _ in range(_PLACE_TRIES): + moment = lower + gap(kind) + if business_hours: + if moment.weekday() >= 5 and random.random() >= _WEEKEND_WEIGHT: + continue + if kind == "timestamp": + hour = random.choices(range(24), weights=HOUR_WEIGHTS)[0] + moment = datetime(moment.year, moment.month, moment.day, hour) + timedelta( + seconds=random.randrange(3600) + ) + if moment < lower: + continue + if inside(moment): + return moment + if kind == "timestamp": + span = int((upper - lower).total_seconds()) + return lower + timedelta(seconds=random.randrange(span)) if span > 0 else lower + span = (upper.date() - lower.date()).days + return lower + timedelta(days=random.randint(0, span)) if span > 0 else lower + + # Column-name stems that place a column at a stage in the created -> updated -> # closed chain. A column matches a stage by tokenizing its name on non- # alphanumeric characters and looking for the stem as a contiguous run of @@ -330,6 +384,14 @@ def _format_value(moment: datetime, kind: str) -> Union[str, date]: return moment.date() +def column_business_hours(column: ColumnDef, time_profile: Optional[TimeProfile]) -> bool: + """Whether `column` is drawn in business hours: its own hint, else the run's.""" + note = column.note or {} + if "business_hours" in note: + return bool(note["business_hours"]) + return bool(time_profile and time_profile.business_hours) + + def _build_dependencies( table_def: TableDef, columns_by_name: dict, temporal_columns: list[ColumnDef] ) -> dict[str, set[str]]: @@ -413,15 +475,20 @@ def _topological_order(table_def: TableDef, deps: dict[str, set[str]]) -> list[s return order -def order_row_times(df, table_def: TableDef, as_of: AsOf = None): +def order_row_times( + df, table_def: TableDef, as_of: AsOf = None, time_profile: Optional[TimeProfile] = None +): """Fix up temporal columns so an earlier-stage column never lands later. Runs unconditionally (every profile, not just non-uniform ones): a row updated before it was created is wrong regardless of how the timestamps were drawn. A column with dependencies is set, per row, to the latest of - those dependencies plus a random gap (capped at the window's end), unless - either side is null for that row -- a nullable column that happened to - come back null keeps its null, rather than this pass forcing a value in. + those dependencies plus a random gap, inside the window and in the + column's business-hours shape (see `place_after`; `time_profile` is the + run's, a column's own `business_hours` hint replacing it), unless either + side is null for that row -- a nullable column that happened to come back + null keeps its null, rather than this pass forcing a value in. The run's + growth and seasonality reach the column through the one it follows. Leaves `df` untouched when the table holds no `after` hint and no recognisable created/updated/closed-style name pair, and never modifies a @@ -456,7 +523,7 @@ def order_row_times(df, table_def: TableDef, as_of: AsOf = None): other_kind = _column_kind(columns_by_name[name].data_type) assert other_kind is not None required_kinds[name] = other_kind - scale = _TIMESTAMP_GAP_SCALE_SECONDS if kind == "timestamp" else _DATE_GAP_SCALE_DAYS + business_hours = column_business_hours(columns_by_name[column_name], time_profile) for idx in df.index: if _parse_value(df.at[idx, column_name], kind) is None: @@ -475,14 +542,7 @@ def order_row_times(df, table_def: TableDef, as_of: AsOf = None): continue latest_required = max(earlier) - gap = random.expovariate(1 / scale) - if kind == "timestamp": - moment = latest_required + timedelta(seconds=round(gap)) - else: - moment = latest_required + timedelta(days=max(0, round(gap))) - if moment > cap: - moment = cap - + moment = place_after(latest_required, kind, cap, business_hours) df.at[idx, column_name] = _format_value(moment, kind) return df diff --git a/model2data/generate/when.py b/model2data/generate/when.py index 9d7b4b1..8689caf 100644 --- a/model2data/generate/when.py +++ b/model2data/generate/when.py @@ -45,12 +45,12 @@ from model2data.generate.faker import AsOf, generate_column_values from model2data.generate.options import TimeProfile from model2data.generate.timeline import ( - _DATE_GAP_SCALE_DAYS, - _TIMESTAMP_GAP_SCALE_SECONDS, _build_dependencies, _format_value, _parse_value, _topological_order, + column_business_hours, + place_after, ) from model2data.parse.dbml import ColumnDef, TableDef @@ -123,7 +123,13 @@ def _temporal_kind(column: ColumnDef) -> Optional[str]: class _Order: """What a filled date or timestamp of one table must sit between, row by row.""" - def __init__(self, table_def: TableDef, frame: pd.DataFrame): + def __init__( + self, + table_def: TableDef, + frame: pd.DataFrame, + time_profile: Optional[TimeProfile] = None, + ): + self.time_profile = time_profile self.by_name = {column.name: column for column in table_def.columns} temporal = [ column @@ -154,6 +160,11 @@ def before(self, name: str, row: int) -> Optional[datetime]: found = self._moments(followers, row) return min(found) if found else None + def place(self, name: str, lower: datetime, kind: str, upper: datetime) -> datetime: + """A moment for `name` a random gap after `lower`, in the column's shape.""" + business_hours = column_business_hours(self.by_name[name], self.time_profile) + return place_after(lower, kind, upper, business_hours) + def settle(self, name: str, row: int, cap: datetime) -> None: """Move what follows `name` in this row to no earlier than what it follows, in order. @@ -172,17 +183,10 @@ def settle(self, name: str, row: int, cap: datetime) -> None: lower = self.after(other, row) if value is None or lower is None or value >= lower: continue - self.frame.iat[row, column] = _format_value(min(lower + _gap(kind), cap), kind) + self.frame.iat[row, column] = _format_value(self.place(other, lower, kind, cap), kind) moved.add(other) -def _gap(kind: str) -> timedelta: - """How far after what it follows a filled value lands: the ordering pass's own gap.""" - if kind == "timestamp": - return timedelta(seconds=round(random.expovariate(1 / _TIMESTAMP_GAP_SCALE_SECONDS))) - return timedelta(days=max(0, round(random.expovariate(1 / _DATE_GAP_SCALE_DAYS)))) - - def _draw_like( column: ColumnDef, held: list, @@ -223,7 +227,7 @@ def apply_when( columns = when_columns(table_def) if not columns: return - order = _Order(table_def, frame) + order = _Order(table_def, frame, time_profile) for column in columns: name = column.name match = matching(frame, order.by_name, conditions(column) or {}) @@ -274,7 +278,7 @@ def _moment( span = max(0, int((upper - start).total_seconds())) moment = start + timedelta(seconds=random.randint(0, span)) elif lower is not None: - moment = lower + _gap(kind) + moment = order.place(name, lower, kind, upper) else: held = [ parsed diff --git a/tests/fixtures/determinism.json b/tests/fixtures/determinism.json index 1034608..60a4f53 100644 --- a/tests/fixtures/determinism.json +++ b/tests/fixtures/determinism.json @@ -729,7 +729,7 @@ "dbt_det/seeds/raw/feature_limits.csv": "c991751064bf749aa1277c53eca6ba4e5941566f6ab1b083d3e49d14eb4ffab2", "dbt_det/seeds/raw/invoices.csv": "19489a185cd13149b16f76e62e6b508de437d67648c30f7e6d753bd305ec6268", "dbt_det/seeds/raw/organizations.csv": "bb9019c41083da011f0512d4a5ea4225fa766a8c9d3feaabdfb6713d19a412e0", - "dbt_det/seeds/raw/subscriptions.csv": "de43c0cacdd8589a8c26c5a65b8b3efbc6e12874ad7ed19bf4d4bbcf0880dcf3", + "dbt_det/seeds/raw/subscriptions.csv": "02bb173269e98ca870de3cfc875f23b54630cc11a6162d35d0640ae4193d0296", "dbt_det/seeds/raw/users.csv": "d55263c0ea25908f11d2a300106749af4022e2f05b0a7e452ef171f2e707ea46" }, "table-seed": { diff --git a/tests/test_determinism.py b/tests/test_determinism.py index cbfd207..9cf26f3 100644 --- a/tests/test_determinism.py +++ b/tests/test_determinism.py @@ -1,15 +1,16 @@ -"""The same model, seed and `--as-of` write byte-identical files in every 1.x release. +"""The same model, seed and `--as-of` write byte-identical files on the same version. That is the product's promise: a committed dataset, a diffed CI run and a teaching exercise all lean on it. This module enforces it. Each case runs the real CLI in-process, hashes every file it writes, and compares against the hashes in -`fixtures/determinism.json`, which were recorded from a published release. +`fixtures/determinism.json`. -If a case fails, output changed for the same seed. That is a breaking change and -needs a major version; it is not a snapshot to refresh. Faker or pandas releases -that change the data show up here too (CI runs this file against the newest of both). +If a case fails, output changed for the same seed. That is not a snapshot to +refresh: either the change is unintended (find it), or it is deliberate and the +changelog says which models it affects and why. Faker or pandas releases that +change the data show up here too (CI runs this file against the newest of both). -Re-recording is for adding or deliberately versioned changes only: +Re-recording is for adding cases or deliberate, changelogged changes only: MODEL2DATA_RECORD_DETERMINISM=1 uv run pytest tests/test_determinism.py @@ -123,9 +124,9 @@ def test_same_seed_same_bytes(case_id, tmp_path, recorded): f"DETERMINISM BROKEN in case {case_id!r}: the same model, seed and --as-of no " "longer write the same files.\n" f" changed files: {changed}\n new files: {added}\n missing files: {missing}\n" - "Changing output for the same seed breaks model2data's promise of byte-identical " - "output across 1.x releases and needs a MAJOR version. Do not just re-record the " - "hashes. If the change comes from a Faker or pandas release, pin or work around it." + "Do not just re-record the hashes: re-record only for a deliberate change the " + "CHANGELOG describes (which models, and why). If the change comes from a Faker or " + "pandas release, pin or work around it." ) diff --git a/tests/test_timeline.py b/tests/test_timeline.py index cd915a9..b0f4970 100644 --- a/tests/test_timeline.py +++ b/tests/test_timeline.py @@ -10,13 +10,15 @@ fixed expectation so a future refactor can't silently drift it further. """ +import random from datetime import date, datetime, timedelta +import pandas as pd import pytest from model2data.generate.core import generate_data_from_dbml from model2data.generate.options import TimeProfile -from model2data.generate.timeline import order_row_times +from model2data.generate.timeline import order_row_times, place_after from model2data.parse.dbml import ColumnDef, TableDef ANCHOR = date(2026, 3, 15) @@ -315,6 +317,68 @@ def test_after_hint_is_honoured(): assert datetime.fromisoformat(ordered) <= datetime.fromisoformat(shipped) +def _bookings(scheduled_note: dict, rows: int = 2000, profile=None) -> pd.DataFrame: + tables = { + "bookings": TableDef( + name="bookings", + columns=[ + ColumnDef("id", "bigint", {"pk"}), + ColumnDef("booked_at", "timestamp", {"not null"}), + ColumnDef("scheduled_for", "timestamp", {"not null"}, note=scheduled_note), + ], + ) + } + return generate_data_from_dbml( + tables, [], base_rows=rows, seed=1, as_of=ANCHOR, time_profile=profile + )["bookings"] + + +def _working_share(values) -> float: + moments = [datetime.fromisoformat(v) for v in values] + return sum(1 for t in moments if t.weekday() < 5 and 8 <= t.hour < 18) / len(moments) + + +def test_a_column_with_after_keeps_its_business_hours(): + """The gap decides roughly when; the column's `business_hours` decides the hour.""" + df = _bookings({"after": "booked_at", "business_hours": True}) + + assert _working_share(df["scheduled_for"]) >= 0.65 + assert _working_share(df["booked_at"]) < 0.4 + for booked, scheduled in zip(df["booked_at"], df["scheduled_for"], strict=True): + assert datetime.fromisoformat(booked) <= datetime.fromisoformat(scheduled) + + +def test_the_runs_business_hours_reach_a_column_with_after(): + df = _bookings({"after": "booked_at"}, profile=TimeProfile(business_hours=True)) + + assert _working_share(df["scheduled_for"]) >= 0.65 + + +def test_a_column_with_after_does_not_pile_up_on_as_of_midnight(): + """A gap past the window is drawn again, not set to the window's end.""" + df = _bookings({"after": "booked_at"}) + cap = datetime(ANCHOR.year, ANCHOR.month, ANCHOR.day) + + scheduled = [datetime.fromisoformat(v) for v in df["scheduled_for"]] + assert max(scheduled) < cap + # Two rows may share a second by chance; the old clamp put dozens on one. + assert max(scheduled.count(moment) for moment in scheduled) <= 2 + + +def test_place_after_stays_between_its_bounds(): + random.seed(0) + lower = datetime(2026, 3, 14, 23, 0) + upper = datetime(2026, 3, 15) + for business_hours in (False, True): + moments = [place_after(lower, "timestamp", upper, business_hours) for _ in range(200)] + assert all(lower <= moment < upper for moment in moments) + assert len(set(moments)) > 150 + + dates = [place_after(lower, "date", upper, True) for _ in range(50)] + assert {moment.date() for moment in dates} <= {date(2026, 3, 14), date(2026, 3, 15)} + assert place_after(upper, "timestamp", upper, False) == upper + + def test_start_end_suffix_pair_is_honoured(): tables = { "subscriptions": TableDef(