Feature/configurable db

[adefabian]

Customer‑overridable MeasurementDB: read any source schema with the stock solver

Summary

Impulse's solver expects data in its silver model. Customers whose data is stored
differently have no supported way to plug in their layout: SolverConfig covers renames and
narrow‑vs‑wide, but nothing beyond that. This PR lets a customer select their own MeasurementDB
implementation by name in the report config. It reshapes their tables into the silver model at
read time; the stock DefaultSolver then runs unchanged, and every read path (solve,
incremental detection, ad‑hoc) sees the same data. The change is additive: if nothing is
configured, the built‑in MeasurementDB is used. The one breaking change is stricter parsing of
query_engine (below).

Breaking change

query_engine now rejects unknown keys. A config with a misspelled or leftover key under
query_engine (for example "measurment_db") fails at load time with Extra inputs are not permitted instead of being silently ignored. The deprecated batch_size alias still works.

Changes

• Extension point: new AbstractMeasurementDB (impulse_query_engine/measurement_db_abstract.py)
  with a read-only ws property that raises a TypeError if a subclass skips
  super().__init__(config, ws). The built-in MeasurementDB subclasses it and registers itself.
• Registry: register_measurement_db / resolve_measurement_db
  (impulse_query_engine/measurement_db_registry.py).
• Config: query_engine.measurement_db and query_engine.measurement_db_config, validated when
  the config is loaded; QueryEngine forbids unknown keys (impulse_reporting/config/config_parser.py).
• Report: Report.create_measurement_db builds the selected implementation
  (impulse_reporting/core/report.py).
• Docs: new Extensibility → Custom measurement DB page, query_engine rows on the configuration
  page, regenerated API reference, updated impulse-config skill.

Review feedback addressed

• Optional tables: the docs now say that the solver only uses an optional table, and only calls
  its override, when the matching source entry is set.
• ws: read-only property that raises if super().__init__(config, ws) is skipped, plus the
  matching rule in the docs.
• Misspelled query_engine keys: extra="forbid" on QueryEngine (see Breaking change).
• Decision table: the "query planning / execution" row now points to custom solvers and links
  their docs.

Test Plan

• Unit tests: AbstractMeasurementDB contract and registry (including the ws property),
  extra-table pinning, build_measurement_db_config, parse-time rejection of misspelled
  query_engine keys and of bad measurement_db_config keys, and Report failing with a clear error
  for a DB that skips super().__init__
• Integration tests (custom_measurement_db_report_test.py): the selected DB serves all reads, its
  config reaches the DB, incremental detection reads through it, and a declared extra table is pinned
• Docs site builds without broken links
• Demo notebooks re-run (checked statically: they use no unknown query_engine keys)

Checklist

• Code follows project style guidelines
• Self-review completed
• No new linter warnings introduced

---

Design

Throughout this doc, Acme is an example customer whose raw layer does not match the silver model.

Contents

1. Why this is necessary
2. The concept
3. The Acme example: workflow, data schema and clustering keys
4. Technical details
5. Accepted risks

---

1. Why this is necessary

1.1 Customers don't store data in the silver model

Impulse's query engine assumes silver tables such as container_metrics, channel_metrics,
channels, poi_channels and the EAV *_tags. Real customers may store data in a different way.

Acme's raw layer, for example, keys containers by a string session id and channels by a 3‑part
string signal name, stores chunked statistics instead of one row per channel, keeps wide reference
tables instead of EAV tags, and stores raw points instead of intervals.

Renames and narrow‑vs‑wide shape are already handled by DefaultSolver + SolverConfig. Acme needs
more than that: unions, joins, a grain collapse and an EAV unpivot.

Example: a row filter that must run before the reshape. Acme's metadata and statistics tables
carry a marker column data_type, and only rows with data_type = 3 may be used: they are the one
representative row per session. Without that filter, joining these rows to the recording sessions
fans out and duplicates sessions. Acme's app already applies it before that join.

Stock Impulse cannot express this. SolverConfig.<table>.filters run after the MeasurementDB
has returned its frame (QuerySolver.scoped_container_metrics, query_solver.py:79-80). After a
reshape, that frame is already joined and collapsed: the data_type column is gone and the
duplicates have already been produced. The filter has to be applied to the raw rows, before the
join and the aggregation, which is only possible inside a custom MeasurementDB.

1.2 Performance depends on the customer's clustering keys

Customers' existing pipelines are fast because they read narrowly, filtering on their tables'
clustering keys (for Acme: date, recording session and signal). The stock Impulse read path does not
filter on those keys, so on the same tables it reads far more data. To reach comparable performance,
reads must filter on the customer's clustering keys at the raw table (§3.2).

1.3 There is no extension point today

The only way to support Acme's layout today is to change Impulse itself.
Report.create_measurement_db (report.py:216-241) always builds the built‑in MeasurementDB, and
Report.create_solver (report.py:317) is a closed match over the Solvers enum that only
returns DefaultSolver.

---

2. The concept

2.1 In one picture

The MeasurementDB is the layer between the stored tables and everything that reads them. This PR
makes it replaceable by name: the customer ships a small MeasurementDB subclass (plus a config
class) in their own package, and the report config selects it.

flowchart LR
    RAW["Customer raw tables<br/>(any schema)"] -->|"prefilter on raw<br/>clustering keys"| CDB["Customer MeasurementDB<br/>(reshape → silver)"]
    CDB --> S["Stock DefaultSolver<br/>solve()"]
    CDB --> I["Incremental detection"]
    CDB --> A["Ad‑hoc / exports"]
    CFG["Report config<br/>query_engine.measurement_db = 'AcmeMeasurementDB'"] -.->|"resolved by name"| REG["measurement_db registry"]
    REG -.-> CDB

Loading

Every consumer reads through the same customer DB, so they all see identical silver‑shaped data.
In the report config, only two optional keys are new:

"query_engine": {
  "measurement_db": "AcmeMeasurementDB",
  "measurement_db_config": { "recording_session_table": "acme.raw.recording_session" }
}

2.2 What it offers customers with different schemas

• Bring any source schema. Session‑based ids, compound signal names, chunked statistics, wide
  reference tables, raw points: all are reshaped when the tables are read.
• Stock solver. Customers don't touch query planning, so solver improvements in Impulse apply
  unchanged. Histograms, events, statistics and POI all run on the stock solver.
• Correct incremental reporting. Detection reads the same reshaped container_metrics as the
  solve, so the silver↔gold join aligns (§4.4).
• Consistent snapshots. Extra source tables are pinned to the run's Delta snapshot together with
  the silver tables (§4.5).
• File pruning on the customer's clustering keys. The prefilter runs on the raw columns, so Delta
  skips files outside the selected dates, sessions and signals (§3.2).
• Safe and isolated. Name‑only registration; customer code lives in its own package, with no fork
  of Impulse core.
• Backward compatible. Existing configs keep using the built‑in MeasurementDB.

2.3 Which extension point to use

Your data differs from silver by…	Use
Column names, narrow vs. wide layout	DefaultSolver + SolverConfig (no code)
Different tables, unions, joins, grain changes, EAV unpivots, pre‑reshape row filters	Custom MeasurementDB (this PR)
Query planning/execution behaviour config can't express	A custom solver: usable via query.solve(solver=…); reports only accept the built-in solvers (see the "Custom solvers" docs)

2.4 Alternatives considered

The obvious alternative is to leave Impulse unchanged and have the customer materialize the
silver model upstream: a scheduled job that writes container_metrics, channel_metrics,
channels, poi_channels and the tag tables, or SQL views over the raw layer.

We want customers to be able to adopt Impulse smoothly, including those who don't want to build
and run another set of tables and a pipeline just to feed it. A custom MeasurementDB gives them the
flexibility to define their own reshape in code, on top of the data they already have:

	Materialized silver (job)	SQL views over raw	Custom MeasurementDB (this PR)
New tables to maintain	Yes, a full silver copy	No	No
New pipeline to run and monitor	Yes (schedule, backfills, failures)	No	No
Freshness	Lags behind raw until the next job run	Live	Live
Clustering‑key pruning on the raw tables	Only on the new copy's own layout	Lost once filters sit on top of joins and aggregations inside the view	Kept: the prefilter runs on the raw columns before the reshape (§3.2)
Snapshot pinning	Built in (silver tables are Delta)	Not possible (views are never pinned)	Built in for declared extra tables (§4.5)
Where the reshape logic lives	Customer's pipeline	SQL in the catalog	Customer's package, versioned and unit‑testable
Impulse changes	None	None	Small and additive (§4.1)

Materializing silver upstream remains fully supported: a customer who prefers it keeps using the
built‑in MeasurementDB and needs none of this.

A pluggable solver that reshapes inside its read stages is the other alternative. It is the
wrong layer, because the solver is not the only reader of silver data: incremental detection and
changed‑entity scoping read container_metrics outside solve(). The details are in §4.4.

---

3. The Acme example: workflow, data schema and clustering keys

3.1 Acme's raw layer vs. the silver model

Acme raw layer

erDiagram
    SESSION ||--o{ SIGNAL            : "session_id = recording_session_id"
    SESSION ||--o{ SIGNAL_STATISTICS : "session_id = recording_session_id"
    SESSION ||--o{ POI               : "session_id = recording_session_id"
    SIGNAL }o--|| SIGNAL_METADATA    : "signal_name"
    SIGNAL_STATISTICS }o--o| REF_TEST : "test_id"
    REF_TEST }o--|| REF_VEHICLE      : "vehicle_id"
    REF_VEHICLE }o--|| REF_FLEET     : "fleet_id"

    SIGNAL {
        string recording_session_id FK
        string signal_name
        double value_double
        string value
        long timestamp_abs
        date dt
    }
    SIGNAL_STATISTICS {
        string recording_session_id FK
        string signal_name
        int count
        float min
        float max
        float mean
        long start_ts
        long end_ts
        int test_id
        date dt
    }
    POI {
        string recording_session_id FK
        string poi_type
        string value
        long timestamp_abs
        date dt
    }
    SESSION {
        string session_id PK
        timestamp start_datetime
        timestamp end_datetime
    }
    REF_TEST {
        int test_id PK
        int vehicle_id FK
        string test_name
    }
    REF_VEHICLE {
        int vehicle_id PK
        int fleet_id FK
        string vin
    }
    REF_FLEET {
        int fleet_id PK
        string fleet_name
    }
    SIGNAL_METADATA {
        string signal_name PK
        string unit
    }

Loading

Impulse silver model (what the solver requires), shown with Impulse's reference id types
(long container_id, int channel_id). Acme's ids are strings; see the id contract in §4.3.

erDiagram
    CONTAINER_METRICS ||--o{ CONTAINER_TAGS  : "container_id"
    CONTAINER_METRICS ||--o{ CHANNEL_METRICS : "container_id"
    CONTAINER_METRICS ||--o{ POI_CHANNELS    : "container_id"
    CHANNEL_METRICS ||--o{ CHANNEL_TAGS      : "container_id, channel_id"
    CHANNEL_METRICS ||--o{ CHANNELS          : "container_id, channel_id"

    CONTAINER_METRICS {
        long container_id PK
        timestamp start_dt
        timestamp stop_dt
        int num_channels
        long odo_start
        long odo_stop
        array dtc_error_codes
    }
    CHANNEL_METRICS {
        long container_id FK
        int channel_id
        int sample_count
        float min
        float max
        float mean
    }
    CHANNELS {
        long container_id FK
        int channel_id FK
        long tstart
        long tend
        double value
    }
    POI_CHANNELS {
        long container_id FK
        int channel_id
        long timestamp
        double value_double
        string value_string
    }
    CONTAINER_TAGS {
        long container_id FK
        string key
        string value
    }
    CHANNEL_TAGS {
        long container_id FK
        int channel_id FK
        string key
        string value
    }

Loading

3.2 Clustering keys and the prefilter

Acme's raw tables are liquid‑clustered on dt, recording_session_id, signal_name: the
partition date, the recording‑session identity and the physical signal identity. Acme's existing
pipeline is fast because it reads narrowly on exactly those keys; if Impulse reads the same tables
without that scope, it full‑scans them. So the custom DB pushes a clustering‑key prefilter at the
raw read:

Prefilter field (measurement_db_config)	Predicate on the raw column	Applied to
recording_session_id: list[str]	recording_session_id IN (…)	all reads
signal_name: list[str]	signal_name IN (…)	channel‑grain reads only (would distort container aggregates)
start_dt / stop_dt	dt >= … / dt <= …, no cast (a single day sets both to the same date)	all reads

Which clustering keys each read pushes

Silver read	Raw table	dt	recording_session_id	signal_name
channels()	signal	✅	✅	✅
channel_metrics()	signal_statistics	✅	✅	✅
channel_tags()	signal_metadata	✅	✅	✅
container_metrics(), container_tags()	signal_statistics	✅	✅	— (would change container aggregates such as num_channels, odo_*)
poi_channels()	poi	✅	✅	— (POI rows are not per signal)

Rule: apply the prefilter on the raw physical columns, before the reshape, and compare dt
to a same‑type literal without a cast. Renaming alone does not stop pruning (Catalyst pushes
filters through aliases). What does stop it is a CAST/function on the clustering column, a type
mismatch, or an optimization barrier between filter and scan (groupBy, join, checkpoint). A
reshape contains all of those, so a filter applied after it cannot prune the source read.

Known gap (customer package, owned by the customer team): the 5‑minute variant tables
(signal_statistics_5m, signal_metadata_5m) are unioned in after the prefilter, so today they
are read without any clustering‑key filter. The fix is to apply the same prefilter to them before
the union (see §4.6).

3.3 Typical Acme workflow

An Acme analyst builds a report in the web app in three selection steps. Each step becomes one
clustering‑key prefilter, so the report run only touches the files for that date range, those
recording sessions and those signals.

flowchart TD
    D["1. Pick date range<br/>start / stop"] --> C["2. Narrow candidate recording sessions<br/>fleet · vehicle (VIN) · geo area · vehicle attributes ·<br/>POI · datalogger · distance / odometer"]
    C --> P["Pick recording sessions<br/>(picker shows run id · test id · date)"]
    P --> S["3. Pick logical signals<br/>e.g. ENGINE_SPEED, VEHICLE_SPEED, ODOMETER"]
    S --> R["Resolved to physical signal names<br/>ENGINE_SPEED:12:A1, …"]
    R --> A["4. Define events, histograms, statistics<br/>on the logical signals"]
    A --> G["App generates the report config<br/>measurement_db = AcmeMeasurementDB + prefilter"]
    G --> RUN["5. Report run"]
    RUN --> PIN["pin_versions<br/>(silver + extra tables)"]
    PIN --> RD["Prefiltered raw reads<br/>dt · recording_session_id · signal_name"]
    RD --> RS["Reshape → silver"]
    RS --> SOL["Stock DefaultSolver → gold"]

Loading

Step	Analyst action	Written to measurement_db_config	Prunes on clustering key
1	Date range (start_ts / stop_ts)	start_dt / stop_dt (date part); for a single day both are the same date	dt
2	Filter criteria narrow a candidate list, then the analyst picks sessions	recording_session_id: [...] (the picked ids)	recording_session_id
3	Logical signals, resolved to their physical 3‑part names	signal_name: [...]	signal_name
4	Events and aggregations on logical signals	report definition (not a prefilter)	—

The filter criteria in step 2 (fleet, VIN, geo, attributes, POI, odometer…) only build the
candidate list in the app. The report itself filters on the picked recording session ids,
which is why they can be pushed down as a clustering key.

3.4 How AcmeMeasurementDB maps Acme to silver

Silver read	From	Reshape
channels()	signal	drop raw string value; value_double→value, timestamp_abs→timestamp (µs), ids
channel_metrics()	signal_statistics (+ 5‑min variant)	collapse chunks per (container, channel); fold POI channel‑metrics stub
container_metrics()	signal_statistics (+ 5‑min variant)	collapse per container (num_channels, odo_*, bounds via session); fold dtc_error_codes[]
container_tags()	signal_statistics + ref_*	join test→vehicle→fleet; unpivot test_name/vin/fleet_name to EAV
channel_tags()	signal_metadata (+ 5‑min variant)	unpivot unit + channel_name to EAV
poi_channels()	poi	poi_type→channel_id, value→value_string
channel_mapping()	signal_metadata	alias = first component of signal_name; if several physical names share an alias, the lexically first wins

A second customer with a different layout writes its own subclass and config class, registers it
under its own name, and reuses the same framework mechanism.

3.5 Acme's report config

{
  "source": {
    "container_metrics_table": "acme.raw.signal_statistics",
    "container_tags_table": "acme.raw.signal_statistics",
    "channel_metrics_table": "acme.raw.signal_statistics",
    "channel_tags_table": "acme.raw.signal_metadata",
    "channel_mapping_table": "acme.raw.signal_metadata",
    "channels_uri": "acme.raw.signal",
    "poi_channels_uri": "acme.raw.poi"
  },
  "query_engine": {
    "solver": "DefaultSolver",
    "data_type": "RAW",
    "measurement_db": "AcmeMeasurementDB",
    "measurement_db_config": {
      "recording_session_table": "acme.raw.recording_session",
      "start_dt": "2026-04-01",
      "stop_dt": "2026-04-30",
      "recording_session_id": ["<session-id-1>"],
      "column_name_mapping": {
        "channels": { "recording_session_id": "container_id", "signal_name": "channel_id" }
      },
      "…": "further extra tables and prefilter keys"
    }
  }
}

• source keeps its shape: one table per silver role. With a custom DB these entries point at
  the customer's raw tables, which the base read methods (super().<table>(spark)) read and pin.
• query_engine.measurement_db_config holds everything source cannot express: extra tables,
  column mappings, prefilter keys, customer‑specific flags.
• query_engine.solver_config keeps solver behaviour only (e.g. channel_mapping.join_keys).
  Column renames belong in measurement_db_config, because the DB already returns internal names.

---

4. Technical details

4.1 Framework changes (Impulse side, additive)

1. An abstract contract. AbstractMeasurementDB(ABC)
   (impulse_query_engine/measurement_db_abstract.py, re‑exported from measurement_db.py).
   It implements __init__(config, ws) (including the telemetry verify_workspace_client), a
   read-only ws property that raises a TypeError if a subclass skipped
   super().__init__(config, ws), and the .query property, and declares as abstract container_metrics, channel_metrics, channels,
   container_tags, channel_tags, poi_channels, has_poi_channels, channel_mapping,
   unit_conversion, channel_uri and pin_versions (called by Report at run start, §4.5).
   MeasurementDB subclasses it; Report.get_db / create_measurement_db are typed with it.

2. A name registry. register_measurement_db(name, config_cls) and resolve_measurement_db(name)
   (impulse_query_engine/measurement_db_registry.py), storing a frozen
   MeasurementDatabaseRegistration(db_cls, config_cls). The decorator raises TypeError unless the
   class subclasses AbstractMeasurementDB; config_cls is a MeasurementDBConfig subclass. The
   built‑in registers itself as ("MeasurementDB", MeasurementDBConfig). Re‑registering a name
   replaces the previous entry.

3. A config selector. query_engine.measurement_db: str = "MeasurementDB" and
   query_engine.measurement_db_config: dict | None in config_parser.py, next to solver /
   solver_config. QueryEngine now forbids unknown keys (extra="forbid"), so a misspelled key
   fails when the config is loaded instead of silently selecting the built-in DB (see Breaking change).

   Key	Type	Default	Purpose
   query_engine.measurement_db	str	"MeasurementDB"	Registered name of the MeasurementDB implementation to use
   query_engine.measurement_db_config	dict	null	Keyword arguments for that implementation's config class, merged with the source tables

4. Factory resolution. build_measurement_db_config(source_dict, query_engine) in
   config_parser.py resolves the registration and builds one config object,
   config_cls(**source_dict, **measurement_db_config, table_locations="unity_catalog").
   Report.create_measurement_db keeps its existing source_dict preparation and then calls it;
   the result is db_cls(config, ws). With no selector this is exactly the previous
   MeasurementDBConfig(**source_dict) plus the built‑in MeasurementDB.

No change to the pinning logic in MeasurementDB.

4.2 How the customer's config reaches the DB

flowchart LR
    S["source<br/>(silver-role tables)"] --> B["build_measurement_db_config<br/>config_cls(**source, **measurement_db_config)"]
    J["query_engine.measurement_db_config<br/>(customer settings)"] --> B
    N["query_engine.measurement_db<br/>'AcmeMeasurementDB'"] -->|"resolve_measurement_db"| R["registration<br/>db_cls · config_cls = AcmeDBConfig"]
    R --> B
    B --> C["AcmeDBConfig instance"]
    C --> F["AcmeMeasurementDB(config, ws)"]

Loading

1. Parse time: a validator on ImpulseConfig calls build_measurement_db_config once as a
   check, so these fail as a Pydantic ValidationError when the config is loaded:
   • an unregistered name (the message lists the registered names);
   • a missing required key (a keyword‑only argument without a default);
   • an unknown or misspelled key, in measurement_db_config or anywhere under query_engine;
   • a key present in both sections;
   • any check the customer's __init__ performs.
2. Build time: Report.create_measurement_db calls build_measurement_db_config again and
   returns db_cls(config, ws).

The config class is therefore constructed twice, so its __init__ must not have side effects.

4.3 What a customer implements

class AcmeDBConfig(MeasurementDBConfig):
    """Silver-role tables (from `source`) + Acme's extras (from `measurement_db_config`)."""

    def __init__(
        self,
        *,
        recording_session_table: str,      # required: no default, so a missing value is reported
        signal_statistics_5m_table=None,
        ref_tables=(),
        recording_session_id=(),           # clustering-key prefilter
        signal_name=(),
        start_dt=None,
        stop_dt=None,
        column_name_mapping=None,          # per-table physical -> internal renames
        **base_kwargs,                     # container_metrics_table, channels_uri, ..., table_locations
    ):
        super().__init__(**base_kwargs)    # keeps pinned_versions, debug_tables, table_locations
        self.recording_session_table = recording_session_table
        self.signal_statistics_5m_table = signal_statistics_5m_table
        self.ref_tables = list(ref_tables)
        self.recording_session_id = list(recording_session_id)
        self.signal_name = list(signal_name)
        self.start_dt, self.stop_dt = start_dt, stop_dt
        self.column_name_mapping = column_name_mapping or {}

    def configured_table_uris(self) -> list[str]:
        # Extra tables are pinned by the inherited MeasurementDB.pin_versions.
        extras = [self.recording_session_table, self.signal_statistics_5m_table, *self.ref_tables]
        return super().configured_table_uris() + [u for u in extras if u]


@register_measurement_db("AcmeMeasurementDB", AcmeDBConfig)
class AcmeMeasurementDB(MeasurementDB):
    def __init__(self, config: AcmeDBConfig, ws):                # Impulse calls db_cls(config, ws)
        super().__init__(config, ws)
        self.transformer = AcmeTransformer(config, read_table=self._read_table)  # pinned reads only

    def container_metrics(self, spark):
        raw = super().container_metrics(spark)                  # raw read, pinned
        return self.transformer.container_metrics(spark, raw)   # prefilter + reshape → silver
    # ... one override per silver table; no pin_versions override needed

Read‑method pipeline. Every override follows the same steps:
super().<table>(spark) (raw read) → prefilter → optional union/join of extra tables →
column_name_mapping (physical → internal) → per‑table reshape → silver‑shaped frame with internal ids.

Rules for the customer:

• Subclass MeasurementDBConfig and call super().__init__(**base_kwargs). The base
  initializer sets table_locations, debug_tables and pinned_versions, which pinning and
  _read_table rely on.
• Declare every extra table in configured_table_uris() so it is pinned (§4.5).
• Import the package before the report config is parsed, including in tools that only
  generate or edit configs. An unregistered name fails with
  Unknown measurement DB 'AcmeMeasurementDB'; registered: ['MeasurementDB']. Import the package that registers it before parsing the report config.
• Make required settings required. Only keyword‑only arguments without a default are reported
  as missing; a field with a default like =None is silently accepted when left out.
• Keep the config's __init__ free of side effects; it runs once for the check and once for the
  build.
• Keep the constructor signature __init__(self, config, ws); Impulse builds the DB as
  db_cls(config, ws).
• If you override __init__, call super().__init__(config, ws) and don't assign self.ws
  yourself. Reading db.ws (which Report and db.query do) raises a TypeError otherwise.
• Set the source entry for every optional table you override (container_tags_table,
  channel_tags_table, channel_mapping_table, unit_conversion_table, poi_channels_uri).
  Otherwise the solver doesn't use the table and your override is never called; for channel tags,
  unit conversion and POI this happens without an error.
• Tests and debug mode: MeasurementDBConfig.for_debug(...) returns the base class, so a custom
  config provides its own debug constructor, and debug_tables must include the extra tables.
• In Python, the same keys go on QueryEngine:
  QueryEngine(solver="DefaultSolver", measurement_db="AcmeMeasurementDB", measurement_db_config={...}).

Supported for subclasses. A custom DB and its config may rely on these members, and changing
them is a breaking change for implementers:

• MeasurementDBConfig.__init__ keyword arguments, configured_table_uris() and table_locations
• MeasurementDB._read_table(spark, uri)
• the base read methods (super().<table>(spark)) as the raw, pinned read of each configured table

4.4 Why the reshape belongs in the MeasurementDB, not in the solver

Layer	Job	Who reads it
MeasurementDB (read layer)	Where and how silver data is read	Every consumer: solve(), incremental container detection, changed‑entity scoping, ad‑hoc analysis, exports
QuerySolver (engine)	How a TSAL query is planned and executed	Its stages run inside solve() and the container pipeline, and each stage reads the DB itself

Incremental processing breaks with a solver‑side reshape. Incremental reporting reads
container_metrics outside solve() and joins it against gold. The call chain:

1. Report starts incremental mode with pre_filtered_containers_df = self._detect_upserted_containers()
   (report.py:1068).
2. _detect_upserted_containers (report.py:1428-1464) gets its inputs from
   _container_detection_args (report.py:1507-1527), which reads
   silver_containers = self._filtered_container_metrics() (report.py:1524).
3. _filtered_container_metrics (report.py:1419-1426) calls solver.filter_container_tags and
   solver.filter_container_metrics, which end in QuerySolver.scoped_container_metrics
   (query_solver.py:46-81) and the raw read query.db.container_metrics(spark) (query_solver.py:73).
4. ContainerUpsertDetector.detect_upserted_containers (container_detector.py:48-100) left‑anti‑joins
   and inner‑joins that frame against gold measurement_dimension on container_id and compares the
   last_modified freshness columns (container_detector.py:137-207).

A reshape that lives in the solver breaks this chain in three places:

• The incremental subset re‑enters the solver already processed. The detected containers are
  passed back into the solve as pre_filtered_containers_df. scoped_container_metrics then skips
  the DB read and uses the subset as‑is (query_solver.py:70-71), but still re‑applies the column
  mapping (query_solver.py:74-76). That is harmless for renames (withColumnRenamed on a missing
  column is a no‑op). It is not harmless for a real reshape: a solver that collapses chunk‑grain
  statistics into per‑container rows would have to tell raw input (line 73) from already‑reshaped
  input (line 71) inside the same method, or it reshapes twice. With the reshape in the DB, both
  branches carry the same silver shape and the mapping is a no‑op.
• Every non‑solve reader has to remember to go through the solver. The comment at
  report.py:1520-1523 records the bug that already happened once: "Reading the raw table here
  would break configs that remap the physical container‑id column (Incremental container detection ignores column_name_mapping and assumes a physical container_id column #99)." Incremental container detection ignores column_name_mapping and assumes a physical container_id column #99 fixed detection by
  routing it through the solver pipeline. _changed_scope (report.py:1480-1505) reuses the same
  silver_containers frame and joins it against gold ids (report.py:1500-1505). Each new reader of
  container_metrics repeats this risk. With the reshape in the DB, any reader, including a plain
  db.container_metrics(spark), gets the reshaped frame by construction.
• The id mapping must be identical on both sides of the silver↔gold join. Acme's container_id
  is its session id, renamed from the physical recording_session_id. Gold measurement_dimension
  stores the id the solve produced, and detection compares against it. If the rename is applied in
  more than one code path and they drift, detection sees a different column or value than the solve
  wrote. Every container then looks new on every run (silent full reprocessing), or updated
  containers are missed. That is exactly the Incremental container detection ignores column_name_mapping and assumes a physical container_id column #99 bug. A DB‑side reshape applies the mapping in one place.

The solver reads the DB in many places, not one hook. DefaultSolver stages read the DB directly
at default_solver.py:316 (container_tags), :503 (channel_tags), :556 and :652
(channel_metrics), :1142 (channels) and :1209 (poi_channels), plus query_solver.py:73 and
:195. A solver‑side reshape would have to override every stage that touches one of these tables and
follow upstream changes to them. A DB override implements the read methods behind a stable, abstract
contract (§4.1).

Fewer plan workarounds. Reshaping inside solver stages means re‑reading the same raw tables within
one plan and joining across the copies. That trips
MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION and needs .distinct() /
localCheckpoint workarounds. The read layer is the natural single owner of those reads.

4.5 Snapshot consistency (Delta version pinning)

At the start of each run Report calls self.db.pin_versions(self.spark) (report.py:1039), so all
lazy stages read one Delta snapshot (issue #87). How it works today:

• MeasurementDB.pin_versions pins the tables in MeasurementDBConfig.configured_table_uris()
  (measurement_db.py:85-100, :140-170).
• _read_table(spark, uri) looks up config.pinned_versions[uri] and adds versionAsOf
  (measurement_db.py:172-181).
• Views and non‑Delta tables are never pinned; they warn and read the latest version
  (measurement_db.py:112-138).

A reshape reads extra tables that are not in that list (sessions, reference tables, 5‑minute
statistics), so they are not pinned automatically.

Decision: the implementer declares its extra tables; Impulse's inherited pin_versions pins them.
The pinning logic itself stays unchanged; the only addition is declaring pin_versions on the
abstract class.

• Impulse: AbstractMeasurementDB declares pin_versions(spark) as an abstract method.
  Report already calls it, so this only makes the existing de‑facto contract explicit.
  Otherwise a registered implementation that doesn't inherit MeasurementDB would pass registration
  and then fail with AttributeError at report.py:1039.
• Implementer (a custom DB that reads tables beyond the configured silver ones):
  1. Declare the extra tables on the config subclass: override configured_table_uris() to
     return super().configured_table_uris() plus the extra URIs. The inherited pin_versions
     iterates exactly that list, so the extras are pinned together with the silver tables, including
     the base handling of debug mode, views and non‑Delta tables. configured_table_uris() is only
     used by pin_versions (measurement_db.py:166), so extending it has no other effect.
  2. Read every extra table through self._read_table(spark, uri), never spark.read.table. This
     includes transformers: they receive the reader or the frame from the DB and do not read tables
     themselves.
  3. Use the same URI string to declare and to read; the pin lookup is by exact name.
  4. In debug mode (tests), include the extra tables in debug_tables, since _read_table looks
     them up by name there.

A DB that implements AbstractMeasurementDB without inheriting MeasurementDB has no inherited
pin_versions and must implement pinning itself.

4.6 Scope of this PR

• Framework (additive; one breaking change, see Breaking change):
  • new measurement_db_abstract.py: AbstractMeasurementDB, with pin_versions on the contract
    and a read-only ws property;
  • new measurement_db_registry.py: register_measurement_db, resolve_measurement_db,
    MeasurementDatabaseRegistration;
  • measurement_db.py: MeasurementDB subclasses the ABC and registers itself; its __init__ and
    query moved to the ABC unchanged;
  • config_parser.py: query_engine.measurement_db / measurement_db_config,
    build_measurement_db_config, the parse‑time validator on ImpulseConfig, and
    extra="forbid" on QueryEngine;
  • report.py: registry resolution in Report.create_measurement_db;
  • user docs: query_engine rows in docs/impulse/docs/config/configuration.md, a new
    Extensibility → Custom measurement DB page
    (docs/impulse/docs/extensibility/custom_measurement_db.md), the regenerated API reference, and the
    impulse-config skill.
• Minimum Impulse version for implementers: pinning extra tables via configured_table_uris()
  needs a databricks-impulse that has version pinning (issue Pin silver input tables to fixed Delta versions for a report run #87). Older releases don't pin at all.
• Follow‑up (Impulse): optionally add a Report(db=…) instance‑injection escape hatch.

Out of scope: owned by the customer team (separate repo). AcmeMeasurementDB and its
transformers live in the customer package and are not part of this PR. To follow the implementer
rules in §4.3 and §4.5, that team:

• moves its reshape settings into a MeasurementDBConfig subclass and declares its extra tables
  (sessions, reference tables, 5‑minute statistics) in configured_table_uris();
• routes the six bare spark.read.table calls in its transformers through the injected _read_table;
• prefilters the 5‑minute variant tables before the union (§3.2 known gap);
• switches its report configs and source adapter to "solver": "DefaultSolver" +
  "measurement_db": "AcmeMeasurementDB".

This PR defines the contract they build against; Acme appears in this doc as the reference use case.

---

5. Accepted risks

This PR provides the option to plug in a custom MeasurementDB. The customer who implements one
takes responsibility for it:

• Correct output is the implementer's responsibility. Impulse does not validate the frames a
  custom DB returns. The implementer must provide the columns, types, row grain and time unit the
  solver expects; otherwise the report fails at solve time or silently computes wrong results. The
  reference is the built‑in MeasurementDB plus the silver schemas in
  impulse_query_engine/schema.py (for data_type: RAW, channels() returns point rows as in
  CHANNELS_SCHEMA_WITHOUT_RLE).
• Declaring extra tables is the implementer's responsibility (§4.5). A table missing from
  configured_table_uris(), or read outside _read_table, is not pinned and reads the latest version.
• View‑backed sources are never pinned. Impulse skips views when pinning (measurement_db.py:125-131).
• The reshape runs at read time, on every report run and every read path (§2.4). The
  clustering‑key prefilter keeps that bounded (§3.2).
• Parsing a report config needs the customer package. Name and key validation runs when the
  config is parsed, so every tool that parses a config (including ones that only generate or edit
  configs) must import the package that registers the DB first.
• Base members become supported API for subclasses: MeasurementDBConfig.__init__ and
  configured_table_uris(), and MeasurementDB._read_table (§4.3). Changing them is a breaking
  change for implementers.

This pull request and its description were written by Isaac.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.29%. Comparing base (aec178e) to head (5f91906).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #109      +/-   ##
==========================================
+ Coverage   90.21%   90.29%   +0.08%     
==========================================
  Files          66       68       +2     
  Lines        6009     6060      +51     
  Branches      757      758       +1     
==========================================
+ Hits         5421     5472      +51     
  Misses        466      466              
  Partials      122      122              

Flag	Coverage Δ
query_engine	86.80% <100.00%> (+0.13%)	⬆️
reporting	94.91% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/impulse_query_engine/measurement_db.py	87.61% <100.00%> (-0.78%)	⬇️
...rc/impulse_query_engine/measurement_db_abstract.py	100.00% <100.00%> (ø)
...rc/impulse_query_engine/measurement_db_registry.py	100.00% <100.00%> (ø)
src/impulse_reporting/config/config_parser.py	97.21% <100.00%> (+0.18%)	⬆️
src/impulse_reporting/core/report.py	94.04% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tombonfert]

Thanks, the registry and config-class split reads cleanly. Three things I'd like addressed before merge:

1. Docs: optional tables are enabled by their source slot (custom_measurement_db.md)

The solver only reads container_tags, channel_tags, channel_mapping and unit_conversion when the matching source.*_table is set (e.g. default_solver.py:486, :669). If the slot is empty, an override of that method is never called. For channel_tags and unit_conversion this happens without an error: the solver falls back to wide mode or skips unit conversion. The current wording ("Override one method per silver table you need to reshape") suggests the override alone is enough. Could you document that the source slot turns the optional table on and the override reshapes what that slot reads?

2. Telemetry relies on custom DBs calling super().__init__ (measurement_db_abstract.py:25)

Report and QueryBuilder take their workspace client from db.ws (report.py:133, query_builder.py:38). If a custom DB overrides __init__ without calling super().__init__, or sets self.ws itself, the product tag and the connectivity check are lost without any error. A read-only property turns this into a clear error on both the Report and the standalone path:

def __init__(self, config: "MeasurementDBConfig", ws: WorkspaceClient):
    self.config = config
    self._ws = verify_workspace_client(ws, "databricks-impulse", __version__)

@property
def ws(self) -> WorkspaceClient:
    try:
        return self._ws
    except AttributeError:
        raise TypeError(
            f"{type(self).__name__}.__init__ must call super().__init__(config, ws)"
        ) from None

Please also add "call super().__init__(config, ws)" to the Rules section of the docs.

3. Misspelled query_engine keys are silently ignored (config_parser.py:362)

QueryEngine doesn't forbid extra keys, so a typo like "measurment_db" falls back to the built-in DB. The report then runs on the raw tables and writes wrong gold data. The docs promise a parse-time failure (custom_measurement_db.md:98). Setting extra="forbid" on QueryEngine fixes this. Since configs with stray keys will start failing, it's worth a changelog note.

[tombonfert]

LGTM!