Repository navigation
hoglake: attribute every DDL request; pyhoglake 1.4.0; server image 1.4.0 - #153
Merged
Merged
Conversation
….4.0 hoglake 1.4.0 (#372) records author and message on DDL: required on drop, truncate, table replacement and retention reduction, optional on create and alter. millpond only creates and alters, so nothing was refused, but its catalog, namespace and table creation, spec declaration and schema evolution were unattributed (hoglake #366). Every DDL call now carries the sink's existing author, millpond/<table>/<ordinal>, the one its data commits already use, and a deterministic message of the form "<why>: <what>; ENV=value", naming the topic and the MILLPOND_* / HOGLAKE_* settings that produced the definition: first flush of a topic creating the table with N columns from the batch and its partition/sort settings; the two spec-recovery paths saying which spec was missing; schema evolution naming the columns added or the promotion made. Nothing in it changes between attempts, so a retried declaration sends the same text. AlreadyExists and CommitConflict handling is untouched. pyhoglake[fast-upload]>=1.4.0 for the keyword arguments, with the exclude-newer carve-out moved past its upload. The hoglake test stack's DEFAULT_IMAGE is the 1.4.0 server index digest; the suite's own drops now carry author and message, which that server requires, and the snapshot helpers distinguish data commits from the DDL snapshots that now share the sink's author. Unit 1424 passed; integration 103; hoglake-integration 50 and hoglake-e2e 7 against the 1.4.0 image; e2e 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes PostHog/hoglake#366.
hoglake 1.4.0 (PostHog/hoglake#372) records
authorandmessageon DDL — required on drop, truncate, table replacement and retention reduction, optional on create and alter. millpond only creates and alters, so nothing is refused by a 1.4.0 server; this attributes what it does.What
millpond/<table>/<ordinal>, the identity its data commits already carry, now on every DDL request.<why>: <what>; ENV=value…, deterministic (no clock), naming the topic and the settings that produced the definition. Examples:first flush of events_json: create table analytics.events with 5 columns from the batch; HOGLAKE_PARTITION_BY=identity(team_id), day(_inserted_at); MILLPOND_SORT_BY=team_id,uuid; MILLPOND_TYPED_COLUMNS=uuid:uuid,team_id:bigintrecovery for events_json (table exists without its sort order): declare sort order; MILLPOND_SORT_BY=uuidschema evolution from events_json: add columns c1 (string), c2 (long)/… promote column count from int to longcreate_catalog,create_namespace,create_table, spec declaration (both recovery paths), batched and per-columnadd_column,promote_column.AlreadyExistsError/CommitConflictErrorhandling unchanged; a retried declaration sends identical text.pyhoglake[fast-upload]>=1.4.0, with theexclude-newercarve-out moved past its upload (2026-10-21T18:54Z).DEFAULT_IMAGE→ the 1.4.0 server index digest. The suite's own table drops carryauthor/message(that server requires them); the snapshot helpers now separate data commits from DDL snapshots, which share the sink's author.Tests
TestDdlAttribution(17) — exact wire payload per call site,unsetrendering, only-sent ops in the message, both recovery paths, identical text on retry, per-column fallback, every DDL call on a cold bootstrap attributed, retries afterAlreadyExistsErrorandCommitConflictError.TestDdlAttributionagainst the real server records the request bodies httpx sent and assertsauthor/messageon catalog/namespace/table creation and all three alters, then the persisted attribution viaGET /v1/catalogs/{c}/snapshots. Catalog creation precedes snapshot history, so it is checked on the wire only.just test1424 passed;test-integration103;test-hoglake-integration50 andtest-hoglake-e2e7 against the 1.4.0 image;test-e2e4; lint and format clean.