Repository navigation
feat(clickhouse): declare the cloud data clusters - #95026
Merged
Merged
Conversation
Contributor
🤖 CI report
|
orian
marked this pull request as ready for review
September 4, 2026 10:50
orian
force-pushed
the
pawel/chore/hcl-data-cloud-layers
branch
from
September 4, 2026 10:50
3493d4c to
895d797
Compare
Contributor
Prompt To Fix All With AI### Issue 1
posthog/clickhouse/hcl/roles/data/dev/tables.hcl:1301-1303
**Backfill throttles are discarded**
This full `sharded_raw_sessions_v3` override replaces the common settings map with only `index_granularity`. The equivalent overrides in prod-us and prod-eu do the same. This drops `parts_to_delay_insert`, `max_delay_to_insert`, and `parts_to_throw_insert`, which migration 0178 and the canonical sessions-v3 builder explicitly install to make backfills self-regulating. As a result, generated or reconciled cloud schemas can recreate this table without delaying or rejecting inserts when it accumulates too many parts.
### Issue 2
posthog/clickhouse/hcl/exclude.hcl:68-70
**Comment records change history**
This comment preserves a dated operational change and a pending dump refresh. The repository requires code comments to explain enduring rationale rather than record change history. Keep the lasting reason for these exclusions here and move transient status to the PR or tracking system. This repository requirement must be satisfied before merging.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: 3493d4c | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+1301
to
+1303
| settings = { | ||
| index_granularity = "8192" | ||
| } |
Contributor
There was a problem hiding this comment.
This full sharded_raw_sessions_v3 override replaces the common settings map with only index_granularity. The equivalent overrides in prod-us and prod-eu do the same. This drops parts_to_delay_insert, max_delay_to_insert, and parts_to_throw_insert, which migration 0178 and the canonical sessions-v3 builder explicitly install to make backfills self-regulating. As a result, generated or reconciled cloud schemas can recreate this table without delaying or rejecting inserts when it accumulates too many parts.
Knowledge Base Used: Ingestion storage and event data
Prompt To Fix With AI
This is a comment left during a code review.
Path: posthog/clickhouse/hcl/roles/data/dev/tables.hcl
Line: 1301-1303
Comment:
**Backfill throttles are discarded**
This full `sharded_raw_sessions_v3` override replaces the common settings map with only `index_granularity`. The equivalent overrides in prod-us and prod-eu do the same. This drops `parts_to_delay_insert`, `max_delay_to_insert`, and `parts_to_throw_insert`, which migration 0178 and the canonical sessions-v3 builder explicitly install to make backfills self-regulating. As a result, generated or reconciled cloud schemas can recreate this table without delaying or rejecting inserts when it accumulates too many parts.
**Knowledge Base Used:** [Ingestion storage and event data](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog/-/docs/event-ingestion-storage.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
|
😎 Stack merged successfully - details. |
The data role was modeled only for the local node. The three cloud data clusters had no env block, so nothing in either repo described what they run except posthog-cloud-infra's overrides, and reconcile could not see them at all. They get dev, prod-us and prod-eu env blocks composing a cloud layer, a prod group layer, and per-env layers. Composition reproduces the public shape of a node dump exactly: zero operations on all three environments. The materialized columns stay out, as they do for the sessions replica. So do the credentialed dictionaries: the server redacts their password at introspection, and hclexp refuses to write the placeholder to a cluster, so those keep their declaration in posthog-cloud-infra. Objects that both the local node and the cloud clusters run move to a new roles/data/common layer. It sits outside the layers posthog-cloud-infra vendors, so its composition is untouched until that repo adopts these layers; hoisting into roles/data/shared instead would have collided with its overrides today. Every local golden is byte-identical, so the move changes nothing that exists. Cross-cluster validation resolves for these envs for the first time, because the posthog cluster stops being absent. What it found is live state rather than a modelling mistake, so nineteen proxies are skipped by name in check.sh, each one a drift row rather than a declaration to change.
Composition now reproduces the public shape of a node dump exactly on dev, prod-us and prod-eu: zero operations on all three. Three things were wrong. Objects only the local node and dev run had been hoisted into a layer every data stack composes, so the prod clusters declared tables they do not have. A handful of objects sat in the cloud layer despite being absent from one cloud env. And four views were declared with the local node's query, which differs from the cloud one. Each now sits in the narrowest layer its consumers share, and an environment whose shape differs restates it rather than declaring it again. The views are the reason the pin moves: override was rejected on view and dictionary until chschema#232, so a per-cluster view could not be expressed at all. Ten more proxies join the cross-cluster skip list. Their remote is genuinely absent from the prod data clusters, which is the sessions-family relocation still in progress, not something these declarations can fix. The duplicates baseline is untouched at two, every local golden is byte-identical, and posthog-cloud-infra still composes.
…e drift Rebasing onto master crossed three changes that touch the same objects. Billing usage storage moved to the aux cluster. The local node now reads billing_usage_records through aux while a cloud data node holds its own, so the two stop sharing a declaration, and sharded_billing_usage_records leaves the data role entirely. Both prod clusters dropped the person and group property columns from the flag evaluation tables, and dev dropped the $session_id bloom filter from sharded_events. Both are adopted as patches so the declarations still describe what the nodes run. The scratch tables excluded on 2026-09-04 are gone from the dumps, so their exclude entries go with them. The patterns for recurring job artifacts stay. Composition still reproduces each node dump exactly, and every local golden is byte-identical to master's.
orian
force-pushed
the
pawel/chore/hcl-data-cloud-layers
branch
from
September 7, 2026 10:12
895d797 to
bc5582c
Compare
The delta from the previous pin is the locate command and its docs, so nothing that composes or canonicalizes changed: every golden and generated sql file is byte-identical at the new pin.
rorylshanks
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Nobody could say what the three cloud data clusters run. The data role was modeled for the local node only, so reconcile skipped those nodes entirely and every drift on them was invisible.
hclexp plan. The cloud data nodes have never been checked.flag_evaluationsshows what that costs: posthog declares it, inroles/data/local, which no cloud stack composes. Every cloud node runs the table and no layer they compose describes it.Changes
sharded_eventsis declared as its stable 62-64 migration-managed columns; the 179-317 the materialization engine adds are layered on there.roles/data/commonfor what the local node and the cloud both run,roles/data/cloudfor all three cloud envs,roles/data/prodfor the two prod clusters, then per-env. An environment whose shape genuinely differs restates it withoverride = truerather than declaring it twice.roles/data/commonsits outside the layers posthog-cloud-infra vendors, so that repo's composition is untouched until it adopts these layers. Hoisting intoroles/data/sharedinstead collided with its overrides, which the compose gate caught.posthogcluster stops being absent. 29 proxies are skipped by name incheck.sh, each a drift row rather than a declaration to change.$session_idbloom filter fromsharded_events.Note
The pin moves to
sha-13aa3d0. Four views resolve to a different query per cluster, andoverridewas rejected onviewanddictionaryuntil PostHog/chschema#232, so that could not be expressed at all. The tree does not load on the older pin.Zoo paths and engine blocks become public with this, which decision 13 already accepted. No hostnames, credentials, or customer identifiers are in the diff.
How did you test this code?
check.shpasses with the pinned image: five envs validate, 31 compositions match their goldens, duplicates match the baseline of 2, sql is fresh.hclexp diffof each composed data role against a node dump from PostHog/clickhouse-schema reports zero operations for dev, prod-us and prod-eu.check-cloud.shconfirms posthog-cloud-infra still composes in all three envs with its duplicates baseline unchanged.check-live.shnever ran.check.shin CI.Automatic notifications
Docs update
manifest.hcldocuments why the data role's materialized columns are absent.exclude.hclgains the per-customer, job-artifact and transient patterns the cloud data clusters need, each with its reason.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Written with Claude Code. Skills invoked:
superpowers:executing-plans,/stacking-prs,/writing-pr-descriptions.Originally stacked on #93456, which restored the ops env layers and fixed the kafka engine model. That merged, so this now targets master directly and was rebased onto it.
Two decisions shaped the result. The first is where an object lives: the initial pass hoisted anything the local node and the cloud shared into
roles/data/shared, which broke posthog-cloud-infra's composition and made the prod clusters declare tables they do not have. Layer placement is now driven by which nodes actually carry the object, measured from the dumps. The second is what stays private: the split is decided by whether every node carries the same shape, which is measurable, rather than by whether a creator can be found in the source, which turned out to be unreliable both ways.overrideon views came from a gap found while doing this and filed as PostHog/chschema#231.Public artifact: everything here derives from this repository, its two sibling checkouts, and the node dumps. Scratch tables carrying a person's name, a per-customer table, and dated job artifacts were excluded rather than declared; the owning team dropped them from the clusters separately.