From 6578e4361967e3efaba7e947e617d9098c910504 Mon Sep 17 00:00:00 2001 From: Fabian Hirschmann Date: Sun, 5 Jul 2026 13:50:48 +0200 Subject: [PATCH] agenticdata: optional labs 7-10 (managed dataform, ai.forecast, sharing, day-2 ops), not wired into TUTORIAL.md --- content/agenticdata/labs/10_day2_ops.md | 199 ++++++++++++++++++ .../agenticdata/labs/7_managed_dataform.md | 189 +++++++++++++++++ content/agenticdata/labs/8_forecast.md | 177 ++++++++++++++++ content/agenticdata/labs/9_sharing.md | 131 ++++++++++++ .../src/optional/day2_ops/AGENTS.md | 98 +++++++++ .../src/optional/day2_ops/POSTMORTEM.md | 80 +++++++ .../src/optional/day2_ops/chaos.sh | 48 +++++ .../src/optional/day2_ops/dashboard.json | 134 ++++++++++++ .../src/optional/day2_ops/prompts.md | 51 +++++ .../optional/forecast/challenge_anomalies.sql | 28 +++ .../optional/forecast/challenge_backtest.sql | 31 +++ .../forecast/fct_revenue_forecast.sqlx | 35 +++ .../src/optional/forecast/prompts.md | 28 +++ .../src/optional/managed_dataform/prompts.md | 28 +++ .../managed_dataform/push_to_dataform.sh | 136 ++++++++++++ .../optional/sharing/listing_description.md | 41 ++++ .../src/optional/sharing/prompts.md | 15 ++ 17 files changed, 1449 insertions(+) create mode 100644 content/agenticdata/labs/10_day2_ops.md create mode 100644 content/agenticdata/labs/7_managed_dataform.md create mode 100644 content/agenticdata/labs/8_forecast.md create mode 100644 content/agenticdata/labs/9_sharing.md create mode 100644 content/agenticdata/src/optional/day2_ops/AGENTS.md create mode 100644 content/agenticdata/src/optional/day2_ops/POSTMORTEM.md create mode 100755 content/agenticdata/src/optional/day2_ops/chaos.sh create mode 100644 content/agenticdata/src/optional/day2_ops/dashboard.json create mode 100644 content/agenticdata/src/optional/day2_ops/prompts.md create mode 100644 content/agenticdata/src/optional/forecast/challenge_anomalies.sql create mode 100644 content/agenticdata/src/optional/forecast/challenge_backtest.sql create mode 100644 content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx create mode 100644 content/agenticdata/src/optional/forecast/prompts.md create mode 100644 content/agenticdata/src/optional/managed_dataform/prompts.md create mode 100755 content/agenticdata/src/optional/managed_dataform/push_to_dataform.sh create mode 100644 content/agenticdata/src/optional/sharing/listing_description.md create mode 100644 content/agenticdata/src/optional/sharing/prompts.md diff --git a/content/agenticdata/labs/10_day2_ops.md b/content/agenticdata/labs/10_day2_ops.md new file mode 100644 index 0000000..15dbcfa --- /dev/null +++ b/content/agenticdata/labs/10_day2_ops.md @@ -0,0 +1,199 @@ +## Lab 10: Day-2 Ops — Dashboards, Alerts, and the 3 A.M. Page + + +{{ author('Fabian Hirschmann', 'https://linkedin.com/in/fhirschmann') }} + + + + +You built a pipeline. Tonight, at 3 a.m., it will break — and nobody will know until the Monday revenue meeting stares at Friday's numbers. In this lab you become Cymbal's on-call engineer: you build a **Cloud Monitoring dashboard as code** (authored by agy), create the **alert** Datastream doesn't ship, and then — deliberately — **break your own replication** with a chaos script you are not allowed to read first. You'll watch the incident unfold on your dashboard, diagnose it with agy as your read-only co-responder, fix it with one line of SQL, and prove that not a single row was lost. + +This lab expects **Labs 1–3 to be complete**: `cymbal-cdc-stream` replicating into `cymbal_bronze`, and the medallion built. It also needs **terminal 2 (the `bk-tunnel` IAP tunnel) and terminal 3 (the simulator) still running** — an incident is only interesting while live traffic is flowing. If the simulator stopped, restart it from Lab 2's *Watch it flow* section before continuing. + +(The commands below contain your generated database password, rendered into the tutorial. If you see empty quotes instead, run the `bk-start` reload step from the end of Lab 1.) + +### About day-2 operations + +Day 1 is building the pipeline; **day 2** is everything after: watching it, alerting on it, and fixing it at 3 a.m. Cloud Monitoring gives you the three pieces. **Metrics** are the numeric time series every managed service emits — Datastream reports how far it lags behind the source (**data freshness**, in seconds), Cloud SQL reports connections and CPU, BigQuery's Storage Write API reports ingested rows. **Dashboards** arrange those series into one pane of glass, and because a dashboard is just a JSON document deployed via API, it can live in git and be code-reviewed like everything else you built today. **Alerting policies** watch a metric against a condition and open an **incident** when it's violated — and here is today's teaching point: Datastream emits excellent metrics but ships **not a single alert out of the box**. If bronze freezes tonight, nobody's phone buzzes unless *you* make it so. That's this lab. + +Learn more: +- [Monitor a Datastream stream](https://docs.cloud.google.com/datastream/docs/monitor-a-stream) +- [Dashboards via the API](https://docs.cloud.google.com/monitoring/dashboards/api-dashboard) +- [Alerting overview](https://docs.cloud.google.com/monitoring/alerts) + +### Prepare the observability surface + +First, bring terminal 1 back to the repo root — Lab 3 parked it in the Dataform folder, and every repository path in this lab (and each `agy` session you'll start) expects to run from `~/bootkon`: + +```bash +cd ~/bootkon +``` + +Enable the two services (idempotent — on most projects they are already on) and grant yourself the Monitoring and Logging roles. Sandbox users are typically project owners already; the explicit grants keep this lab portable: + +```bash +gcloud services enable monitoring.googleapis.com logging.googleapis.com +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member="user:{{ GCP_USERNAME }}" --role="roles/monitoring.editor" --condition=None >/dev/null +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member="user:{{ GCP_USERNAME }}" --role="roles/logging.viewer" --condition=None >/dev/null +``` + +Then a pre-flight check — this must print `RUNNING` (if not, revisit Lab 2 before injecting chaos into a stream that's already down): + +```bash +gcloud datastream streams describe cymbal-cdc-stream --location={{ REGION }} --format='value(state)' +``` + +### Author the dashboard with agy + +A dashboard definition is a config artifact — authoring work, so it goes to agy; you review, deploy, and verify. The full brief — which six charts, the exact metric types, the API shape — lives in AGENTS.md, right next to the artifact. Read it: it pins the metric names character for character, because monitoring metric types are exactly the kind of detail an agent will confidently misremember. + +The repository ships a finished dashboard as reference, so clear the stage first for the authentic experience: + +```bash +rm content/agenticdata/src/optional/day2_ops/dashboard.json +``` + +Start `agy` from `~/bootkon` and hand over the brief: + +``` +/goal Read content/agenticdata/src/optional/day2_ops/AGENTS.md and create content/agenticdata/src/optional/day2_ops/dashboard.json exactly as it specifies: valid Cloud Monitoring Dashboard API JSON, six widgets in the given order, the exact metric types from the brief, and the 120-second threshold line on the freshness chart. No comments, no trailing commas. +``` + +**Review what agy wrote**: open dashboard.json and check the six `metric.type` strings against the brief's table. If agy went off-script, git has your back: + +```bash +git -C ~/bootkon restore content/agenticdata/src/optional/day2_ops/dashboard.json +``` + +### Deploy and admire + +Exit agy — deploying is operating, so it's your job, and it's one command: + +```bash +gcloud monitoring dashboards create \ + --config-from-file=content/agenticdata/src/optional/day2_ops/dashboard.json +``` + +❗ If this fails with `INVALID_ARGUMENT` and a field path (e.g. an unknown field or a malformed filter string), that's agy's JSON, not your command — paste the full error into agy and let it repair the file, or restore the reference above and re-run. + +Open [Monitoring → Dashboards](https://console.cloud.google.com/monitoring/dashboards) and click Cymbal Day-2 Ops. Set the time range to **1 hour** and read your pipeline's vital signs, live: + +- **Datastream — data freshness** hugs the bottom of the chart, safely under the red 120 s SLO line: Datastream is keeping up with the WAL. +- **Datastream — throughput** shows a steady trickle — your simulator commits a change every second or three, and each one becomes events here. +- **Cloud SQL — connections/CPU** are calm and boring. Boring is the goal. +- **BigQuery Storage Write API — ingested rows** is the destination-side echo of the same trickle. (This chart can lag a few minutes behind — metric ingestion has latency; remember that number, it matters later.) + +One pane of glass, three services, and every line on it is explained by a system *you* built. Take ten seconds to feel good about this. Then let's talk about what's missing. + +### The missing alert + +A dashboard only works if someone is looking at it. At 3 a.m., nobody is — that's what **alerting policies** are for, and Datastream doesn't create any for you. The one that matters most for CDC is data freshness: *"page me when bronze is more than two minutes behind production."* + +Datastream's console has a shortcut for exactly this: + +1. Open [Datastream](https://console.cloud.google.com/datastream/streams) and click cymbal-cdc-stream. +2. On the stream's page, scroll until the **Data freshness** graph appears (on the details view — or on the **Monitoring** tab, depending on console version) and click the **Create alerting policy** link next to it. It drops you into the Monitoring policy editor with the freshness metric preselected. (Link missing or moved? Same destination by hand: [Monitoring → Alerting](https://console.cloud.google.com/monitoring/alerting) → **Create policy** → select the metric **Datastream Stream → Stream freshness**.) +3. Configure the trigger: condition type **Threshold**, **Above threshold**, value `120` (the metric's unit is seconds — the same SLO line agy drew on your dashboard). Leave the rolling window at its default. +4. On the notifications step, **skip the notification channels** — email delivery is not part of today's game (and unverified on sandbox accounts). The [Alerting](https://console.cloud.google.com/monitoring/alerting) incidents list *is* your pager for this lab; you'll keep it open. +5. Name the policy `cymbal-freshness-slo` and create it. + +A word on the threshold: freshness sits near 0 when the stream is healthy and the simulator writes continuously, so 120 s is far above any normal jitter — this alert cannot false-positive on you. It can only mean one thing: replication has stopped while the source keeps moving. (Production tip: pair a threshold alert with a **metric absence** condition on the same metric — a stream that stops *reporting* entirely should page too.) Time to arrange exactly that: replication stopped, while the source keeps moving. + +### Break it + +Time to earn the pager. The repository ships a chaos script. The rule of this exercise: **run it now, read it later** — real incidents don't send you their source code in advance. You have a dashboard; that's more than most on-calls get. + +```bash +content/agenticdata/src/optional/day2_ops/chaos.sh +``` + +Note the timestamp it prints — your incident timeline starts there. The script will taunt you. It has earned the right. + +### First signals + +Go back to your **Cymbal Day-2 Ops** dashboard and watch the incident develop over the next couple of minutes (charts refresh on their own, or use the refresh button): + +- **Data freshness** stops hugging zero and starts climbing — a straight, relentless ramp toward the red SLO line and past it. +- **Throughput** decays to zero. No events are arriving from the source. +- And the tell that makes this a *good* mystery: **Cloud SQL connections and CPU don't move.** Check terminal 3 — the simulator is still cheerfully inserting orders. Postgres is fine. Production is fine. Only the replica is freezing. + +Now the producer's view: on the [Datastream](https://console.cloud.google.com/datastream/streams) page, open cymbal-cdc-stream — the stream no longer looks healthy, and as the lockout persists its status flips from **Running** to **Failed**, with the stream's error details naming the reason. + +What about your pager? Be honest about physics: metric ingestion plus alert evaluation means the incident appears on the [Alerting](https://console.cloud.google.com/monitoring/alerting) page **3–8 minutes after the chaos** — monitoring is fast, not instant. Don't sit there refreshing; the diagnosis below is designed to fill exactly that window. (When it fires, the incident shows up at the top of the Alerting page under **Incidents** — check back after the next section.) + +### Diagnose with your co-responder + +You know *that* it's broken. An on-call needs *why*. This is investigation, not authoring — so agy joins as a **read-only co-responder**: it proposes the commands, but every single one needs your approval before it runs. That's the incident-response pattern worth taking home: the human authorizes, the agent investigates. Back in terminal 1, start `agy` (from `~/bootkon`) and paste the symptom: + +``` +Our BigQuery bronze dataset stopped receiving new rows a few minutes ago, and data freshness on the Datastream stream cymbal-cdc-stream (location us-central1) keeps climbing. The source Postgres itself is healthy and still taking writes. Investigate the Datastream side with read-only commands only (gcloud datastream streams describe, gcloud logging read) and tell me your diagnosis. Ask me before each command, and do not change anything. +``` + +Approve each command as it comes and read the output together with agy. If agy meanders, these two are the heart of the matter — run them yourself: + +```bash +gcloud datastream streams describe cymbal-cdc-stream --location={{ REGION }} \ + --format='yaml(state, errors)' +``` + +```bash +gcloud logging read 'resource.type="datastream.googleapis.com/Stream" severity>=WARNING' \ + --freshness=15m --limit=5 --format='yaml(timestamp, severity, jsonPayload)' +``` + +Somewhere in that output sits the confession, in classic Postgres phrasing: `role "datastream_user" is not permitted to log in`. The replication user — the identity Datastream has used since Lab 2 — can't connect. Production traffic (user `postgres`) was never touched; only the replication lane was closed. + +Diagnosis in hand, you've earned the right to read the crime scene. Open chaos.sh — yes, *now* — and let agy narrate: + +``` +Read content/agenticdata/src/optional/day2_ops/chaos.sh and explain, statement by statement, how it broke replication -- and why the simulator kept writing happily the whole time. +``` + +Two statements: `ALTER ROLE datastream_user NOLOGIN` bars new connections, and `pg_terminate_backend` killed the existing one. Note what the script deliberately did *not* touch: the publication `cymbal_pub` and the slot `cymbal_slot`. Keep that in mind for the recovery — it's about to pay off. + +### Fix it like it's 3:07 a.m. + +Leave agy open — it witnessed the whole incident, and the challenge will put that memory to work. Open a **new terminal tab** (`+`) instead: the fix is one line, and — operating surface — you type it, through the tunnel, as the superuser: + +```bash +PGPASSWORD="{{ BK_DB_PASSWORD }}" psql -h localhost -p 5432 -U postgres -d cymbal \ + -c "ALTER ROLE datastream_user LOGIN;" +``` + +❗ Don't panic when the stream doesn't recover the very second you run this — Datastream retries failed connections **with backoff**, so it may take a few minutes to notice the door is open again. Watch the [stream page](https://console.cloud.google.com/datastream/streams): the status returns to **Running** on its own — a `Failed` stream resumes automatically once the cause is fixed, no restart needed ([stream lifecycle](https://docs.cloud.google.com/datastream/docs/stream-states-and-actions)). On your dashboard, freshness tips over its peak and slides back toward zero. + +Now the question that separates an outage from an annoyance: **did we lose data?** During the lockout the simulator kept writing — did those rows make it? Ask both ends of the pipe. The source: + +```bash +PGPASSWORD="{{ BK_DB_PASSWORD }}" psql -h localhost -p 5432 -U postgres -d cymbal \ + -c "SELECT MAX(order_id) FROM cymbal.orders;" +``` + +And the replica, in [BigQuery](https://console.cloud.google.com/bigquery) (the editor, not table preview): + +```sql +SELECT MAX(order_id) AS latest_order, COUNT(*) AS total_orders +FROM `{{ PROJECT_ID }}.cymbal_bronze.cymbal_orders` +``` + +Within a couple of minutes of the stream turning healthy, the two `MAX(order_id)` values match (give or take the row the simulator inserted while you were switching windows). Every order placed *during* the outage is present — nothing skipped, nothing replayed. This is the Lab 2 lesson cashing its cheque: the **replication slot** `cymbal_slot` kept retaining WAL while Datastream was locked out, and the moment the connection came back, Datastream resumed reading from exactly where it stopped. The outage delayed the data; it never lost it. + +### Watch the incident close + +One loose end: your pager. Back on [Monitoring → Alerting](https://console.cloud.google.com/monitoring/alerting), find the `cymbal-freshness-slo` incident that opened mid-chaos. You don't have to do anything — once freshness has been back under 120 s for the policy's window, Monitoring **resolves the incident automatically** (like the firing, this takes a few minutes after the metric recovers; the incident's timeline shows both edges). Open the incident and read it as your future 3 a.m. self: when it started, how long it lasted, which metric and threshold. Trust, but verify — and now, *monitor*. + +### Challenge: run the incident review + +**\[TASK\]** Take up to 10 minutes — pick at least one: + +1. **The postmortem.** Every real incident ends with a blameless write-up, and yours had an unusually well-documented perpetrator. In the *same* agy session (it witnessed the whole thing), have agy write a blameless postmortem — summary, impact, timeline with your actual times, root cause, detection, resolution, action items — to `POSTMORTEM.md` in the lab's folder, overwriting the shipped example (prompt in the lab folder's own prompts.md). Then compare against the reference with `git -C ~/bootkon diff content/agenticdata/src/optional/day2_ops/POSTMORTEM.md` — did agy catch that the dashboard beat the pager? +2. **Design your own chaos.** Propose the next drill to your table — but *discuss reversibility before anyone runs anything*. The safe list is short: today's script only flipped a role attribute. Dropping `cymbal_pub` or `cymbal_slot` is **off the table** — kill the slot and Datastream loses its position in the WAL; that's the `Failed permanently` state, and the road back is a re-backfill, not a one-liner. A genuinely reversible (and sneaky) one: `gcloud compute instances stop cymbal-jump --zone={{ REGION }}-a` — then explain why freshness now stays at *zero* while it's the *simulator* that dies. Which plane did you just break? (Afterwards: `start` the VM; `bk-tunnel` reconnects on its own, then restart the simulator in terminal 3.) +3. **Page on the logs, not just the lag.** The error log line you found above appeared minutes before the freshness alert fired. In [Logs Explorer](https://console.cloud.google.com/logs), query `resource.type="datastream.googleapis.com/Stream" severity>=ERROR`, then use **Create alert** to turn it into a [log-based alert](https://docs.cloud.google.com/logging/docs/alerting/log-based-alerts) — and discuss: when do you page on symptoms (freshness) versus causes (error logs)? + +Note: If you are stuck and cannot figure out how to proceed after a few minutes, ask your team captain. + +### Success + +🎉 Phenomenal{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! You gave Cymbal's pipeline what every production system deserves: a dashboard someone can read at 3 a.m., the alert the platform didn't ship, and an on-call — you — who diagnosed a dead replica with an AI co-responder, fixed it with one line, and proved zero data loss. The pipeline broke, and it didn't matter. That's operations. 📟✅ diff --git a/content/agenticdata/labs/7_managed_dataform.md b/content/agenticdata/labs/7_managed_dataform.md new file mode 100644 index 0000000..69d4c7c --- /dev/null +++ b/content/agenticdata/labs/7_managed_dataform.md @@ -0,0 +1,189 @@ +## Lab 7: From Laptop to Schedule — the Medallion on Managed Dataform + + +{{ author('Fabian Hirschmann', 'https://linkedin.com/in/fhirschmann') }} + + + + +Lab 3 ended with a promise: *"In production you would run the same project on a schedule in managed Dataform."* This lab cashes it. You push the exact project agy wrote — not a rewrite, the very same SQLX files — into a **managed Dataform repository**, walk its interactive dependency graph, and put it on a 15-minute schedule that runs as a service account instead of you. When you leave this lab, the medallion refreshes itself. + +Before you start: this lab assumes **Labs 1–3 are complete** — the Dataform project in `content/agenticdata/src/dataform` compiles and has been run at least once — and that **terminal 2 (the IAP tunnel) and terminal 3 (the data simulator)** from Lab 2 are still running. The simulator matters here: live writes are what make a schedule worth having. + +### About managed Dataform + +Managed Dataform is the hosted side of the tool you used in Lab 3 — same Dataform core, same SQLX, same `workflow_settings.yaml`, but Google runs it. A **repository** is a Google-hosted git repository (or a connected external one); **development workspaces** are personal editing branches with the compile loop built into the browser; a **release configuration** compiles the code from a git branch into an immutable **compilation result**; and a **workflow configuration** executes compilation results on a cron schedule, as a service account. The division of labor for Cymbal: the CLI stays the developer loop where agy authors and you review, and managed Dataform becomes the *operator* — because a production pipeline should not depend on anyone's laptop, terminal, or working hours. + +Learn more: +- [Create a Dataform repository](https://docs.cloud.google.com/dataform/docs/create-repository) +- [Release configurations](https://docs.cloud.google.com/dataform/docs/release-configurations) +- [Workflow configurations](https://docs.cloud.google.com/dataform/docs/workflow-configurations) +- [The Dataform REST API](https://docs.cloud.google.com/dataform/reference/rest) + +### Set up the execution identity + +Who runs the pipeline at 3 a.m.? Not you — and that is the whole point of this section. Scheduled Dataform executions involve two identities: the Google-managed **Dataform service agent** (it compiles your code and orchestrates runs), and a **custom service account** that the actual BigQuery jobs run as — every repository must have one; the console won't even offer the service agent for that job. You already own a perfect candidate: `dataquality-service-account` from Lab 1's bootstrap, which holds BigQuery read and job roles. It only lacks write access. + +IAM comes *first* in this lab for the same reason the Datastream service agent came first in `bk-bootstrap`: grants take a few minutes to propagate, and doing them now means they're live by the time you execute. Enable the service and materialize the service agent: + +```bash +gcloud services enable dataform.googleapis.com +gcloud beta services identity create --service=dataform.googleapis.com --project={{ PROJECT_ID }} +``` + +Let yourself administer Dataform (sandbox owners have this implicitly; the explicit grant keeps the lab portable): + +```bash +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member=user:{{ GCP_USERNAME }} --role=roles/dataform.admin --condition=None +``` + +Upgrade the execution account from reader to writer — it will be rebuilding silver and gold: + +```bash +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member=serviceAccount:dataquality-service-account@{{ PROJECT_ID }}.iam.gserviceaccount.com \ + --role=roles/bigquery.dataEditor --condition=None +``` + +And let the Dataform service agent *act as* that account — it executes your workflows by minting short-lived tokens for it: + +```bash +for role in roles/iam.serviceAccountTokenCreator roles/iam.serviceAccountUser; do + gcloud iam service-accounts add-iam-policy-binding \ + dataquality-service-account@{{ PROJECT_ID }}.iam.gserviceaccount.com \ + --member=serviceAccount:service-{{ PROJECT_NUMBER }}@gcp-sa-dataform.iam.gserviceaccount.com \ + --role=$role +done +``` + +That impersonation chain — service agent → token → execution account → BigQuery job — is exactly how your pipeline will run without your credentials anywhere in the loop. ([Control access with IAM](https://docs.cloud.google.com/dataform/docs/access-control)) + +### Create the repository + +Now give the project a home. Open [Dataform](https://console.cloud.google.com/bigquery/dataform) in the console (it lives inside BigQuery) and click Create repository: + +1. Repository ID: `cymbal-dataform` +2. Region: `{{ REGION }}` +3. Service account: select `dataquality-service-account@{{ PROJECT_ID }}.iam.gserviceaccount.com` — note that the default Dataform service agent is not on offer: a repository *must* execute as a custom service account, which is precisely the identity separation you just wired up. +4. Leave the remaining defaults and click **Create**. + +### Create a development workspace + +Open cymbal-dataform and click Create development workspace. Workspace ID: `cymbal-dev`, then **Create**. + +Open the workspace — it's empty and offers to initialize itself with boilerplate. **Don't click Initialize workspace**: your project already exists, agy wrote it in Lab 3, and it arrives through the API in a minute. + +### Push the project through the API + +Here is a fact worth a coffee-break story: managed Dataform has **no gcloud commands**. The CLI from Lab 3 compiles and runs projects but cannot manage repositories — those have exactly two surfaces: the console and the REST API. Every button you clicked above is a REST call underneath. So we upload the Lab 3 project the way any CI system would: with the API. The repository ships a script for it — run it (takes a few seconds): + +```bash +cd ~/bootkon +content/agenticdata/src/optional/managed_dataform/push_to_dataform.sh +``` + +(It defaults to `cymbal-dataform` / `cymbal-dev`; if you named things differently, pass your names as two arguments.) + +You should see one `writeFile` line per project file, a `commit`, and a `push`. Now read what you just ran — open push_to_dataform.sh. It is a crash course in the Dataform API, six calls end to end: + +1. `GET` the workspace — fail fast if it doesn't exist. +2. `:makeDirectory` — recreate the folder tree. +3. `:writeFile` — one call per file, contents base64-encoded (JSON can't carry raw bytes). +4. `:commit` — one git commit inside the workspace, author and message included. +5. `:push` — the workspace branch onto the repository's default branch, `main`. +6. `:installNpmPackages` — best effort, and usually a no-op here: with Dataform core 3, `workflow_settings.yaml` replaces `package.json`, so there is nothing to install until a project adds dependency packages. + +Two details deserve your attention. `gcloud auth print-access-token` turns your logged-in identity into the bearer token every call carries — the same token gcloud uses internally. And the script deliberately does **not** upload `.df-credentials.json`: that file points the *CLI* at your local credentials, managed Dataform brings its own identity, and credentials never belong in a repository. Trust, but verify: refresh the `cymbal-dev` workspace in the console — `workflow_settings.yaml` and agy's `definitions/` are all there, already committed and pushed. + +### Admire the compiled graph + +The workspace compiles your project automatically. Click the Compiled graph tab: there it is — the medallion as an interactive DAG. Six bronze declarations on the left, five silver staging models, three gold marts, and every assertion agy wrote (the reference solution compiles to 26 nodes, 12 of them assertions — one more of each if you built Lab 3's challenge assertion; agy's count varies). This graph is what the `${ref(...)}` calls in agy's SQLX have been encoding all along — in Lab 3 you saw it as lineage *after* running; here you see it *before*, computed from code alone. + +❗ If the tab shows a compilation error mentioning `Failed to resolve workflow_settings.yaml` or missing packages, open `workflow_settings.yaml` inside the workspace and click **Install packages** — then the graph appears. + +Click `fct_daily_revenue`: the metadata pane lists its dependencies and dependents, and the query pane shows the compiled SQL with every `${ref(...)}` resolved to a real table name. Try filtering the graph by tag `gold`. + +### Compile a release + +A workspace is for humans; production runs from git. A **release configuration** tells Dataform which branch to compile and how. On the repository page, open the Releases & scheduling tab, and under Release configurations click **Create**: + +1. Release ID: `cymbal-release` +2. Git commitish: `main` (the default — exactly where your push landed) +3. Leave the frequency and the compilation overrides empty: your `workflow_settings.yaml` already carries project and datasets, and today you compile by hand. + +Create it, open cymbal-release, and click New compilation. After a few seconds a **compilation result** appears in the Manual/API compilation results table. + +So what did you just build? A compilation result is a frozen snapshot of your *SQL* — the code as it stood on `main` at that moment. The *data* is not frozen: re-executing the same result re-runs that SQL against whatever is in bronze then. New data needs no new compilation; only new code does. In production you'd give the release configuration a schedule (say, daily) so merged changes compile themselves. + +### Put it on a schedule + +Now the piece that replaces your keyboard: under Workflow configurations click **Create**: + +1. Configuration ID: `cymbal-15min` +2. Release configuration: `cymbal-release` +3. Authentication: **Execute with selected service account** — pick `dataquality-service-account` if it isn't already preselected from the repository. +4. Schedule frequency: `*/15 * * * *` — every quarter-hour (leave the timezone at UTC; quarter-hours are quarter-hours everywhere). +5. Under workflow actions, choose **Selection of tags** and pick `silver` and `gold` — the same slices you ran by hand in Lab 3, assertions included. +6. Click **Create**. + +❗ The schedule fires at the next quarter-hour boundary — which may well land after this lab ends. Don't wait for it; the next section triggers the run right now, and the ticks keep coming all event long either way. + +While the clock runs, ask your co-engineer to cement the concept. In your agy session (terminal 1): + +``` +Explain the managed Dataform model like I run data platforms for a living: what exactly is frozen in a compilation result, why does re-executing the same compilation result still pick up new bronze rows, and when would I schedule release compilations more often than daily? +``` + +### Trigger the first run + +The scheduler is armed, but a demo shouldn't wait for a clock. Trigger the workflow configuration yourself — with the *same API call the scheduler makes* at :00, :15, :30 and :45: + +```bash +curl -s -X POST \ + -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + -H "Content-Type: application/json" \ + -d '{"workflowConfig": "projects/{{ PROJECT_ID }}/locations/{{ REGION }}/repositories/cymbal-dataform/workflowConfigs/cymbal-15min"}' \ + https://dataform.googleapis.com/v1/projects/{{ PROJECT_ID }}/locations/{{ REGION }}/repositories/cymbal-dataform/workflowInvocations +``` + +The response is a **workflow invocation** in state `RUNNING` — your pipeline, executing, with your terminal completely uninvolved from here on. + +❗ If the call (or the invocation itself) fails immediately with `PERMISSION_DENIED` and the message mentions the service account or the Dataform service agent, IAM propagation from the first section is still in flight — the same 3–4 minute lesson as the Datastream agent in Lab 2. Wait a few minutes and run the command again. + +### Watch gold refresh itself + +Back on the repository page, open the Workflow execution logs tab. Your invocation is there — **Running**, then **Succeeded** after a minute or two at seed-data scale. Click it: every model and every assertion is listed with its own status and timing. This page is where an operator lives; nobody watches a terminal at 3 a.m. + +Trust, but verify the identity too: open [BigQuery](https://console.cloud.google.com/bigquery), click **Job history** at the bottom, and switch to the **Project history** tab — the jobs from this run were executed by `dataquality-service-account`, not by you. + +Now the payoff. Run this in the BigQuery editor: + +```sql +SELECT order_date, currency, orders, gross_revenue +FROM `{{ PROJECT_ID }}.cymbal_gold.fct_daily_revenue` +ORDER BY order_date DESC, currency +LIMIT 8 +``` + +The top rows are *today* — and the numbers are already bigger than anything your Lab 3 run could have seen: the simulator has been writing orders all along, Datastream mirrored them into bronze, and the execution you just triggered folded them into gold. Re-run the query after the next quarter-hour tick (check the execution logs for the scheduled entry) and watch today's `orders` climb — no terminal, no `dataform run`, no you. + +### Challenge: run the production playbook + +**\[TASK\]** Take up to 10 minutes — pick at least one: + +1. **A gold-only lane**: create a second workflow configuration `cymbal-gold-only` on the same release configuration, scoped via *Selection of tags* to `gold` only, and trigger it with the same `curl` (swap the configuration ID at the end of the `workflowConfig` path in the request body). In the execution log, only the three marts and their assertions ran — scoping by tags is how production separates a cheap, frequent refresh from a full rebuild. +2. **Fail like production**: put the deliberately failing bronze assertion from Lab 3's challenge on a schedule of its own. If you didn't build it then, take the reference (`cp content/agenticdata/src/dataform_reference/definitions/assert_bronze_payments_non_negative.sqlx content/agenticdata/src/dataform/definitions/`), re-run the push script, and click **New compilation** — new code needs a new compilation result. Then create a workflow configuration `cymbal-challenge` scoped to the tag `challenge`, schedule `0 3 * * *` (the traditional hour for data-quality surprises), and trigger it via `curl`. Watch it turn **Failed** in the execution logs — 982 planted negative payments, caught by a robot. Your `cymbal-15min` lane stays green: it never runs the `challenge` tag. In production, the next step would be an alert on failed invocations — a story for an operations lab. +3. **Policy tags that survive robots**: your silver and gold tables are now rebuilt every 15 minutes — and a rebuilt table does not keep column settings that were attached by hand in the console, which matters the moment governance (policy tags on PII columns) enters the picture. Ask agy: *"stg_customers is rebuilt by managed Dataform every 15 minutes. Explain why a BigQuery policy tag attached by hand to its email column disappears on rebuild, and show how to declare it in the SQLX config with bigqueryPolicyTags so every rebuild reapplies it."* ([Control column access](https://docs.cloud.google.com/dataform/docs/policy-tags)) + +Note: If you are stuck and cannot figure out how to proceed after a few minutes, ask your team captain. And leave everything running when you're done — at this data size, the quarter-hourly executions cost cents, and nothing that follows depends on switching them off; cleaning up the schedule after the event is entirely optional. + +### Success + +🎉 Magnificent{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! The medallion left your laptop: agy's project now lives in a hosted repository, compiles from `main` on demand, and rebuilds itself every 15 minutes as a service account — while you read execution logs like an operator instead of running jobs like a machine. Wherever the afternoon goes next, gold will be fresher than you left it. ⏰ diff --git a/content/agenticdata/labs/8_forecast.md b/content/agenticdata/labs/8_forecast.md new file mode 100644 index 0000000..0fc5b3c --- /dev/null +++ b/content/agenticdata/labs/8_forecast.md @@ -0,0 +1,177 @@ +## Lab 8: The Crystal Ball — Revenue Forecasting with AI.FORECAST and Data Canvas + + +{{ author('Fabian Hirschmann', 'https://linkedin.com/in/fhirschmann') }} + + + + +So far your platform explains the past. In this lab it starts predicting the future: you forecast Cymbal's daily revenue with a **single SQL function** — `AI.FORECAST`, backed by the **TimesFM** foundation model built into BigQuery — then let agy materialize the forecast as a proper gold mart in your Dataform project, and chart actuals against forecast in a **data canvas** using natural language. + +Before you start: this lab assumes **Labs 1–3 are complete** — the medallion is built and `cymbal_gold.fct_daily_revenue` exists. Work in your main terminal as usual, and leave the tunnel (terminal 2) and the simulator (terminal 3) running untouched — everything here reads from BigQuery. The final section additionally needs the published `cymbal-data-agent` from Lab 5; it is clearly marked and skippable if you haven't done that lab. + +### About TimesFM and AI.FORECAST + +**TimesFM** is a foundation model for time series from Google Research, pre-trained on billions of time points — the forecasting equivalent of a pre-trained language model: it predicts new series *zero-shot*, without being trained on your data. BigQuery ML ships it **built in**, exposed through the GA table-valued function **`AI.FORECAST`**: no `CREATE MODEL`, no training job, no endpoint to deploy or manage — you hand it a query and get forecast rows back, accuracy comparable to classic statistical models like ARIMA. It has siblings: `AI.EVALUATE` scores forecasts against actuals, and `AI.DETECT_ANOMALIES` flags outliers — both wait for you in the challenge. + +Learn more: +- [The AI.FORECAST function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-forecast) +- [The built-in TimesFM model](https://docs.cloud.google.com/bigquery/docs/timesfm-model) +- [The AI.EVALUATE function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-evaluate) +- [The AI.DETECT_ANOMALIES function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-detect-anomalies) +- [BigQuery data canvas](https://docs.cloud.google.com/bigquery/docs/data-canvas) + +### Ask for the future + +How much history do you have to forecast from? Cymbal's seed data spans exactly **540 days** — 1 January 2025 through 24 June 2026 — and `fct_daily_revenue` holds one row per day *per currency*: 5 × 540 = 2,700 rows, plus a few partial rows for whatever the simulator had written by your last `dataform run`. Plenty (TimesFM needs only 3 points to say anything at all, and picks its context window automatically). + +Open [BigQuery](https://console.cloud.google.com/bigquery) and run your first forecast — two weeks of EUR revenue: + +```sql +SELECT * +FROM AI.FORECAST( + ( + SELECT order_date, gross_revenue + FROM `{{ PROJECT_ID }}.cymbal_gold.fct_daily_revenue` + WHERE currency = 'EUR' AND order_date < CURRENT_DATE() + ), + data_col => 'gross_revenue', + timestamp_col => 'order_date', + horizon => 14) +ORDER BY forecast_timestamp +``` + +A word on the arguments: the first is just a query — the series to forecast; `data_col` and `timestamp_col` name the value and time columns; `horizon => 14` asks for 14 points into the future. The `WHERE` clause makes two deliberate choices: **one currency**, because a series that mixes euros and złoty is numeric nonsense (more on that in a minute), and **no current day**, because the simulator is still writing it — a half-finished day would drag the first forecast points down. + +In a few seconds you get exactly 14 rows: `forecast_value` is the median prediction, flanked by `prediction_interval_lower_bound` and `_upper_bound` at the default 95% `confidence_level` (an empty `ai_forecast_status` means all went well). Note where the forecast starts: TimesFM continues the series from where your *data* ends — the day after the last complete day of history — not from today's wall-clock date. + +❗ At a full-room event, a hundred first-ever `AI.FORECAST` calls can land at the same moment. If you see a transient error mentioning quota or rate limits, wait a few seconds and re-run the query — the function is stateless, and nothing is left half-built in your project. + +### Materialize the forecast with Dataform + +Ad-hoc queries convince you; marts serve everyone else — dashboards, colleagues, and (below) your data agent. Let's make the forecast a real gold table, produced by the same pipeline that builds the rest of the medallion. This time the whole table is forecast per currency in one call using `id_cols`, which tells TimesFM to treat each currency as its own independent series — the honest alternative to summing incompatible currencies into one. Writing SQLX is authoring work, so it goes to agy. Change into your Dataform project from Lab 3 and start your co-engineer: + +```bash +cd ~/bootkon/content/agenticdata/src/dataform +agy +``` + +Then hand over the task (the prompt is also in prompts.md): + +``` +/goal Create definitions/fct_revenue_forecast.sqlx: a Dataform table in schema "cymbal_gold" carrying exactly one tag, "forecast" (not silver, not gold). It materializes a 14-day daily revenue forecast per currency from ${ref("fct_daily_revenue")} (columns order_date DATE, currency STRING, gross_revenue NUMERIC), excluding the current incomplete day (order_date < CURRENT_DATE()). Use BigQuery's GA table-valued function AI.FORECAST with its exact named arguments: the SELECT as first argument, then data_col => 'gross_revenue', timestamp_col => 'order_date', id_cols => ['currency'], horizon => 14, confidence_level => 0.95. From its output keep currency, forecast_timestamp cast to DATE as forecast_date, and forecast_value, prediction_interval_lower_bound, prediction_interval_upper_bound rounded to 2 decimals as forecast_revenue, lower_bound, upper_bound. Add a table description and column descriptions, then run `dataform compile` and fix any errors until it compiles cleanly. +``` + +**Review what agy wrote**: open fct_revenue_forecast.sqlx and compare with the shipped reference in src/optional/forecast. If agy went off-script, the reference has your back: + +```bash +cp ~/bootkon/content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx ~/bootkon/content/agenticdata/src/dataform/definitions/ +``` + +Exit agy and build it — the run finishes in well under a minute: + +```bash +dataform run --tags forecast +``` + +The tag is the point: the new model carries *only* `forecast`, so your Lab 3 commands — `dataform run --tags silver` and `--tags gold` — still execute exactly what they did before, while the forecast refreshes only when asked. Re-run it any time; since bronze keeps moving and gold is one `dataform run` away, tomorrow's forecast can always ride on today's data. + +Trust, but verify — back in the [BigQuery editor](https://console.cloud.google.com/bigquery): + +```sql +SELECT currency, COUNT(*) AS days, MIN(forecast_date) AS first_day, MAX(forecast_date) AS last_day +FROM `{{ PROJECT_ID }}.cymbal_gold.fct_revenue_forecast` +GROUP BY currency +ORDER BY currency +``` + +Five currencies, 14 days each — 70 rows of future, starting the day after your history ends. + +### Chart it in a data canvas + +Numbers convince analysts; lines convince everyone. **Data canvas** is BigQuery's Gemini-powered analysis surface: you assemble tables, queries, and charts as connected nodes on a canvas and drive them with natural language. + +In [BigQuery](https://console.cloud.google.com/bigquery), next to the **SQL query** button in the editor tab bar, click the **Create new** arrow, select **AI and knowledge**, then Data canvas. Click **Search for data** and search in plain words: + +``` +cymbal revenue +``` + +The search reads the catalog's metadata (if you did Lab 4, that's the metadata you curated). Add both `fct_daily_revenue` and `fct_revenue_forecast` to the canvas. Then, on the `fct_daily_revenue` table node, click **Query** and describe what you want instead of writing it: + +``` +One row per day and series: the last 60 complete days of EUR actuals from fct_daily_revenue (order_date as day, gross_revenue as revenue, series = 'actual', excluding the current date), combined with all EUR rows of cymbal_gold.fct_revenue_forecast (forecast_date as day, forecast_revenue as revenue, series = 'forecast'). +``` + +Gemini writes the SQL — **read it before you run it**, same as you review agy all day. If it took a wrong turn, replace the node's SQL with the known-good version: + +```sql +WITH actuals AS ( + SELECT order_date, gross_revenue + FROM `{{ PROJECT_ID }}.cymbal_gold.fct_daily_revenue` + WHERE currency = 'EUR' AND order_date < CURRENT_DATE() +) +SELECT 'actual' AS series, order_date AS day, gross_revenue AS revenue +FROM actuals +WHERE order_date >= (SELECT DATE_SUB(MAX(order_date), INTERVAL 59 DAY) FROM actuals) +UNION ALL +SELECT 'forecast', forecast_date, forecast_revenue +FROM `{{ PROJECT_ID }}.cymbal_gold.fct_revenue_forecast` +WHERE currency = 'EUR' +ORDER BY day +``` + +(Why anchor the 60-day window on the *data's* `MAX(order_date)` rather than on today? Same lesson as the warm-up: the seed history ends on 24 June 2026, so a window counted back from today's date could miss the actuals entirely.) + +Run the node, then click **Visualize** and pick the line chart. You can keep refining conversationally — try: + +``` +Line chart over day, one line per series, forecast in a different color +``` + +Look at the seam where the two series meet: the forecast picks up the level and the weekly rhythm of the actuals and carries them forward, with no model training anywhere in sight. Export the chart as a PNG if you want a souvenir. + +❗ Data canvas is a **Gemini in BigQuery** feature, and some sandboxes gate it by org policy. If **Data canvas** doesn't appear under *Create new*, you lose the conversation but not the picture: run the known-good query above in the plain SQL editor and open the Visualization tab (formerly *Chart*) in the query results pane — a line chart over `day` shows the same seam. + +### Teach the data agent to see the future + +**If you completed Lab 5**, your `cymbal-data-agent` is about to get an upgrade — otherwise skip ahead to the challenge; nothing below is needed there. + +First, establish the "before". Open [BigQuery Agents](https://console.cloud.google.com/bigquery/agents_hub), on the **Agent Catalog** tab find the ``cymbal-data-agent`` card, open it, and start a conversation. Then stress-test it Lab 5 style with something forward-looking: + +``` +What will revenue look like next week? +``` + +The best it can do is admit its world ends at the latest `order_date` — or worse, improvise. That's the right behavior with nothing governed to stand on. Now there is something: a forecast mart, built by your pipeline, documented in your catalog. Wire it in: + +1. Back on the **Agent Catalog** tab, open the agent for editing via the three-dot Open actions menu → **Edit**. +2. Under *Knowledge sources*, click Add source, search for ``fct_revenue_forecast``, and add it as the fourth source. + +Then append one rule to the agent instructions — it extends the "never sum across currencies" discipline into the future: + +``` +- fct_revenue_forecast: use for forward-looking questions ("what will", "next week", "forecast", "projection"). forecast_revenue is the median TimesFM forecast per currency and forecast_date; always report lower_bound and upper_bound alongside it, and never sum forecasts across currencies. For dates beyond the last forecast_date, say the forecast horizon ends there instead of guessing. +``` + +Click **Save**, then **Publish** — the published resource is what the API (and your Lab 6 analyst, if it's still running) sees. Now start a new conversation and ask for the future — anchored to the *data*, not the wall clock, same lesson as the warm-up: + +``` +What does our revenue forecast look like, per currency? +``` + +Watch the SQL it generates: a query against `cymbal_gold.fct_revenue_forecast`, per currency, medians with their intervals — governed data in, honest uncertainty out. That's the difference between an agent that *sounds* confident and one that has something to be confident about. (Re-ask the *"next week"* question too: if the seed's 14-day forecast window still covers next week, you now get numbers; at a later event, the agent tells you the horizon ends before then instead of guessing — the rule you just wrote, doing its job either way.) + +### Challenge: interrogate the crystal ball + +**\[TASK\]** Take up to 10 minutes — pick at least one: + +1. **Backtest it**: how good would the forecast have been? `AI.EVALUATE` re-forecasts the past and scores it against what actually happened — hold out the last two seed weeks (11–24 June 2026) and measure. Run the shipped query challenge_backtest.sql in the BigQuery editor and compare the five currencies: high-volume EUR usually beats thin PLN — volume smooths series, and smooth series forecast better. Discuss with your table how you would keep such a backtest honest in production (hint: it's one more tagged Dataform model). +2. **Upgrade the engine**: the default model is TimesFM 2.0. Re-run the warm-up query with `model => 'TimesFM 2.5'` — the newer generation with a much longer context window (up to 15,360 points vs 2,048). On 540 days of history the numbers will differ only modestly; ask agy when the difference *would* matter. +3. **Anomaly patrol**: `AI.DETECT_ANOMALIES` uses the same TimesFM machinery in reverse — forecast the past, then flag actuals that fall outside the prediction interval. Run challenge_anomalies.sql over the last 60 seed days: Cymbal's steady synthetic revenue should yield few flags — possibly none, which is itself the finding. If you did Lab 5, you already asked the agent about "days with unusually high revenue" — this puts that hunch on statistical footing. + +Note: If you are stuck and cannot figure out how to proceed after a few minutes, ask your team captain. + +### Success + +🎉 Visionary{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! One function call turned 540 days of governed history into a two-week revenue outlook — no model trained, no endpoint managed — and you shipped it properly: a tagged Dataform mart, a natural-language chart, and (if your agent got the upgrade) forward-looking answers with honest error bars. Cymbal no longer just knows what happened — it knows what's coming. 🔮 diff --git a/content/agenticdata/labs/9_sharing.md b/content/agenticdata/labs/9_sharing.md new file mode 100644 index 0000000..e8cab24 --- /dev/null +++ b/content/agenticdata/labs/9_sharing.md @@ -0,0 +1,131 @@ +## Lab 9: The Data Exchange — Sharing the Gold + + +{{ author('Fabian Hirschmann', 'https://linkedin.com/in/fhirschmann') }} + + + + +Remember where this story started: Cymbal's analytics team drowning in one-off CSV exports. You built the cure for the *inside* of the company — but Cymbal's partners still get their numbers the old way: somebody exports a CSV, attaches it to an email, and it's stale before it lands. In this lab you close that loop with **BigQuery sharing**: publish the gold marts as a **listing** in your own **data exchange**, then switch chairs and subscribe to it exactly like a partner would — receiving a live, read-only, zero-copy view instead of an attachment. + +This lab assumes Labs 1–3 are complete (the `cymbal_gold` marts exist and have been built at least once) and that your two background terminals from Lab 2 are still open: the tunnel (terminal 2) and the data simulator (terminal 3) — the simulator is what makes the zero-copy proof land. If you also did Lab 4's data-product challenge, you'll recognize the shape: the catalog *described* the contract; now you *distribute* it. (Nothing below depends on Lab 4.) + +### About BigQuery sharing + +**BigQuery sharing** is BigQuery's built-in distribution channel — you may know it as **Analytics Hub**: Google renamed it, the console page is labeled *Sharing (Analytics Hub)*, and the API keeps the old name. The moving parts: a **data exchange** is a catalog you own — a container for listings, created in your project, no organization-level setup required. A **listing** is one shareable asset (here: a BigQuery dataset) plus the metadata a consumer needs to trust it. When someone subscribes, they get a **linked dataset** in their own project: a read-only *pointer* to your data. Because BigQuery separates storage from compute, a thousand subscribers cost you no extra storage — and publishers stay in control, with egress restrictions, usage metrics, and revocable subscriptions. + +Learn more: +- [Introduction to BigQuery sharing](https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction) +- [Manage listings](https://docs.cloud.google.com/bigquery/docs/analytics-hub-manage-listings) +- [View and subscribe to listings](https://docs.cloud.google.com/bigquery/docs/analytics-hub-view-subscribe-listings) + +### Enable sharing and put on both hats + +BigQuery sharing has its own API and its own IAM roles: **Analytics Hub Admin** runs exchanges and listings, **Analytics Hub Subscriber** consumes them. You'll play both sides today, so grant yourself both (idempotent, safe to re-run): + +```bash +gcloud services enable analyticshub.googleapis.com +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member=user:{{ GCP_USERNAME }} --role=roles/analyticshub.admin --condition=None +gcloud projects add-iam-policy-binding {{ PROJECT_ID }} \ + --member=user:{{ GCP_USERNAME }} --role=roles/analyticshub.subscriber --condition=None +``` + +❗ **Permission denied on the Sharing page?** Fresh IAM grants can take a minute to propagate — wait a moment and reload the page. + +### Create the exchange + +The exchange is Cymbal's shop window — create it in the console: + +1. Open [Sharing (Analytics Hub)](https://console.cloud.google.com/bigquery/analytics-hub) and click Create exchange. +2. Fill in: + - Display name: `cymbal-exchange` + - Region: **US** (the multi-region) — a listing can only share a dataset that lives in the exchange's region, and your medallion lives in the `US` multi-region. + - Description: `Cymbal's curated data products` (primary contact is optional). +3. Click **Create exchange**. When the *Exchange permissions* step appears, **Skip** it — you granted yourself everything you need a minute ago (in real life, this is where you'd invite publishers and subscribers by email). + +Notice what you did *not* need: an organization admin. Exchanges are project-level resources — any team that owns a project can open a shop. + +### Draft the data contract with agy + +A listing is only as trustworthy as its description: a consumer who can't tell what's in the tables, how fresh they are, or what "revenue" means will fall back to emailing you for a CSV. Turning live schemas into honest prose is authoring work — agy's job. Run this one-shot in terminal 1 (the prompt is also in prompts.md): + +```bash +agy -p "Read the table schemas: bq show --schema {{ PROJECT_ID }}:cymbal_gold.fct_daily_revenue ; bq show --schema {{ PROJECT_ID }}:cymbal_gold.dim_customer_360 ; bq show --schema {{ PROJECT_ID }}:cymbal_gold.fct_product_performance. Then write a listing description for sharing these three tables with Cymbal's partners — a data contract as prose, in markdown: one paragraph on what the dataset is, a short section per table (grain, key columns, what questions it answers), a freshness section (the tables are fed by live Datastream CDC from the production Postgres and rebuilt by Dataform runs), and a definitions section (gross revenue is SUM(qty * unit_price) excluding cancelled orders; lifetime value is the gross revenue of a customer's non-cancelled orders; in fct_daily_revenue amounts are per currency and must never be summed across currencies; lifetime_value and the fct_product_performance amounts sum across order currencies without conversion, so present them as activity indicators, not accounting figures). Output only the markdown." +``` + +Review the draft — you are about to sign it in Cymbal's name. A known-good contract ships with the repository: open listing_description.md to compare — or to copy from directly, if agy's draft went off-script or you'd rather not select text in a terminal: + +```bash +cat content/agenticdata/src/optional/sharing/listing_description.md +``` + +### Publish the listing + +1. Back on the Sharing page, open cymbal-exchange and click Create listing. +2. Resource type: **BigQuery dataset** → select `cymbal_gold`. +3. Listing details: + - Display name: `Cymbal Gold — Commerce Analytics` + - Description: `Analysis-ready commerce marts: daily revenue, customer 360, product performance. Live CDC + Dataform.` + - Paste your reviewed data contract into the **Documentation** field (it takes markdown). +4. Find the **data egress controls** and tick *Disable copy and export of shared data* — partners get to query the gold, not to bulk-exfiltrate it. (Note the other two options: you can also restrict copying of query results and API-level table exports.) +5. Click Publish. + +Your gold layer is now a product with a front door — description, contract, contact, and house rules. + +### Subscribe like a partner + +Time to switch chairs. Open your freshly published listing from the exchange page — this is exactly what a consumer sees: your description, your contract, your contact. + +1. Click Subscribe. A *Create linked dataset* dialog appears. +2. Fill in: + - Project: `{{ PROJECT_ID }}` — yes, the same project; a publisher can subscribe to their own listing, which is exactly how you test what partners will experience. + - Linked dataset name: `cymbal_gold_linked` + - Primary region: **US** (the multi-region, matching the source). +3. Click Save. + +❗ **The dialog refuses your own project?** Pair up with a neighbor and subscribe to each other's listings instead — the Challenge below has the exact steps. Name the linked dataset `cymbal_gold_linked` and everything below works the same, on their live data instead of your own. + +Now look at what arrived: open [BigQuery](https://console.cloud.google.com/bigquery), expand your project, and find cymbal_gold_linked — wearing a small *linked* badge on its dataset icon. Click into `fct_daily_revenue`: same schema, same descriptions — but read-only. No copy job ran, no bytes moved; this dataset is a pointer. + +### Prove it's a pointer, not a copy + +"Zero-copy" is a fine slogan — trust, but verify. In the BigQuery query editor (**Compose new query**), run the source and the linked dataset side by side (order counts can be totaled across currencies; revenue can't — your own contract says so — hence euros only): + +```sql +SELECT 'source' AS via, SUM(orders) AS orders, + ROUND(SUM(IF(currency = 'EUR', gross_revenue, 0)), 2) AS eur_revenue +FROM `{{ PROJECT_ID }}.cymbal_gold.fct_daily_revenue` +UNION ALL +SELECT 'linked', SUM(orders), + ROUND(SUM(IF(currency = 'EUR', gross_revenue, 0)), 2) +FROM `{{ PROJECT_ID }}.cymbal_gold_linked.fct_daily_revenue` +``` + +Two identical rows — of course: it's the same storage. Expect roughly 450,000 non-cancelled orders; the exact number depends on how long your simulator has been running. Now make the *source* move. Your simulator (terminal 3) has been writing to Postgres all along, and Datastream has been keeping bronze fresh — refresh silver from live bronze and rebuild gold, same commands as Lab 3, each run finishing in well under a minute: + +```bash +cd ~/bootkon/content/agenticdata/src/dataform +dataform run --tags silver && dataform run --tags gold +``` + +Re-run the query above. Both rows climbed — **in lockstep**. Nobody re-exported anything, no partner pipeline re-ran, and there is no second copy that could go stale. The CSV attachment could never do that. + +### Check the meters + +One more publisher privilege: back on [Sharing (Analytics Hub)](https://console.cloud.google.com/bigquery/analytics-hub), open `cymbal-exchange`, switch to its Usage metrics tab, and pick your listing from the *Listings* menu. As the publisher you see total subscriptions, subscribers per organization, jobs executed, bytes scanned, and which shared tables get queried most ([monitor listings](https://docs.cloud.google.com/bigquery/docs/analytics-hub-monitor-listings)). Your own subscription should already be counted; the job charts may trail the queries you just ran — check back later. Under the hood the same numbers live in a SQL view, ready for the day you want to build a mart about your marts: `` `region-us`.INFORMATION_SCHEMA.SHARED_DATASET_USAGE ``. + +And the egress control? Try copying a table out of `cymbal_gold_linked` (three-dot menu next to the table → *Copy*) — you ticked the box, so the copy is denied. Querying: yes. Exfiltrating: no. + +### Challenge: subscribe to a neighbor's listing + +**\[TASK\]** Take up to 10 minutes — data sharing is more fun with two projects: + +1. Team up with a neighbor. Each of you opens your own listing, clicks *Set permissions* → *Add principal*, and adds the other's account as an **Analytics Hub Subscriber**. Then find their listing — *Search listings* on the Sharing page, filter to *Private* — and subscribe to it as `cymbal_gold_neighbor`: you are now watching *their* simulator's numbers, live, from your project. ❗ This one depends on the sandbox's org policy: **domain-restricted sharing** may block grants to other accounts. If IAM refuses, that's the policy working as intended — move on to option 2. +2. Plan B (works everywhere): on *Search listings*, filter to **public** listings and subscribe to one Google publishes — the **Google Trends** listing is a classic. Top search terms land as a linked dataset next to your gold, zero copies, same mechanism you just built for Cymbal. + +Note: If you are stuck and cannot figure out how to proceed after a few minutes, ask your team captain. + +### Success + +🎉 Superb{% if MY_NAME %}, {{ MY_NAME }}{% endif %}! The CSV-export era at Cymbal is officially over: the gold layer is now a published data product with a prose contract, an egress policy, usage metering — and subscribers who get a live pointer instead of a stale attachment. Publish once, subscribe anywhere. 📬→📡 diff --git a/content/agenticdata/src/optional/day2_ops/AGENTS.md b/content/agenticdata/src/optional/day2_ops/AGENTS.md new file mode 100644 index 0000000..005f26e --- /dev/null +++ b/content/agenticdata/src/optional/day2_ops/AGENTS.md @@ -0,0 +1,98 @@ +# Brief: the Cymbal Day-2 Ops dashboard (Cloud Monitoring, as code) + +Written for an AI coding agent. Your task: create **`dashboard.json`** in THIS +directory (`content/agenticdata/src/optional/day2_ops/`) — a Cloud Monitoring +dashboard definition that a human deploys with +`gcloud monitoring dashboards create --config-from-file=dashboard.json`. + +The dashboard watches one CDC pipeline end to end: +Cloud SQL Postgres (`cymbal-oltp`) → Datastream (`cymbal-cdc-stream`) → +BigQuery (`cymbal_bronze`). It must answer, at a glance and at 3 a.m.: +*is data flowing, and if not, which hop is broken?* + +## Output contract + +- Valid **JSON only**: double quotes, no comments, no trailing commas. +- Use the **Cloud Monitoring Dashboard API v1** shape, exactly this skeleton: + +```json +{ + "displayName": "Cymbal Day-2 Ops", + "gridLayout": { + "columns": "2", + "widgets": [ + { + "title": "", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"...\" resource.type=\"...\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "", "scale": "LINEAR" } + } + } + ] + } +} +``` + +- `filter` is a single string: `metric.type="..." resource.type="..."`, + space-separated. Do not add per-resource filters — this project contains + exactly one stream and one instance, so project scope is enough. +- Do not invent widgets, fields, or metric types beyond this brief. + +## The six widgets, in exactly this order + +`gridLayout` places widgets row by row (2 columns), so this order pairs each +pipeline hop with its source-side counterpart: + +| # | title | metric.type | resource.type | aligner | yAxis label | +|---|---|---|---|---|---| +| 1 | `Datastream - data freshness (s)` | `datastream.googleapis.com/stream/freshness` | `datastream.googleapis.com/Stream` | `ALIGN_MAX` | `seconds` | +| 2 | `Cloud SQL - PostgreSQL connections` | `cloudsql.googleapis.com/database/postgresql/num_backends` | `cloudsql_database` | `ALIGN_MEAN` | `connections` | +| 3 | `Datastream - throughput (events/s)` | `datastream.googleapis.com/stream/event_count` | `datastream.googleapis.com/Stream` | `ALIGN_RATE` | `events/s` | +| 4 | `Cloud SQL - CPU utilization` | `cloudsql.googleapis.com/database/cpu/utilization` | `cloudsql_database` | `ALIGN_MEAN` | `utilization (0-1)` | +| 5 | `BigQuery Storage Write API - ingested rows/min` | `bigquerystorage.googleapis.com/write/uploaded_row_count` | `bigquery_project` | `ALIGN_SUM` | `rows/min` | +| 6 | `Datastream - unsupported events` | `datastream.googleapis.com/stream/unsupported_event_count` | `datastream.googleapis.com/Stream` | `ALIGN_SUM` | `events` | + +These metric types are **verified against the July 2026 metrics list** — use +them character for character, even if you believe a different name. In +particular: Datastream metrics live under `datastream.googleapis.com/stream/*` +(singular `stream`), and the Datastream monitored resource type is the full +string `datastream.googleapis.com/Stream`. + +## Widget 1 extra: the SLO line + +Widget 1 (data freshness) additionally gets a threshold line, as a sibling of +`dataSets` inside its `xyChart`: + +```json +"thresholds": [ + { "value": 120, "label": "Freshness SLO (120 s)" } +] +``` + +Do NOT add `color` or `direction` fields — the API rejects both inside an +xyChart threshold (`INVALID_ARGUMENT`, verified live against the API; those +fields belong to scorecard thresholds). `value` and `label` are enough; the +console draws the line red on its own. + +`stream/freshness` is a GAUGE in **seconds**: how far Datastream lags behind +the source WAL (0 when fully caught up). 120 seconds is Cymbal's freshness +SLO — the same threshold the on-call alert uses. + +## Done when + +- `dashboard.json` parses as JSON (`python3 -m json.tool dashboard.json`). +- It contains exactly 6 widgets in the order above, with the exact metric + types, aligners, and the one threshold from this brief. diff --git a/content/agenticdata/src/optional/day2_ops/POSTMORTEM.md b/content/agenticdata/src/optional/day2_ops/POSTMORTEM.md new file mode 100644 index 0000000..85db398 --- /dev/null +++ b/content/agenticdata/src/optional/day2_ops/POSTMORTEM.md @@ -0,0 +1,80 @@ +# Postmortem: bronze freeze — Datastream replication login lockout + +*Reference example, shipped with the lab. Yours (written with agy from your +own incident timeline) will differ in times and detail — compare with +`git -C ~/bootkon diff content/agenticdata/src/optional/day2_ops/POSTMORTEM.md`.* + +- **Status:** resolved +- **Severity:** SEV-2 — analytics data staleness; **no data loss** +- **Systems:** Cloud SQL `cymbal-oltp` → Datastream `cymbal-cdc-stream` → BigQuery `cymbal_bronze` + +## Summary + +The CDC stream `cymbal-cdc-stream` stopped replicating for roughly 15 minutes +after the replication role `datastream_user` was set to `NOLOGIN` and its +active replication connection was terminated. The source database remained +fully available; operational writes were never affected. Bronze (and every +layer built on it) served increasingly stale data until the role was restored +and Datastream reconnected. The replication slot `cymbal_slot` retained WAL +for the entire outage, so replication resumed from the exact position where +it stopped — zero events lost. + +## Impact + +- BigQuery `cymbal_bronze` data freshness degraded from <60 s to a peak of + ~15 minutes. Downstream silver/gold marts and the Cymbal data agent served + stale (not wrong) numbers. +- No impact on the order platform: Postgres accepted every write throughout. +- No data loss: post-recovery, `MAX(order_id)` in bronze caught up with the + source within minutes. + +## Timeline (UTC, T0 = chaos injection) + +| Time | Event | +|---|---| +| T0 | `chaos.sh` sets `datastream_user` to `NOLOGIN` and terminates its backend. | +| T0+1m | Dashboard: *Datastream — data freshness* starts climbing; throughput drops to 0. Cloud SQL connections/CPU unchanged — first hint that the source is healthy. | +| T0+2m | Freshness crosses the 120 s SLO line on the dashboard. | +| T0+3–8m | Alerting policy `cymbal-freshness-slo` fires; incident opens on the Alerting page. | +| T0+5m | Datastream console shows the stream unhealthy; stream errors report a failed login for `datastream_user`. | +| T0+~10m | Diagnosis: agy (read-only, human-approved commands) correlates `gcloud datastream streams describe` state/errors with Cloud Logging entries: `role "datastream_user" is not permitted to log in`. Reading `chaos.sh` confirms root cause. | +| T0+~12m | Fix applied by on-call: `ALTER ROLE datastream_user LOGIN;` | +| T0+~15m | Datastream retry succeeds; stream returns to `RUNNING`; freshness falls back toward 0; bronze catches up losslessly from the replication slot. | +| T0+~20m | Freshness below SLO; the incident auto-resolves. | + +## Root cause + +The replication role `datastream_user` was altered to `NOLOGIN` and its active +session terminated (a controlled chaos drill). Datastream could neither keep +nor re-establish its source connection and entered a retry loop. + +## Detection + +The custom data-freshness alerting policy (threshold 120 s on +`datastream.googleapis.com/stream/freshness`) — created that same afternoon. +**Datastream ships no default alerts**; without this policy, detection would +have waited for a human to notice stale dashboards. + +## What went well + +- The dashboard localized the fault in seconds: stream metrics sick, source + metrics healthy → the problem sits between them. +- The replication slot did its job: WAL was retained, recovery was lossless. +- The agent-assisted diagnosis loop (human authorizes, agent investigates) + produced the root cause before the incident was 15 minutes old. + +## What went poorly + +- Alert latency (metric ingestion + evaluation) meant 3–8 quiet minutes + before the page — the dashboard beat the pager. +- No log-based alert on Datastream error logs existed; it would have fired + on the first failed login attempt. + +## Action items + +| # | Action | Type | +|---|---|---| +| 1 | Add a log-based alert on Datastream error logs (`resource.type="datastream.googleapis.com/Stream"`, severity ≥ ERROR). | detect | +| 2 | Document the `datastream_user` runbook: check role attributes (`\du`) before deeper debugging. | mitigate | +| 3 | Alert on `stream/unsupported_event_count` > 0 — the "silent partial failure" cousin of this incident. | detect | +| 4 | Review who can `ALTER ROLE` in production; replication roles should be change-controlled. | prevent | diff --git a/content/agenticdata/src/optional/day2_ops/chaos.sh b/content/agenticdata/src/optional/day2_ops/chaos.sh new file mode 100755 index 0000000..94be744 --- /dev/null +++ b/content/agenticdata/src/optional/day2_ops/chaos.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# ===================================================================== +# Cymbal SRE chaos drill -- "The Quiet Bronze" +# +# You were told to run this WITHOUT reading it first. If you are here +# now, you are either mid-diagnosis (welcome -- the crime scene is +# below) or you peeked. We will pretend it is the former. +# +# WHAT THIS DOES -- two SQL statements, nothing more: +# +# 1. ALTER ROLE datastream_user NOLOGIN; +# Datastream's replication user may not open NEW connections. +# +# 2. SELECT pg_terminate_backend(pid) FROM pg_stat_activity +# WHERE usename = 'datastream_user'; +# ...and its CURRENT connection is killed. Datastream now +# retries, fails to log in ("role \"datastream_user\" is not +# permitted to log in"), and the stream stalls. Postgres itself +# is perfectly healthy -- the simulator writes as user +# `postgres` and never notices a thing. Bronze freezes while +# the source keeps humming: the worst kind of quiet. +# +# WHY IT IS SAFE (strictly reversible): +# The ONLY thing touched is the datastream_user role attribute -- +# never the data, never the publication (cymbal_pub), never the +# replication slot (cymbal_slot). The slot keeps retaining WAL while +# Datastream is locked out, so nothing is lost -- just delayed. +# +# THE FIX (one line, typed by a human, not by this script): +# ALTER ROLE datastream_user LOGIN; +# ===================================================================== +set -euo pipefail +: "${BK_DB_PASSWORD:?BK_DB_PASSWORD is not set -- run: source ~/.bashrc}" + +export PGPASSWORD="$BK_DB_PASSWORD" +psql -v ON_ERROR_STOP=1 \ + -h "${BK_CYMBAL_DB_HOST:-localhost}" -p "${BK_CYMBAL_DB_PORT:-5432}" \ + -U postgres -d cymbal <<'SQL' +ALTER ROLE datastream_user NOLOGIN; +SELECT pg_terminate_backend(pid) AS replication_connection_terminated +FROM pg_stat_activity +WHERE usename = 'datastream_user'; +SQL + +echo +echo "[chaos] $(date -u +%H:%M:%S) UTC -- injected. Write this time down: your incident timeline starts now." +echo "[chaos] Postgres is fine. The simulator is fine. Something else is not." +echo "[chaos] Do NOT read this script yet. Go look at your dashboard." diff --git a/content/agenticdata/src/optional/day2_ops/dashboard.json b/content/agenticdata/src/optional/day2_ops/dashboard.json new file mode 100644 index 0000000..1f0ea3e --- /dev/null +++ b/content/agenticdata/src/optional/day2_ops/dashboard.json @@ -0,0 +1,134 @@ +{ + "displayName": "Cymbal Day-2 Ops", + "gridLayout": { + "columns": "2", + "widgets": [ + { + "title": "Datastream - data freshness (s)", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"datastream.googleapis.com/stream/freshness\" resource.type=\"datastream.googleapis.com/Stream\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_MAX" + } + } + }, + "plotType": "LINE" + } + ], + "thresholds": [ + { + "value": 120, + "label": "Freshness SLO (120 s)" + } + ], + "yAxis": { "label": "seconds", "scale": "LINEAR" } + } + }, + { + "title": "Cloud SQL - PostgreSQL connections", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"cloudsql.googleapis.com/database/postgresql/num_backends\" resource.type=\"cloudsql_database\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_MEAN" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "connections", "scale": "LINEAR" } + } + }, + { + "title": "Datastream - throughput (events/s)", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"datastream.googleapis.com/stream/event_count\" resource.type=\"datastream.googleapis.com/Stream\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_RATE" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "events/s", "scale": "LINEAR" } + } + }, + { + "title": "Cloud SQL - CPU utilization", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"cloudsql.googleapis.com/database/cpu/utilization\" resource.type=\"cloudsql_database\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_MEAN" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "utilization (0-1)", "scale": "LINEAR" } + } + }, + { + "title": "BigQuery Storage Write API - ingested rows/min", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"bigquerystorage.googleapis.com/write/uploaded_row_count\" resource.type=\"bigquery_project\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_SUM" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "rows/min", "scale": "LINEAR" } + } + }, + { + "title": "Datastream - unsupported events", + "xyChart": { + "dataSets": [ + { + "timeSeriesQuery": { + "timeSeriesFilter": { + "filter": "metric.type=\"datastream.googleapis.com/stream/unsupported_event_count\" resource.type=\"datastream.googleapis.com/Stream\"", + "aggregation": { + "alignmentPeriod": "60s", + "perSeriesAligner": "ALIGN_SUM" + } + } + }, + "plotType": "LINE" + } + ], + "yAxis": { "label": "events", "scale": "LINEAR" } + } + } + ] + } +} diff --git a/content/agenticdata/src/optional/day2_ops/prompts.md b/content/agenticdata/src/optional/day2_ops/prompts.md new file mode 100644 index 0000000..7dbf5ee --- /dev/null +++ b/content/agenticdata/src/optional/day2_ops/prompts.md @@ -0,0 +1,51 @@ +# Canonical agy prompts — Lab: Day-2 Ops + +The prompts quoted in the day-2 ops lab, collected in one place so table +captains can copy them quickly. agy output varies by design — every artifact +has a reference in this folder that participants can fall back to. This file +lives next to the lab's other assets while the module is optional; merge it +into `content/agenticdata/src/prompts.md` when the lab is wired into +TUTORIAL.md. + +## Author the dashboard + +Run from `~/bootkon`: + +``` +/goal Read content/agenticdata/src/optional/day2_ops/AGENTS.md and create content/agenticdata/src/optional/day2_ops/dashboard.json exactly as it specifies: valid Cloud Monitoring Dashboard API JSON, six widgets in the given order, the exact metric types from the brief, and the 120-second threshold line on the freshness chart. No comments, no trailing commas. +``` + +Fallback: `git -C ~/bootkon restore content/agenticdata/src/optional/day2_ops/dashboard.json` + +## Diagnose the incident (read-only co-responder) + +``` +Our BigQuery bronze dataset stopped receiving new rows a few minutes ago, and data freshness on the Datastream stream cymbal-cdc-stream (location us-central1) keeps climbing. The source Postgres itself is healthy and still taking writes. Investigate the Datastream side with read-only commands only (gcloud datastream streams describe, gcloud logging read) and tell me your diagnosis. Ask me before each command, and do not change anything. +``` + +Fallback — the two commands that hold the answer, run by hand: + +``` +gcloud datastream streams describe cymbal-cdc-stream --location=us-central1 --format='yaml(state, errors)' +gcloud logging read 'resource.type="datastream.googleapis.com/Stream" severity>=WARNING' --freshness=15m --limit=5 --format='yaml(timestamp, severity, jsonPayload)' +``` + +## Explain the chaos script + +``` +Read content/agenticdata/src/optional/day2_ops/chaos.sh and explain, statement by statement, how it broke replication -- and why the simulator kept writing happily the whole time. +``` + +Fallback: the header comment of `chaos.sh` explains itself, statement by statement. + +## Challenge — the postmortem + +Run in the *same* agy session that witnessed the incident: + +``` +/goal Write a blameless postmortem of the incident you just helped diagnose to content/agenticdata/src/optional/day2_ops/POSTMORTEM.md, overwriting the shipped example. Sections: summary, impact, timeline with the actual times from this session (the chaos script printed T0), root cause, detection, resolution, and action items. Blameless means: name systems and decisions, not people. +``` + +Fallback: the shipped `POSTMORTEM.md` *is* the reference — +`git -C ~/bootkon restore content/agenticdata/src/optional/day2_ops/POSTMORTEM.md` +brings it back after comparing with `git -C ~/bootkon diff`. diff --git a/content/agenticdata/src/optional/forecast/challenge_anomalies.sql b/content/agenticdata/src/optional/forecast/challenge_anomalies.sql new file mode 100644 index 0000000..a5abbef --- /dev/null +++ b/content/agenticdata/src/optional/forecast/challenge_anomalies.sql @@ -0,0 +1,28 @@ +-- Lab challenge: anomaly patrol with AI.DETECT_ANOMALIES. +-- +-- The same TimesFM machinery in reverse: forecast the past, then flag actual +-- days that fall outside the prediction interval. Checks the last 60 complete +-- days of each currency's series. Run this in the BigQuery editor of your own +-- project (the unqualified dataset name resolves there automatically). +-- +-- Cymbal's steady synthetic revenue should yield few flags -- possibly none, +-- which is itself the finding. Drop the `WHERE is_anomaly` filter to see the +-- per-day anomaly probabilities and bounds for every checked point. +SELECT + currency, + time_series_timestamp AS day, + time_series_data AS revenue, + ROUND(anomaly_probability, 2) AS anomaly_probability +FROM + AI.DETECT_ANOMALIES( + ( + SELECT order_date, currency, gross_revenue + FROM cymbal_gold.fct_daily_revenue + WHERE order_date < CURRENT_DATE() + ), + data_col => 'gross_revenue', + timestamp_col => 'order_date', + id_cols => ['currency'], + target_last_n_points => 60) +WHERE is_anomaly +ORDER BY currency, day diff --git a/content/agenticdata/src/optional/forecast/challenge_backtest.sql b/content/agenticdata/src/optional/forecast/challenge_backtest.sql new file mode 100644 index 0000000..aea4541 --- /dev/null +++ b/content/agenticdata/src/optional/forecast/challenge_backtest.sql @@ -0,0 +1,31 @@ +-- Lab challenge: backtest the revenue forecast with AI.EVALUATE. +-- +-- Holds out the last two seed weeks (2026-06-11 .. 2026-06-24 = 14 days), +-- forecasts them from everything before, and scores the forecast against +-- what actually happened -- per currency, because each currency is its own +-- series. Run this in the BigQuery editor of your own project (the +-- unqualified dataset name resolves there automatically). +-- +-- mape = mean absolute percentage error (lower is better); +-- rmse = root mean squared error, in the row's currency. +SELECT + currency, + ROUND(mean_absolute_percentage_error, 1) AS mape, + ROUND(root_mean_squared_error, 0) AS rmse +FROM + AI.EVALUATE( + ( + SELECT order_date, currency, gross_revenue + FROM cymbal_gold.fct_daily_revenue + WHERE order_date < '2026-06-11' + ), + ( + SELECT order_date, currency, gross_revenue + FROM cymbal_gold.fct_daily_revenue + WHERE order_date BETWEEN '2026-06-11' AND '2026-06-24' + ), + data_col => 'gross_revenue', + timestamp_col => 'order_date', + id_cols => ['currency'], + horizon => 14) +ORDER BY currency diff --git a/content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx b/content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx new file mode 100644 index 0000000..f80152d --- /dev/null +++ b/content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx @@ -0,0 +1,35 @@ +config { + type: "table", + schema: "cymbal_gold", + tags: ["forecast"], + description: "14-day daily gross revenue forecast per currency, generated by BigQuery ML's built-in TimesFM model via AI.FORECAST. Carries only the 'forecast' tag so the Lab 3 silver/gold runs stay untouched; rebuild with `dataform run --tags forecast`.", + columns: { + currency: "ISO 4217 currency code — each currency is forecast as its own independent series", + forecast_date: "Forecasted calendar date (UTC), starting the day after the last complete day of history", + forecast_revenue: "Median (50% quantile) forecasted gross revenue in the row's currency", + lower_bound: "Lower bound of the 95% prediction interval", + upper_bound: "Upper bound of the 95% prediction interval" + } +} + +SELECT + currency, + DATE(forecast_timestamp) AS forecast_date, + ROUND(forecast_value, 2) AS forecast_revenue, + ROUND(prediction_interval_lower_bound, 2) AS lower_bound, + ROUND(prediction_interval_upper_bound, 2) AS upper_bound +FROM + AI.FORECAST( + ( + SELECT order_date, currency, gross_revenue + FROM ${ref("fct_daily_revenue")} + -- Exclude the current, incomplete day: the simulator is still writing it, + -- and a partial day would drag the first forecast points down. + WHERE order_date < CURRENT_DATE() + ), + data_col => 'gross_revenue', + timestamp_col => 'order_date', + id_cols => ['currency'], + horizon => 14, + confidence_level => 0.95 + ) diff --git a/content/agenticdata/src/optional/forecast/prompts.md b/content/agenticdata/src/optional/forecast/prompts.md new file mode 100644 index 0000000..d6cbbc0 --- /dev/null +++ b/content/agenticdata/src/optional/forecast/prompts.md @@ -0,0 +1,28 @@ +# Canonical agy prompts — optional forecast lab + +Local mirror for the optional forecast lab, in the same format as +`content/agenticdata/src/prompts.md`. When the lab is wired into TUTORIAL.md, +merge this section into that file and repoint the lab's prompts.md link. + +## Lab 8 — the forecast mart + +Run inside `~/bootkon/content/agenticdata/src/dataform`: + +``` +/goal Create definitions/fct_revenue_forecast.sqlx: a Dataform table in schema +"cymbal_gold" carrying exactly one tag, "forecast" (not silver, not gold). It +materializes a 14-day daily revenue forecast per currency from +${ref("fct_daily_revenue")} (columns order_date DATE, currency STRING, +gross_revenue NUMERIC), excluding the current incomplete day +(order_date < CURRENT_DATE()). Use BigQuery's GA table-valued function +AI.FORECAST with its exact named arguments: the SELECT as first argument, then +data_col => 'gross_revenue', timestamp_col => 'order_date', +id_cols => ['currency'], horizon => 14, confidence_level => 0.95. From its +output keep currency, forecast_timestamp cast to DATE as forecast_date, and +forecast_value, prediction_interval_lower_bound, +prediction_interval_upper_bound rounded to 2 decimals as forecast_revenue, +lower_bound, upper_bound. Add a table description and column descriptions, +then run `dataform compile` and fix any errors until it compiles cleanly. +``` + +Fallback: `cp content/agenticdata/src/optional/forecast/fct_revenue_forecast.sqlx content/agenticdata/src/dataform/definitions/` diff --git a/content/agenticdata/src/optional/managed_dataform/prompts.md b/content/agenticdata/src/optional/managed_dataform/prompts.md new file mode 100644 index 0000000..de65f10 --- /dev/null +++ b/content/agenticdata/src/optional/managed_dataform/prompts.md @@ -0,0 +1,28 @@ +# Canonical agy prompts — optional managed Dataform lab + +Local mirror for the optional managed Dataform lab, in the same format as +`content/agenticdata/src/prompts.md`. When the lab is wired into TUTORIAL.md, +merge this section into that file. Both prompts are explain-only (no `/goal`, +nothing authored), so no restore command is needed. + +## Lab 7 — managed Dataform, explained + +Plain question. Ask while the scheduler waits for its first quarter-hour tick: + +``` +Explain the managed Dataform model like I run data platforms for a living: what exactly is frozen in a compilation result, why does re-executing the same compilation result still pick up new bronze rows, and when would I schedule release compilations more often than daily? +``` + +Fallback: concept question — the answer is in +https://docs.cloud.google.com/dataform/docs/release-configurations and +https://docs.cloud.google.com/dataform/docs/workflow-configurations. + +## Lab 7 challenge — policy tags that survive rebuilds + +``` +stg_customers is rebuilt by managed Dataform every 15 minutes. Explain why a BigQuery policy tag attached by hand to its email column disappears on rebuild, and show how to declare it in the SQLX config with bigqueryPolicyTags so every rebuild reapplies it. +``` + +Fallback: https://docs.cloud.google.com/dataform/docs/policy-tags (the +`bigqueryPolicyTags` column setting; the full tag path comes from the +`cymbal-governance` taxonomy created in Lab 4). diff --git a/content/agenticdata/src/optional/managed_dataform/push_to_dataform.sh b/content/agenticdata/src/optional/managed_dataform/push_to_dataform.sh new file mode 100755 index 0000000..e7cd257 --- /dev/null +++ b/content/agenticdata/src/optional/managed_dataform/push_to_dataform.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# +# push_to_dataform.sh -- beam the local Dataform CLI project into a managed +# Dataform development workspace, using nothing but the Dataform REST API +# and curl. +# +# Managed Dataform has no gcloud commands: its two surfaces are the console +# and the Dataform API. Every button in the console maps to one of the calls +# below -- this script is the console, minus the mouse. +# +# Usage: +# push_to_dataform.sh [REPOSITORY_ID] [WORKSPACE_ID] +# (defaults: cymbal-dataform cymbal-dev; project/region come from +# $PROJECT_ID and $REGION, exported by '. bk') +# +# The six calls, in order: +# 1. GET workspaces/{ws} fail fast if the workspace is missing +# 2. POST workspaces/{ws}:makeDirectory mkdir, one call per directory +# 3. POST workspaces/{ws}:writeFile upload, one call per file (base64 body) +# 4. POST workspaces/{ws}:commit one git commit inside the workspace +# 5. POST workspaces/{ws}:push workspace branch -> default branch (main) +# 6. POST workspaces/{ws}:installNpmPackages best effort; a no-op for this +# project (workflow_settings.yaml replaces package.json in Dataform core 3) +# +# What stays home: .df-credentials.json (your LOCAL credentials pointer -- +# managed Dataform brings its own identity, and credentials never belong in +# a repository), AGENTS.md (agy's brief, not pipeline code), dotfiles, and +# node_modules. + +set -euo pipefail + +REPO_ID="${1:-cymbal-dataform}" +WORKSPACE_ID="${2:-cymbal-dev}" +: "${PROJECT_ID:?PROJECT_ID is not set. Run '. bk' or open a new terminal.}" +LOCATION="${REGION:-us-central1}" +SRC_DIR="$HOME/bootkon/content/agenticdata/src/dataform" + +WS_PATH="projects/${PROJECT_ID}/locations/${LOCATION}/repositories/${REPO_ID}/workspaces/${WORKSPACE_ID}" +API="https://dataform.googleapis.com/v1" +TOKEN="$(gcloud auth print-access-token)" + +# Authenticated JSON call; on failure, print the API's error body to stderr. +api() { # api METHOD PATH [JSON_BODY] + local method="$1" path="$2" body="${3:-}" out + if out=$(curl --silent --show-error --fail-with-body -X "$method" \ + -H "Authorization: Bearer ${TOKEN}" \ + -H "Content-Type: application/json" \ + ${body:+--data "$body"} \ + "${API}/${path}" 2>&1); then + printf '%s' "$out" + else + printf '%s\n' "$out" >&2 + return 1 + fi +} + +# --- 1. The workspace must exist (you created it in the console) ------------ +echo "Checking workspace ${WS_PATH} ..." +if ! api GET "${WS_PATH}" > /dev/null; then + echo "" + echo "Workspace not found. Create the repository '${REPO_ID}' and the" + echo "development workspace '${WORKSPACE_ID}' in the console first (see the" + echo "lab), or pass your own names:" + echo " push_to_dataform.sh " + exit 1 +fi + +cd "${SRC_DIR}" + +# Everything except credentials, agy's brief, dotfiles and node_modules. +# NUL-delimited so a hand-made "my model.sqlx" (space and all) survives intact. +mapfile -d '' -t FILES < <(find . -type f \ + ! -name '.df-credentials.json' \ + ! -name 'AGENTS.md' \ + ! -path '*/node_modules/*' \ + ! -path '*/.*' \ + -print0 | sort -z) + +# --- 2. Recreate the directory tree ----------------------------------------- +mapfile -d '' -t DIRS < <(find . -mindepth 1 -type d \ + ! -path '*/.*' ! -path '*/node_modules*' -print0 | sort -z) +for dir in "${DIRS[@]}"; do + dir="${dir#./}" + echo "makeDirectory ${dir}/" + api POST "${WS_PATH}:makeDirectory" \ + "$(jq -n --arg path "$dir" '{path: $path}')" > /dev/null 2>&1 \ + || echo " (already exists -- fine)" +done + +# --- 3. Upload every file ---------------------------------------------------- +# JSON cannot carry raw bytes, so writeFile takes the contents base64-encoded. +count=0 +for f in "${FILES[@]}"; do + f="${f#./}" + echo "writeFile ${f}" + api POST "${WS_PATH}:writeFile" \ + "$(jq -n --arg path "$f" --arg contents "$(base64 -w0 "$f")" \ + '{path: $path, contents: $contents}')" > /dev/null + count=$((count + 1)) +done + +# --- 4. One git commit for the lot ------------------------------------------- +AUTHOR_EMAIL="$(gcloud config get-value account 2>/dev/null || true)" +COMMIT_BODY="$(jq -n \ + --arg name "${MY_NAME:-Bootkon participant}" \ + --arg email "${AUTHOR_EMAIL:-participant@example.com}" \ + --arg msg "Import the agy-authored medallion project (Lab 3, ${count} files)" \ + '{author: {name: $name, emailAddress: $email}, commitMessage: $msg}')" +# Committing an unchanged workspace is an API error, not a no-op -- but so is +# a real failure (say, an IAM grant still propagating). Tell them apart. +if err=$(api POST "${WS_PATH}:commit" "${COMMIT_BODY}" 2>&1 >/dev/null); then + echo "commit ${count} files" +elif grep -qiE 'no changes|nothing to commit|up to date' <<<"${err}"; then + echo "commit nothing new to commit (files unchanged) -- continuing" +else + printf '%s\n' "${err}" >&2 + exit 1 +fi + +# --- 5. Push the workspace branch to the repository's default branch --------- +api POST "${WS_PATH}:push" '{}' > /dev/null +echo "push workspace -> default branch (main)" + +# --- 6. Install npm packages (best effort) ------------------------------------ +# With Dataform core 3, workflow_settings.yaml replaces package.json, so this +# project has nothing to install -- but if you ever add dependency packages, +# this is the call behind the console's "Install packages" button. +if api POST "${WS_PATH}:installNpmPackages" > /dev/null 2>&1; then + echo "installNpm done" +else + echo "installNpm nothing to install (no package.json) -- fine" +fi + +echo "" +echo "Done. Refresh the '${WORKSPACE_ID}' workspace in the console -- your" +echo "project is there, committed, and already pushed to main." diff --git a/content/agenticdata/src/optional/sharing/listing_description.md b/content/agenticdata/src/optional/sharing/listing_description.md new file mode 100644 index 0000000..6776177 --- /dev/null +++ b/content/agenticdata/src/optional/sharing/listing_description.md @@ -0,0 +1,41 @@ +# Cymbal Gold — Commerce Analytics + +Curated, analysis-ready marts from Cymbal's order platform. This is the same +gold layer Cymbal's own analysts and AI agents use: raw operational data is +replicated live into BigQuery, cleaned and tested in the silver layer, and +aggregated here — no raw PII, no operational tables, no CSV exports. + +## What's inside + +| Table | Grain | Use it for | +|---|---|---| +| `fct_daily_revenue` | one row per day and currency | Revenue and volume trends: `orders`, `units`, `gross_revenue` by `order_date` and `currency`. | +| `dim_customer_360` | one row per deduplicated customer | Customer analytics: `lifetime_orders`, `lifetime_value`, `avg_order_value`, first and last order dates, `country`. | +| `fct_product_performance` | one row per product | Product and category analytics: `units_sold`, `gross_revenue`, `gross_margin` per `sku`, `name`, `category`. | + +## Freshness + +The source of truth is Cymbal's production PostgreSQL order platform, +replicated continuously into BigQuery via Datastream change data capture. +These marts are rebuilt on top of that live replica by Dataform +transformation runs; every row reflects the state of the platform as of the +most recent run. + +## Definitions + +- **Gross revenue** is `SUM(qty * unit_price)` and excludes cancelled orders. +- **Lifetime value (LTV)** is the gross revenue of a customer's non-cancelled + orders. +- `fct_product_performance` additionally excludes returned orders — margin is + only counted on goods that stayed sold. +- In `fct_daily_revenue`, amounts are reported **per currency** (ISO 4217, + normalized in silver) — never sum across currencies without converting + first. `lifetime_value` and the `fct_product_performance` amounts are + summed across order currencies without conversion; treat them as activity + indicators, not accounting figures. +- Customers are **deduplicated** by lowercased email; counts here may be lower + than in the operational system. + +## Contact + +Cymbal data platform team — data-platform@cymbal.example diff --git a/content/agenticdata/src/optional/sharing/prompts.md b/content/agenticdata/src/optional/sharing/prompts.md new file mode 100644 index 0000000..c8ad391 --- /dev/null +++ b/content/agenticdata/src/optional/sharing/prompts.md @@ -0,0 +1,15 @@ +# Canonical agy prompts — optional sharing lab + +Local mirror for the optional BigQuery sharing lab, in the same format as +`content/agenticdata/src/prompts.md`. When the lab is wired into TUTORIAL.md, +merge this section into that file and repoint the lab's prompts.md link. + +## Lab 9 — listing description (the data contract) + +``` +agy -p "Read the table schemas: bq show --schema :cymbal_gold.fct_daily_revenue ; bq show --schema :cymbal_gold.dim_customer_360 ; bq show --schema :cymbal_gold.fct_product_performance. Then write a listing description for sharing these three tables with Cymbal's partners — a data contract as prose, in markdown: one paragraph on what the dataset is, a short section per table (grain, key columns, what questions it answers), a freshness section (the tables are fed by live Datastream CDC from the production Postgres and rebuilt by Dataform runs), and a definitions section (gross revenue is SUM(qty * unit_price) excluding cancelled orders; lifetime value is the gross revenue of a customer's non-cancelled orders; in fct_daily_revenue amounts are per currency and must never be summed across currencies; lifetime_value and the fct_product_performance amounts sum across order currencies without conversion, so present them as activity indicators, not accounting figures). Output only the markdown." +``` + +Fallback: `cat content/agenticdata/src/optional/sharing/listing_description.md` +(a known-good contract ships with the repository — copy it into the listing's +Documentation field directly).