[CamSoper]

Proposed changes

Reworks the Monday #docs-ops digest so Cam can read it in under a minute and know what needs a human this week. It's now one Slack message at about 2.3k characters. Today's post was two messages totaling about 5.1k.

Pipeline: collect → rank → render → post

• scripts/weekly-digest/digest.py (collect, deterministic):
  • Runs scripts/review-v3/sla-sweep.py --dry-run. That's the sweep's own no-mutation contract: no comments, labels, closes, Slack messages, or state writes. The digest ships the resulting verdicts, so "Over review SLA, by team" and "Abandoned" render whether or not the scheduled sweep is on. REVIEW_V3_SLA is untouched.
  • With the ledger bucket resolved, the dry run reads the sweep's per-PR S3 state, so a PR it already warned shows its real remaining notice.
  • New collectors:
    • week-over-week PR and issue throughput;
    • failed scheduled and master runs per workflow, with a consecutive-failure streak that separates broken from flaky;
    • lane switches (DIGEST_VARS), passed from repo variables because GITHUB_TOKEN can't read them. Each switch uses the same on/off semantics as its owning workflow (== '1' vs != '0').
  • The old 24h CI-health line is gone. It said "HEALTHY 100%" today while Audit Logs had failed 4 of 4 scheduled runs.
• scripts/weekly-digest/render.py (render, deterministic):
  • Each PR appears once.
  • Ready to merge is the merge gate's own verdict: digest.py reads each non-draft PR's Sentinel check (preview "would be: success", or an enforcing "success"). Human PRs are listed as links; bot PRs are a count.
  • Awaiting review: one GitHub search link per routed team (from review-routing.yml) for review:no-blockers PRs with that team's review still requested. Each count comes from the same query the link opens. GitHub drops a team's request once any member reviews, so this means "requested, not yet reviewed".
  • Blog and marketing PRs are listed like any other (the earlier one-line blog summary was dropped at Cam's request).
  • SLA batches compress ("8 PRs at 5bd"), and red-CI PRs are named first instead of hiding behind "+N more".
  • A workflow that failed its last 3+ runs is promoted to the top list; flakes collapse to one count.
  • The throughput line only appears on a week-over-week shift of 25% or more.
  • Degradations stay loud: SLA verdicts, review telemetry, or run history unavailable each get a ⚠️ line.
• scripts/weekly-digest/rank.py + rank-prompt.md (the only model call): orders the top five "Needs a human" candidates and adds a 12-word "why" for each. Output is validated: unknown refs and duplicates are dropped. Any failure (no key, API error, max_tokens, bad JSON) falls back to a fixed priority order and says so in the message, so an API hiccup no longer costs the team the week's digest. Replaces synthesis-prompt.md.
• .github/workflows/weekly-digest.yml:
  • collect → rank → render → post;
  • actions: read for run history;
  • DIGEST_VARS;
  • dry_run still previews without posting.

sla-sweep.py, one minimal and tested change (called out per the lane rules): new pure projected_close_days(config, idle_days, warn_age_days), plus closes_in_days / undecided_count fields on author-time actions in the run record. That keeps "closes in N days" in the evaluator instead of copying its warn/close policy into the digest. Mutation paths and behavior are unchanged. Two new tests are in test_sla_sweep.py.

Tests: scripts/weekly-digest/test_digest.py has 31 tests: switch semantics, verdict shaping, run streaks and sorting, dedup, blog routing by role, batch compression, red-CI naming, ranking validation and fallback, mrkdwn escaping, the one-message size, and a regression test for each C&C review item. The suite is wired into scripts/test-review-pipeline.sh, and the review-pipeline-tests.yml path filter now includes scripts/weekly-digest/**. make test-review-pipeline passes locally.

After: rendered from today's real pulumi/docs data (Sep 28, head 6b62478)

Collected live with digest.py at 6b62478, dry run only (nothing posted to Slack). Caveats for this sample:

• This sandbox has no gh and blocks GraphQL and search/*, so a scratch read-only gh shim answered over repo-scoped REST, refusing any non-GET call. Throughput counts came from the repo's issues listing filtered locally instead of the search API. The shim isn't committed; CI uses real gh.
• There's no AWS here, so the review ledger wasn't read. That's why the ⚠️ Review ledger unreadable line shows. CI has credentials, and the line won't appear there unless the bucket really is unreachable.
• There's no API key here, so the top-five order came from running this head's rank-prompt.md through a Sonnet subagent.
• Only the two switch values the brief confirms are passed (REVIEW_V3_SLA unset, BLOG_REVIEW_COUNT=0). The "unknown" switches are this sandbox's gap; CI passes all seven.

📋 Docs weekly · Sep 28
73 open PRs (±0) · 56 issues (−11) vs last week

Needs a human

1. #21831 Reorganize Discovery & governance docs… — File a follow-up; merged with 8 blocking findings unanswered
2. #21911 Blog: Self-driving infrastructure with… — File a follow-up; merged with 4 blocking findings unanswered
3. Audit Logs - Update Events — Fix or disable; failed its last 4 runs
4. #20631 blog: GPU Kubernetes infrastructure… — Review now; 41bd on a 3bd SLA, blog → cnunciato
5. #20949 Add: Platform Engineering in the… — Review: 29bd on 3bd SLA, blog → cnunciato, docs-guild → tatcoo-pulumi

Ready to merge (2, Sentinel green): #21760 · #21953 · +4 bot PRs
Awaiting review (no blockers, team requested): docs-guild 13 · blog 6 · marketing 4 · tools 10

Over review SLA
• blog → cnunciato (SLA 3bd): #21039 26bd · #21083 25bd · #21149 23bd · #21172 22bd · +8 more
• docs-guild → tatcoo-pulumi (SLA 3bd): 8 PRs at 5bd: #21799 red CI · #21793 · #21794 · #21795 · +4 more
• tools → CamSoper (SLA 1bd): 1 PR at 4bd: #21828

Abandoned (author idle >14d)
• #21559 blog: Add post "Best Platform… — author idle 17d, 9 findings open; close or rescue it by hand

Keep or close? (never reviewed): #20579 60d · #21583 15d

Review loop (findings on 87 closed PRs): 47 fixed · 12 ignored at merge · also merged over findings: #21746
Failed runs (scheduled + master): 4 flaky (18 of 922 runs, 11 in Build and deploy testing)
Switches: off: SLA sweep, blog review index
Not listed: 19 in progress within SLA · 8 waiting on author · 13 drafts · 1 bot PRs

With the sweep on, the Abandoned rows read "closes in Nd" (or "closes at the next sweep"), and a *SLA sweep, last 7d*: N escalations · N warned · N closed · N waived line summarizes what the sweep actually did.

A Sentinel-green PR is listed once, as ready, even when the SLA sweep still calls it overdue: the gate accepts an approval from any review team, while the sweep waits on the routed team.

Before: today's actual post (excerpt, about 5.1k characters across 2 messages)

Ready to merge (32, all green + review:no-blockers) — click through:
#21931, #21929, #21927, #21925, #21920, #21919, #21889, #21828, #21810, #21801, #21798, #21797, #21796, #21795, #21794, #21793, #21790, #21787 (also blocked, confirm that's cleared), #21760, #21651, #21491, #21487, #21434, #21412, #21210, #21198, #21172, #21149, #21083, #21039, #20949, #20631.
…
In flight
25 in flight — @workprentice 7, @joeduffy 3, @jkodroff 2, @jeffmerrick 2, remaining 11 authors 1 each (PYoakum, sangharshcs, …).
…
CI health: HEALTHY, 100% success over the last 24 hours (0 failures / 50 runs).
v3 SLA ops: no SLA escalations, staleness warnings, or waives this week.
…
Oldest open (keep-or-kill) #11739 (1872 days), #11654 (1457 days), #11497 (937 days)

Those 32 "ready to merge" PRs were really waiting on approval, and 22 of them were past SLA. The sweep's verdicts now group them by team with days waiting and an owner.

Quality loop

Three subagent reviewers read the rendered output each round, with different lenses: a busy maintainer skimming on a phone, a skeptic hunting filler and redundancy, and an on-call reader asking "what would I miss?"

• Round 1: all three said "needs changes."
• Round 2: two said "converged" and the skeptic asked for small fixes, which went in.

What the rounds changed:

• A 4/4-failing workflow was ranked last; it's now promoted.
• The "last failure" link could point at the wrong run; runs are now sorted by time.
• An 8-link wall of identical waits is now compressed.
• "Rescue or let close" implied a close that can't happen while the sweep is off.
• Issue counts contradicted each other ("8 new" vs "19 opened"); the Issues line is gone.
• The review loop mixed PR counts and finding counts.
• A red-CI PR was hidden behind "+4 more"; it's now named first.
• The run denominator was inflated by 400+ fork-guard skipped runs.

Follow-up: C&C adversarial review

All five fixes and the polish landed in 0251e0fd, each with a regression test. 6b62478 closes two gaps it left: with SLA verdicts unavailable, the footer no longer says "N in progress within SLA", and rank.py's never-fatal path now has its own test. The dispositions are in the PR thread.

Related issues (optional)

None filed. Noticed along the way: Audit Logs - Update Events has failed every scheduled run this week (#21924).

https://claude.ai/code/session_01Gcx8uerksNXPswPTofF6ke
https://claude.ai/code/session_017Ea8K7TekJVG8KiLSXYf4d

🤖 Generated with Claude Code

---

Generated by Claude Code

[github-actions]

Sentinel — merge gate status (preview)

Warning

⚠️ Preview mode — this check is NOT blocking your merge

The Sentinel is in preview (report-only) mode: its check-run concludes neutral whatever the gates below say, so nothing here gates this PR. The rows are informational and safe to ignore for now. It would have concluded failure.

This will be enforced in the near future. Once it is, every row below has to be green before this PR can merge. The rows are the sign-offs a merge needs — some yours, some a reviewer's or a deploy's. Tell us in #docs if one looks wrong.

1 of 5 gates would block this merge once the Sentinel is enforced.

	Gate	What it needs
✅	G1 review-ran	Nothing to do: review current at dde18633b.
✅	G2 findings-answered	Nothing to do: every finding answered.
🔴	G3 right-approver	Needs approval from a member of any review team (pulumi/docs-blog-review, pulumi/docs-guild, pulumi/docs-marketing-review, pulumi/docs-tools) or a repository administrator — no qualifying human approval at this head (bot approvals never count).
➖	G4 infra-evidence	Doesn't apply to this PR: no changed path affects the deploy.
➖	G5 oversized-ack	Doesn't apply to this PR: not oversized.

Break glass: a member of a routing team (.github/review-routing.yml teams:) can apply review:waived, which is logged.

_{Evaluated at head dde18633b.}

[pulumi-bot]

Your site preview for commit dde1863 is ready! 🎉

http://www-testing-pulumi-docs-origin-pr-21952-dde18633.s3-website.us-west-2.amazonaws.com

[github-actions]

Reviewer's guide v5 — not for the author

Tip

Check the ⚠️ items, then approve. Your approval covers only those; links, shortcodes, metadata, and verified claims were checked by machine (receipts on the evidence page). Code samples are read, not compiled.

PR author: nothing here is yours. Your to-do list is the "Author action guide" comment.

Approval needed from: @pulumi/docs-tools — any member's approval satisfies the merge gate.

Note

What this PR changes:

• weekly-digest.yml: replaces the multi-step synthesis job with a digest → rank → render → post pipeline.
• digest.py: gathers more signal, including red-CI PRs and honest run denominators.
• rank.py + rank-prompt.md: new model-ranking step that replaces synthesis-prompt.md.
• render.py: new deterministic renderer for one scannable Slack message.
• sla-sweep.py: emits the SLA data the digest consumes, with new tests.
• test_digest.py: new coverage, registered in make test-review-pipeline and its CI workflow.

The blocking risk is a digest that silently misreports or fails to post, such as a broken workflow step or a mis-ranked SLA breach. The infra review, claim verification, and style lint all ran.

Review confidence:

Dimension	Level	Notes
mechanics	MEDIUM	Scripts were read but not run here; CI tests cover execution
facts	HIGH

⚠️ Check these before approving

No findings to check — but the editorial stances below still need a human eye.

Editorial stances introduced by this PR

Superlatives and comparisons the diff adds. Framing isn't fact-checkable, so confirm each is a stance the docs should take.

• L12 scripts/weekly-digest/rank-prompt.md — "1. Size of the breach relative to the SLA (5 business days on a 1-day SLA beats 5 on a 3-day SLA)." — comparison (found by regex)

✅ What you can rubber-stamp

• Facts: 3 factual claims checked — 3 verified clean.
• Mechanics: no internal links added.
• Style: 0 advisory suggestions left with the author; never blocking.

Pre-existing issues in touched files: 0 — details on the evidence page.

Full evidence: verification trail, investigation log, review history.

_{Review v5 · updated 2026-09-28T23:11:12Z · head commit dde1863}

---

For the reviewer: something look wrong? Comment @claude <your point> #update-review and the review re-checks with your input.

[github-actions]

Author action guide v5 — nothing blocks merge

Note

Nothing here needs an answer from you. A human reviewer still approves the merge.

This PR reworks the weekly digest workflow into a single ranked Slack message fed by the SLA sweep; the review checked the workflow, scripts, prompts, and their tests.

Full evidence: verification trail, investigation log, review history.

_{Review v5 · updated 2026-09-28T23:11:12Z · head commit dde1863}

---

Please don't edit or delete this comment; it's the review's record.

[CamSoper]

Status: the review came back with zero findings (review:no-blockers), CI is green, and the worklist is clean (review-worklist.py --require-clean exits 0). Sentinel preview has G1 and G2 green at 33ef65f95. The only open gate is G3, a docs-tools approval.

Correction: an earlier version of this comment said Sentinel G1 was stale-red. That was my mistake. I'd read the 19:28Z check run, which ran before the review posted. The 19:35Z run is correct, so there's nothing to fix.

---

Generated by Claude Code

[CamSoper]

Independent adversarial review of head 33ef65f9, from the C&C session. Nothing blocking on safety.

Checked and fine

• SLA call is read-only on every path. collect_sla hard-codes --dry-run, and every write/label/close/Slack/state call in sla-sweep.py is behind if not dry_run. Neither REVIEW_V3_SLA nor workflow_dispatch can change that.
• projected_close_days math matches the close rule.
• Switch semantics match each owning workflow's if:.
• Escaping. esc() neutralizes <!channel> and similar in titles and the model's text, and no ${{ }} touches PR data.
• Tests. 45 tests plus make test-review-pipeline pass.

Please fix before approval

1. Confirmed (medium). When SLA verdicts are unavailable, the fallback tells people to merge PRs that still need an approval. At render.py:200-207, verdicts == {} makes every green, non-blog, non-draft review:no-blockers PR a merge-now candidate ("green, no blockers, no required reviewer: merge it"). A synthetic render with sla.available=False filled all five "Needs a human" slots with that. That's exactly the day the digest must not mislead. Emit merge-now/keep-or-kill only when sla.available, or label those rows unverified.
2. Confirmed render (low-medium). render.py:191 prints "closes in Noned". While the sweep is on, a dry run can return a close action with no closes_in_days, for example when a PR crosses the threshold between the :53 sweep and the 14:00 digest, or when a real close failed. Map None / type == "close" to "closes at the next sweep".
3. Plausible. When the ledger bucket isn't found, the dry run uses an empty state dir, so an already-warned PR shows "closes in 7d" when it's really 1d. The workflow comment (about weekly-digest.yml:56) promises a "review ledger unreadable" flag, but that string doesn't exist anywhere. Add a ⚠️ line whenever bucket is None.
4. Plausible. rank.py:93 isn't really "never fatal": IncompleteRead, AttributeError and friends escape the except list and, under set -euo pipefail, fail the whole job. Catch Exception, or add || echo '{}' in the step.
5. Confirmed. render.py:348 renders an unknown sweep state as "off" (a local run without DIGEST_VARS, or unparsable JSON). That contradicts shape_switches' "must not dress up as off" contract. Render "unknown".
6. Nits.
   • The header says "vs last week" even when the deltas failed. The "this week" window is >=cutoff-date (about 7.6 days) against an exact 7, which biases the 25% shift check.
   • Zero candidates prints "Nothing this week." plus the "(model ranking unavailable)" footer.

Output polish (the sample meets the brief overall: one message, about 2.1k characters, phone-scannable):

• "Sweep is off" is said three times: the Switches line, "(the sweep is off: nobody was pinged)", and the Abandoned row. Say it once.
• Item 4 lists bare names ("tatcoo-pulumi, cnunciato") with no role. Give each a role, or use the team.
• The blog/marketing count includes blog: Add post "Best Platform Engineering Tools in 2026" #21559, which is also listed under Abandoned. Either exclude it or note it.

---

Generated by Claude Code

[CamSoper]

All fixed in 0251e0fd, with a regression test for each. 29 digest tests pass, and so does make test-review-pipeline.

1. Merge-it without verdicts. merge-now / keep-or-kill candidates are built only when sla.available. test_no_verdicts_never_says_merge_it reproduces your synthetic case.
2. "closes in Noned". When the sweep is on and there's no projection, the row reads "closes at the next sweep" (test_close_without_projection_reads_next_sweep).
3. Ledger missing. collect_sla records ledger: bool(bucket), and the renderer adds ":warning: Review ledger unreadable …" whenever it's false. The workflow comment now matches the real string.
4. rank.py. It catches Exception, and the workflow step also does || echo '{}' > /tmp/ranking.json.
5. Unknown shown as off. The sweep state is tri-state through the renderer. When it's unknown, abandoned rows say "closes on its own only if the SLA sweep is on", and Switches shows unknown: SLA sweep.
6. Nits. "vs last week" only appears when a delta exists. Throughput now compares two equal whole-day windows (cutoff..yesterday vs. the 7 days before). There's no fallback footer when there's nothing to rank.

Polish:

• "Sweep is off" now appears once, on the Switches line.
• The prompt keeps owners paired with their team; item 4 now reads "docs-guild → tatcoo-pulumi, blog → cnunciato".
• The blog count excludes PRs listed elsewhere (27 → 26 today, since blog: Add post "Best Platform Engineering Tools in 2026" #21559 is under Abandoned).

---

Generated by Claude Code

[CamSoper]

@claude #update-review

0251e0fd addresses the C&C adversarial review (the comment above has the per-item list). Fixes:

• render.py no longer builds merge-now / keep-or-kill candidates without SLA verdicts.
• A close with no projection reads "closes at the next sweep".
• A ⚠️ line flags a missing review ledger.
• The SLA sweep state is tri-state, so unknown never renders as off.
• rank.py catches every exception, and the workflow step falls back to {}.
• Throughput compares two equal 7-day windows.

Each one has a new test in test_digest.py. No change to sla-sweep.py since the last review.

---

Generated by Claude Code

[CamSoper]

Dispositions for the adversarial review of 33ef65f9, as of head 6b62478.

Heads-up for C&C: two sessions worked this review in parallel. The original author session wasn't stopped, and it pushed 0251e0fd while I was mid-fix. I dropped my duplicate rewrite rather than stack a second version of the same fixes. 6b62478 adds only what 0251e0fd missed. Nothing was force-pushed.

#	Item	Fixed in	Test
1	Merge-it without SLA verdicts	0251e0fd (no merge-now/keep-or-kill candidates without verdicts) + 6b62478 (footer said "26 in progress within SLA" in the same degraded case; now "other open", and the ⚠️ names the ready-to-merge list as missing)	test_no_verdicts_never_says_merge_it, test_sla_unavailable_never_claims_within_sla
2	"closes in Noned"	0251e0fd: "closes at the next sweep"	test_close_without_projection_reads_next_sweep
3	Ledger unreadable not flagged	0251e0fd: ⚠️ line whenever the bucket isn't resolved; workflow comment now matches the string	test_missing_ledger_is_flagged
4	rank.py not really never-fatal	0251e0fd: catches Exception, plus || echo '{}' in the step. 6b62478 adds the missing test (fails on 33ef65f)	test_rank_main_is_never_fatal
5	Unknown sweep state shown as off	0251e0fd: tri-state; unknown never reads as off	covered in test_sweep_off_said_once and the unknown-state render
nit	"vs last week" without deltas; unequal windows; footer with nothing ranked	0251e0fd	test_no_candidates_no_fallback_footer_and_no_delta_suffix_without_data
polish	Sweep-off said 3x / bare names / blog double count	0251e0fd: said once (Switches line); owners stay paired with their team; blog count excludes listed PRs (26, not 27)	test_sweep_off_said_once, test_blog_count_excludes_prs_listed_elsewhere

Every new test fails on the code it guards against and passes at 6b62478. The digest suite has 31 tests, and make test-review-pipeline passes.

The PR body now has a dry-run render from live data at 6b62478 (nothing posted to Slack), plus the degraded case (SLA verdicts and model ranking both unavailable) rendered from the same data. On that data, the old head told people to merge #21760, which is 6bd into a 1bd tools review. The new head doesn't.

---

Generated by Claude Code

[CamSoper]

@claude pushed 6b62478 addressing the adversarial review #update-review

6b62478 is a small follow-up to 0251e0fd. render.py no longer claims "in progress within SLA" when SLA verdicts are unavailable, and the unavailable-SLA warning now lists ready-to-merge among the missing sections. test_digest.py adds two regression tests (the degraded footer, and rank.py's never-fatal path). No workflow or sla-sweep.py changes.

---

Generated by Claude Code

[CamSoper]

@claude #update-review

6b62478e is a small follow-up to the C&C review. When SLA verdicts are missing, the footer now says "other open" instead of "in progress within SLA", and the missing-verdicts warning also names the ready-to-merge list. It also adds a regression test that rank.py prints {} and exits 0 on an unexpected exception. CI is green.

---

Generated by Claude Code

[CamSoper]

I verified head 6b62478e against the adversarial review. Nothing blocking remains. All five findings, the nits and the polish items are fixed. The two concurrent sessions didn't clobber each other: 6b62478 sits cleanly on 0251e0fd, with no duplicated logic and nothing overwritten. There are 31 digest tests, 23 test_sla_sweep tests and make test-review-pipeline, and all of them pass. The read-only guarantee still holds, since --dry-run is hard-coded and every mutating call in sla-sweep.py is gated. I rendered six synthetic cases: normal, SLA unavailable, unknown sweep, missing ledger, sweep off and empty. All of them read cleanly and none give misleading advice.

Optional nits. Fix these if you're pushing anyway; none of them block.

1. The ledger ⚠️ (render.py:456) still says "may close sooner than shown" when the sweep is off or unknown, but those states show no close timing. Gate it on enabled == "on" or reword it.
2. The SLA-unavailable ⚠️ (:355) blames "the sweep dry-run failed", but a review-routing.yml load failure (digest.py:329) produces the same state. It also doesn't mention that the "Keep or close?" list is suppressed too.
3. The throughput docstring (digest.py:552) says [start, end), but GitHub's a..b is inclusive on both ends. The code is right; only the comment is wrong.
4. In the unknown sweep state (local runs only), every abandoned row repeats "closes on its own only if the SLA sweep is on".
5. An empty blog/marketing queue still prints "0 open · 0 over SLA". Drop the line when both counts are zero.

---

Generated by Claude Code

[CamSoper]

@claude #update-review

c8d44977 makes three changes Cam asked for:

• Blog/marketing summary line removed. Those PRs are now listed like any other.
• "Ready to merge" now uses the Sentinel's own verdict. digest.py reads each non-draft PR's Sentinel check. Human PRs are listed and bot PRs are counted. This replaces the old merge-now candidate kind.
• Per-team "Awaiting review" search links. Each count comes from the same query the link opens.

A PR the Sentinel marks ready is listed once, as ready, even if the SLA sweep still has it overdue. test_digest.py has 35 tests; they pass, and so does CI.

---

Generated by Claude Code

[CamSoper]

@claude #update-review

dde18633 adds a thread reply, at Cam's request. Every list the digest caps now sends its full version to one reply under the message, and the cut reads "+N more in thread". "Needs a human" says "top 5 of N" and continues from 6 in the thread.

post-to-slack.py --thread posts the reply under the first message, using the ts and channel ID from chat.postMessage. A failed reply only warns; the digest has already posted. When nothing is cut, no reply is posted. There are 41 tests in test_digest.py, including poster tests with post_chunk stubbed out. CI is green.

---

Generated by Claude Code