Repository navigation
feat(lanes): private, aggregatable store for lane transcripts #7676
Description
Refs: #6958
Context
The claude.dev post "Building effective agent automations" (2026-10-08, https://claude.dev/blog/building-effective-agent-automations/) recommends reading each run's transcript to find what went wrong. This repo's lanes publish telemetry but keep no transcript store, so runs cannot be pulled and compared to find errors and bottlenecks or to measure improvement.
Decision
- Q25, C18: the store is swappable. CI always emits an encrypted, gzipped artifact kept 7 days; a puller copies it to a sink set by one config value. The laptop is the first sink; others (object storage, another machine) are a config change.
- Q28, C23: transcripts stay private and easy to aggregate. Published per-run fields go to the telemetry issues below, not here.
- Q33 is open for planning: where the encryption key lives, and the puller's host, schedule and repo.
Acceptance criteria
- Every lane run uploads its transcript, encrypted and gzipped, as a CI artifact with 7-day retention.
- A puller copies artifacts to the sink named by one config value; changing the sink changes only that value. The laptop is the first sink.
- Sink retention: full transcripts 30 days, metrics records 1 year, a 5 GB cap, and an alert if no pull has happened in 5 days.
- Stored transcripts are queryable across runs by lane, model, outcome and date.
- No transcript, key or decrypted content lands in a tracked file or a public artifact.
- The published per-run fields (lane, outcome, duration, turns, models, tokens, estimated cost, run link) live in Utilization tracking: weekly use per subscription and credit spend per org against the near-full target #6973, Outcome metrics that steer allocation toward work that ships and sticks #6974 and Lane telemetry: work-loop #7396 and are out of scope here.
Related
- Utilization tracking: weekly use per subscription and credit spend per org against the near-full target #6973: utilization per subscription and org; consumes published fields, not transcripts.
- Outcome metrics that steer allocation toward work that ships and sticks #6974: outcome metrics that steer allocation; reads published fields, not transcripts.
- Lane telemetry: work-loop #7396: work-loop lane telemetry home; holds the published per-run record.
Out of scope
- The published per-run fields and their format.
Activity
github-actions commentedon Oct 16, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: intent
Blocked by: none
#7676 (comment)
I understand this as: CI uploads each lane transcript as an encrypted, gzipped, 7-day artifact. It encrypts with a public key held as a repo variable. A daily laptop task pulls the artifacts, with the private key in Key Vault read through
vault-exec.The decision settles the key location and the puller host. One question is still open:
Still needed from a person: create the key pair, store the private key in Key Vault, and set the public-key repo variable. The laptop scheduling and the retention and alert checks also need a person.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 18, 2026
Decision recorded.
Question: Every lane run would upload its transcript, encrypted, as a 7-day artifact, and a puller copies it to one sink. This is a public repository, so artifacts must be encrypted before upload. Q33 asks where the decryption key lives and where the puller runs. Options: (a) public key in CI, private key on the laptop, puller a scheduled task on the laptop; (b) key in Azure Key Vault, puller on a fleet host; (c) defer the store until #6973/#6974 show it is needed.
Owner's answer: asymmetric encryption, with the private key in Key Vault, read through
vault-exec, and the puller as a daily task on the laptop. The estate already resolves secrets throughvault-execfrom Key Vault, so a key kept only on the laptop would be a second, unbacked-up way of holding secrets. The public key is not secret, so CI can hold it as a repo variable. (c) does not fit: #6973 and #6974 read the published fields, not transcripts (#7676 "Related"). This is a security decision, so the owner must answer it. Basis: ~/.config/estate/ESTATE.md:34-36; #7676 body.The owner accepted this recommendation as item D3 of a 21-decision batch in an attended backlog-shaping session on 2026-10-18. The batch question: "Accept all 21 batch 7 decisions, with my change to D13 (#6008 goes to the lane, but you merge its PR because it edits AGENTS.md)? I post a 'Decision recorded.' comment on each issue." The owner answered: "Accept, with my D13 change (Recommended)".
Posted by AI from the owner's answer in that session.
kyle-sexton commentedon Oct 25, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.melodic-automation-lanes commentedon Oct 25, 2026
The triage verdict left this issue at
status: needs-decision:15 remaining items
kyle-sexton commentedon Oct 28, 2026
Re-triage. This is not an owner stop. The owner decided the security question, where the key lives and where the puller runs, as D3 on 2026-10-18 (comment). The criteria are recorded in this comment.
The one gap that record left, which repository holds the puller, is decided on the facts: this repository. The artifacts and the one sink config value live here, so the puller needs no cross-repo access. The laptop task calls the puller from this repository's checkout.
Cost: artifacts on a public repository with 7-day retention fall within the free Actions storage for public repositories, so nothing here can bill.
Acceptance criteria (unchanged from the 2026-10-26 record):
Builder: the Automation Lanes session builds this from the owner's machine, because the upload step edits
.github/workflows/**, which the lanes App cannot push. Before the build: create the key pair, put the private key in Key Vault, and set the public key as the repo variable CI reads.agent-readystays off.Checks a person runs: the private key is read through
vault-exec; the puller runs as a daily task on the laptop; sink retention (30 days for transcripts, 1 year for metrics, a 5 GB cap, an alert after 5 days with no pull); stored transcripts are queryable by lane, model, outcome and date.fix(lanes): fail a capped or wrong-key transcript pull
kyle-sexton commentedon Nov 2, 2026
Status: what landed and what is left.
Status check by the issue-triage session against main at
0ed6022913, on the criteria in the 2026-10-28 re-triage comment.Met:
.github/actions/upload-lane-transcript/action.yml:35-51seals and uploads only the sealed file (feat(lanes): upload each model leg's transcript as a sealed 7-day artifact #10749,79acd8c710);:26-33scans the plaintext for credentials before the seal step (fix(lanes): scan every lane and eval upload for credentials before it leaves #11806,11da9482ec);seal-transcript.test.mjschecks the upload is not the bare gzip.Remaining:
upload-lane-transcript(gzip, seal toLANE_TRANSCRIPT_PUBLIC_KEY,retention-days: 7) is wired into intake-triage, pr-verify, pr-unblock, pr-review-main, maintenance-audit-lane-runs and the read runner's reuse summary. Not covered, per feat(lanes): upload each model leg's transcript as a sealed 7-day artifact #10749: the read runner's skill job, the write runner's skill job (pr-run-activity-write.ymlhas no call), the mutation audit, and the review lanes, which run in melodic-software/ci-workflows. No run withLANE_TRANSCRIPT_PUBLIC_KEYset is recorded.upload-lane-transcript/README.mddescribes it.vault-exec; the puller runs as a daily task on the laptop; sink retention (30 days for transcripts, 1 year for metrics, a 5 GB cap, an alert after 5 days with no pull); stored transcripts are queryable by lane, model, outcome and date.Route: the remaining uploads edit
.github/workflowsfiles or a ci-workflows reusable, so they go to a builder session. Creating the key pair, the Key Vault secret and the repository variable is a person's step, and the puller's laptop task and retention checks are host runs.kyle-sexton commentedon Nov 2, 2026
Decision (Lane 5, under the standing rule: least friction that keeps the boundary): the three jobs exempted from transcript upload stay exempt. Each is an accepted gap against "every lane run".
The jobs are
pr-run-activity-read.yml#run,pr-run-activity-write.yml#runandpost-merge-test-mutation.yml#audit. #10749 exempted them because code the model can rewrite runs before a later step that holds a token (id-token: writeon the write runner, the runtime token in the mutation audit). The reasons are recorded at.github/actions/resolve-config/callers.test.mjs:7453-7464. Uploading from those jobs would run an upload step after code that may have been tampered with, while it holds that token, which loosens a boundary. Keeping the exemption changes nothing.If transcripts from these lanes prove necessary, the route is a separate clean job that receives an already sealed record. That is a new design.
The puller is #12428. The model legs inside ci-workflows reusables are ci-workflows#964.
This was generated by AI.
fix(lanes): recover from a metrics append cut short
kyle-sexton commentedon Nov 2, 2026
Status: live on melo-lap-001, first real pulls verified. This stays open only for ci-workflows#964.
kv-melo-github-prod/lane-transcript-private-key. The key id derived from the vault's private key matchesLANE_TRANSCRIPT_PUBLIC_KEY(b9a83392502f6984).origin/mainexport. A failed run raises a desktop toast throughunit-failure-toast@.systemctl --user start lane-transcript-pull.service): run 1 stored 65 transcripts, then failed on oneghcall. Run 2 succeeded:pulled 153 transcript(s); skipped {"pulled":153,"known":67,...,"too_large":0,"bad_zip":0,"sealed_mismatch":0}. Total: 218 transcripts, 6.0 MB, none sealed to another key.query --since 2026-11-02 --outcome successreturns rows by lane, model, outcome and date.UNVERIFIED: that a toast visibly appears on the Windows desktop (exit 0 only); a midnight timer firing, due 2026-11-03 00:00 EST.
This was generated by AI.
kyle-sexton commentedon Nov 3, 2026
ci-workflows#969 merged (328b0c15), so the reusable review and intake legs can now seal and upload transcripts once a caller passes
lane-transcript-public-key. Remaining before this closes: the runner policy allowlist and hosted review callers in melodic-software/standards (PR in progress), theintake-triage.ymlcaller here, then a real sealed reusable-leg artifact through the puller. Lane 5 holds this.fix(runner-policy): admit the reusables' lane-transcript-public-key i…
kyle-sexton commentedon Nov 3, 2026
Done. #12579 merged (71d28c0), so the reusable review and intake callers now pass the transcript key.
Proof: one run of the
lane-transcript-pullunit at 2026-11-03T01:48Z pulled 246 transcripts withsealed_mismatch0 andfetch_failed0. The sink holds 464 rows and 14.5 MB. The reusable-leg runs 41559009992 (code-review-subscription) and 41559010015 (security-review-subscription) are stored and returned bypull.mjs querywith their lane, model, outcome and date.Left elsewhere: hardening of the job that follows main is #12534. The first scheduled timer run is due at midnight EST; Lane 5 checks it and reports any failure here.
kyle-sexton commentedon Nov 3, 2026
Scheduled pull failed: the 2026-11-03T05:00:10Z timer run exited 1 at 05:00:45Z with
pull failed: artifact-list: gh exited 1(journalctl, lane-transcript-pull.service). A manual run at 14:25Z failed the same way.Cause: GitHub returned HTTP 500 with an empty body on
repos/melodic-software/claude-code-plugins/actions/artifacts?per_page=100&page=1(request id E139:12AAB4:2788507:820BC81:6AE9EED9). The 500s were intermittent: at 14:25Z, per_page 100 and 50 failed while 75, 30, 10 and 1 succeeded, and a minute later 8 of 8 calls at per_page 100 returned 200. The repo holds 15,555 artifacts.ghRetriedmakes 3 attempts with 2 s and 4 s waits (scripts/lane-transcripts/pull.mjs:45-46,:88-99), so a burst of 500s longer than about 6 s fails the whole pull. The rate limit was not involved (core 4,762 of 5,000 left).A rerun started at 14:29Z is in progress; its result follows here. The unit-failure toast fired as designed (OnFailure).
kyle-sexton commentedon Nov 3, 2026
Rerun result: exited 0 at 14:52:46Z after 25 min 25 s wall clock (2 min 35 s CPU). Pulled 1,125 transcripts, 442 already known, fetch_failed 0, sealed_mismatch 0, sink 52.5 MB, nothing pruned. The 05:00Z failure was GitHub's transient 500s, not the puller. The next scheduled run is tonight; the 3-attempt, 6 s retry window is short against bursts like this one, and a longer backoff on 5xx would have saved this run.