Skip to content

Reconcile invocation usage and attempt gaps - #1155

Draft
aviggiano wants to merge 2 commits into
mainfrom
codex/1138-ledger-integrity
Draft

aviggiano wants to merge 2 commits into
mainfrom
codex/1138-ledger-integrity

Conversation

@aviggiano

Copy link
Copy Markdown
Collaborator

Usage and attempt reporting now reconciles both canonical ledgers with durable scheduler and adapter evidence. A run cannot report complete usage or attempt history when a known invocation is missing.

What changed

  • Persist an idempotent execution-reconciliation document with canonical counts, runner counts, identified gaps, and explicit completeness flags.
  • Count the union of invocation identities from both canonical ledgers, so disjoint gaps cannot be hidden by taking the larger ledger count.
  • Retain attempt identities even when a task row is unavailable, using the stable attempt-to-node convention shared by the scheduler and ledgers.
  • Include acquired NodeCancelled attempts in terminal attempt reconstruction with a distinct canceled outcome.
  • Rebuild accounting safely when a crash leaves run.json at an authenticated prefix of the usage ledger.
  • Clear workflow-bound reconciliation state during replay or relinking, then rebuild it against the new workflow run.
  • Surface missing-attempt and missing-usage counts through CLI statistics and force current and cumulative usage completeness to false while a gap remains.

Validation

  • Artifact tests: 356 passed.
  • CLI run-statistics tests: 19 passed.
  • Focused runtime reconciliation tests: 2 passed.
  • End-to-end CLI lifecycle test passed.
  • Runtime typecheck and strict lint passed.

Mitigates #1138
Mitigates #1139

@mrthankyou

Copy link
Copy Markdown
Collaborator

Cross-linking #1087 (stats counts operator-cancelled nodes as failed; status counts them as other). This PR covers the ledger half of it. Two notes:

1. Node status still says failed. The new NodeCancelled case records attempt outcome canceled, but node state still maps cancellation to failed in two places: statusFromWorkflowState (cancelled → failed, workflow-sync.ts ~L5991) and the NodeCancelled evidence handler (status: "failed", ~L5954). stats builds status_counts and the Status column from node state (aggregateNodeStatus in run-statistics.ts), so after this PR a cancelled node would show Status failed, Outcome canceled. status would still report it under other. The remaining #1087 fix would count a failed node whose latest attempt is canceled as canceled in status_counts (plus the statsStatusCounts key in cli-result.schema.json). Since this PR already edits run-statistics.ts, I'm holding that change until this merges, to avoid conflicts. Happy to fold it in here instead if you'd rather.

2. Cancels that arrive as NodeFailed. In #1087's run, the interrupted nodes logged CLI aborted See https://smithers.sh/reference/errors. If an operator cancel can reach us as NodeFailed with an abort error rather than NodeCancelled, terminalOutcomeForEvent would still classify it failed/executor-error. I haven't confirmed which event Smithers emits on that path. Worth a check, or a test, if you know it offhand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants