Repository navigation
feat(runner): keep jobs that retry gives up on as dead tickets - #511
Merged
Merged
Conversation
An Abandon record moves a running ticket, with its payload and why, to the state's dead tickets, where no Select takes it. revive makes a dead ticket pending again with a new budget; cancel drops it. Snapshots carry the dead tickets. Settled attempts still complete. Laws: dead_never_selected; complete_never_selected and recovered_never_selected now assume the ticket is not dead; settled_complete, unsafe_abandon, and transient_abandon replace requeue_stop. Closes #444
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A job that retry gives up on is no longer silently completed. The runner now keeps it as a dead ticket, with its payload and the reason (
WhySpent,WhyLate,WhyUnsafe), so a caller can tell a delivered job from an abandoned one and replay it later withrevive.What changes for callers
R.dead(A, s)lists the dead tickets of aState, newest first.R.revive(~A, ~penc, r, id, deadline)makes a dead ticket pending again with a new retry budget.R.cancelnow also drops a dead ticket.Settledattempt still completes.Retry.Settledcovers both a success and a failure that another try would repeat, so the runner cannot tell them apart. The README says so under Trust boundary.This is a breaking change to
State(a third field) andOp(two new records, tags 8 and 9), so runner rises to 0.6.0.0. No sibling imports runner.Laws
dead_never_selected: afterAbandon{i, why}and any ops with noReviveofi, noSelecttakesi.complete_never_selectedandrecovered_never_selectedgain one hypothesis:iis not dead, as a running ticket is not.cancel_never_selectedis unchanged; Cancel now clears all three registries.requeue_stop("plan stops ⇒ Complete") is no longer true.settled_complete,unsafe_abandon, andtransient_abandonreplace it.The proof generalizes the old
goneinvariant toinv(i, P, s), wherePis a predicate on the dead list.goneuses "i is absent", and the dead law uses "anything". The same per-op helpers serve both.Verification
scripts/check.sh runner:PROOF.bendchecks every law, and a deliberately wrongdead_never_selectedbody fails.check.bend: a job that fails every attempt ends dead with its payload, keeps it across a reopen, runs 3 more attempts afterrevive, andcanceldrops it. The op and snapshot codecs round-tripAbandon,Revive, and dead tickets.bench/run.py 1 12runs, and all checksums agree.Closes #444