fix(bulk-submit): lease heartbeats through files, renewed by the claimed duration - #740
Merged
Conversation
… its own duration Closes #739. One output file whose ingestion outlives the lease expired mid-stream: claim_next_manifest saw an abandoned processing manifest, a rival worker claimed it, and the manifest restarted from its first file - an unbounded silent loop, invisible in the counts because re-ingested entries upsert idempotently, and invisible to the #646 stall detection because every re-claim refreshes the lease. Found live in #448's Synthea pass: all 896 Encounters of a 20-file submission re-indexed seven times in as many minutes while the poll answered processing 0%. run_job now drives each file's ingest under a select with a periodic heartbeat at a third of the remaining lease, and ManifestLease carries the duration it was claimed with so the four backends' heartbeat renews by it rather than a hardcoded 60 seconds. The regression test trickles a single file over six seconds under a two-second lease: a rival claim mid-file must come back empty, and every line lands exactly once.
9 tasks
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Covers the LeaseLost arms of the heartbeated ingest loop (codecov patch): after a legitimate takeover, the stale claimant's first beat answers LeaseLost and run_job returns without ingesting anything under the old fencing token.
The four backends each carried the same duration-to-expiry arithmetic; ManifestLease::renewed_expiry owns it now (and the sqlite ring covers it), leaving each heartbeat one call. Mongo still rounds to BSON precision on its side.
smunini
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #739. Found live during #448's Synthea sanity pass — writeup on the issue.
The livelock
run_jobheartbeated the lease only between files. A single output file whose ingestion outlives the lease expires mid-stream;claim_next_manifestsees an abandonedprocessingmanifest and hands it to a rival worker, which restarts from the first file. Entries upsert idempotently, so counts freeze while CPU burns; every re-claim refresheslease_expiry, so the #646/#711 stall detection sees a healthy lease forever. Observed: all 896 Encounters of a 20-file submission re-indexed 7× in 7 minutes, poll stuck atprocessing 0%.Compounding it, all four backends'
heartbeat()renewed by a hardcoded 60 s, ignoring the claimedlease_duration— so no configuration could buy a slow file more time.The fix
run_jobdrives each file's ingest (and the deleted-files pass untouched — it heartbeats per file and those files are refs, not bulk data) under atokio::select!with a periodic heartbeat at a third of the remaining lease (clamped 1–60 s), aborting cleanly onLeaseLost.ManifestLeasenow carrieslease_duration; sqlite/postgres/mongodb/s3heartbeat()renew by it.Regression test
a_file_slower_than_the_lease_stays_leased_to_completion: one file trickled over ~6 s under a 2 s lease. Mid-file, a rivalclaim_next_manifestmust returnNone; the run must complete with every line ingested exactly once. Verified both ways — with the periodic heartbeat neutralized the test fails at the rival-claim assert with the exact production symptom.