Repository navigation
controllers/repos: ignore chacra callbacks older than the stored state - #160
Merged
Merged
Conversation
chacra posts each repo status change from an independent celery task, so a late 'building' could overwrite a newer 'ready'. Store the repo's chacra 'modified' timestamp (new repos.chacra_modified column, alembic revision a1f3c9d2e4b5) and ignore updates whose 'modified' is older than the stored one. Assisted-by: Claude Fable 5.1 Signed-off-by: David Galloway <[email protected]>
djgalloway
force-pushed
the
wip-callback-ordering
branch
from
October 28, 2026 19:47
6f7c33f to
cad2246
Compare
zmc
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
chacra posts one callback per repo status change (
requested,queued,building,ready,failed,deleted) from independent celerycallbacktasks (chacra/asynch/recurring.py, acks_late, retried after 30 s), with nothing ordering them. shaman applied whichever POST arrived last, so a latebuildingcould overwritereadyand the repo stayedbuildingin shaman forever.Evidence. That broke ceph-dev-pipeline build 10373 and a teuthology-suite run on 2026-10-28: shaman showed the rocky/10 default repo of
ceph/wip-guits-main-2026-10-28-0918asbuildingwhile 3.chacra.ceph.com had it ready. shaman's nginx log has thereadyPOST at 13:23:04 and thebuildingPOST at 13:23:15. Cross-checking/api/search/?project=ceph&status=buildingagainst each record'schacra_urlfound 31 such stale records since 2026-10-15, across all three chacra nodes, and none before.Fix. The payload already carries the repo's chacra-side
modified, bumped by abefore_updatelistener on every commit of the row. Within a build chacra commits between postingqueued,buildingandready, so thereadypayload is always newer than thebuildingpayload of the same build, and a new build'srequestedcarries at least the value its previousreadydid. This stores the timestamp in a newrepos.chacra_modifiedcolumn (alembic revisiona1f3c9d2e4b5) and ignores updates whosemodifiedis older than the stored one, logging them at info. Equal values, payloads withoutmodified, unparseable values anddeletedare applied as before, so older chacra versions and existing API behaviour are unaffected.chacra serializes the timestamp with
str(datetime): naive when the row was loaded from its database, aware UTC when the listener just set it. Both forms are parsed and naive values are taken as UTC, which matches what the chacra nodes store (a ready repo's chacramodifiedis within 200 ms of shaman's own UTCmodifiedfor the same repo).Deploying: the new column needs
alembic upgrade headagainst the production database before the new code is started. Neither the ansible playbook (pecan populate) nor the openshift migration job runsupgrade, they onlystamp head.Tested: unit tests on sqlite and postgres 16 (181 passed), migration upgrade and downgrade on sqlite.