Conversation
The status page listed every background job except the archiver, so the only way to tell whether snapshots were still draining to R2 was to open bullboard or query the database by hand. That gap mattered during the failures documented in 0b44cf7, where archive jobs were dying on connection loss and nothing surfaced it. Adds an "Archiving snapshots" row to the job list, plus an Archiving section with the retention window, the age of the oldest snapshot still in Postgres, whether that snapshot is past retention, and the queue's failed job count. Oldest snapshot is read with ORDER BY id ASC LIMIT 1, a primary key lookup. Counting rows past the retention cutoff would be a sequential scan, since GameServerSnapshot has no index on createdAt, and it would be slowest exactly when the table is backlogged and the page is being looked at. Oldest-snapshot age carries the same signal. Two supporting changes: - SNAPSHOT_RETENTION_HOURS moves from the worker to libs/teerank so the frontend renders the same number the worker enforces. Neither container sets it, so both still default to 48, but an override now has to be set on both. - removeOnComplete.age on the archive queue goes 10 minutes -> 6 hours. The job is scheduled every 10 minutes, so the completion record expired at roughly the schedule cadence and the last run would intermittently disappear, reading as Down while the archiver was healthy. The archiving row uses a 30 minute staleness threshold rather than the 10 minutes the other jobs use, because the job is scheduled every 10 minutes and is allowed to run for 5. Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The status page listed every background job except the archiver, so the only way to tell whether snapshots were still draining to R2 was to open bullboard or query the database by hand. This adds an "Archiving snapshots" row to the job list, plus an Archiving section showing the retention window, the age of the oldest snapshot still in Postgres, whether that snapshot is past retention, and the queue's failed job count. Oldest snapshot is read with
ORDER BY id ASC LIMIT 1rather than counting rows past the retention cutoff, which would be a sequential scan (no index oncreatedAt) and slowest exactly when the table is backlogged.Two supporting changes:
SNAPSHOT_RETENTION_HOURSmoves from the worker tolibs/teerankso the frontend renders the number the worker enforces — neither container sets it so both still default to 48, but an override now has to be set on both — andremoveOnComplete.ageon the archive queue goes from 10 minutes to 6 hours, since the job is scheduled every 10 minutes and the completion record was expiring at roughly the schedule cadence, making the row read Down while the archiver was healthy. Verified by rendering/statusagainst a scratch database in both states: no snapshots ("Backlog: Up to date") and a 5-day-old snapshot seeded ("Oldest snapshot: 5 days ago", "Backlog: 3 days past retention / Late").🤖 Generated with Claude Code