Skip to content

fix(occurrence-data): cross-replica generation lock via Redis; tolera… - #960

Merged
rogonion merged 2 commits into
testfrom
fix/occurrence-data-and-na-misrepresentation
Nov 9, 2026
Merged

rogonion merged 2 commits into
testfrom
fix/occurrence-data-and-na-misrepresentation

Conversation

@rogonion

@rogonion rogonion commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

HA sync: the API runs with 2 replicas sharing FULL_OCCURRENCE_DATA_FOLDER on a PVC, and the in-process inflight lock did not stop a second replica from spawning its own OccurrenceGeoJob. kickOff now acquires a Redis lock (SET NX PX, token-checked release via Lua) before spawning the CLI child; background refreshes skip if another replica holds the lock, cold starts wait for it and then re-check staleness. status() reports refreshing from the lock too. All Redis calls are wrapped in a 3s-timeout guard so a Redis outage can never hang the endpoint. ioredis added as a direct dependency.

Verified locally: with the lock key held manually, a stale request served in 87ms with X-Occurrence-Data-Refresh: in-progress and spawned no geojob (status showed refreshing:true); after deleting the key the next request spawned the job, which completed and released the lock.

Geojob: atomicReplaceFile now checks the staging file exists and skips (with a warning) instead of failing hard on ENOENT, and the staging-cleanup unlink no longer warns on ENOENT (moveFile already removed the file). Reproduced empty/missing target directory runs — both complete successfully.

rogonion and others added 2 commits November 9, 2026 15:01
…te missing staging files

HA sync: the API runs with 2 replicas sharing FULL_OCCURRENCE_DATA_FOLDER on a
PVC, and the in-process inflight lock did not stop a second replica from
spawning its own OccurrenceGeoJob. kickOff now acquires a Redis lock
(SET NX PX, token-checked release via Lua) before spawning the CLI child;
background refreshes skip if another replica holds the lock, cold starts wait
for it and then re-check staleness. status() reports refreshing from the lock
too. All Redis calls are wrapped in a 3s-timeout guard so a Redis outage can
never hang the endpoint. ioredis added as a direct dependency.

Verified locally: with the lock key held manually, a stale request served in
87ms with X-Occurrence-Data-Refresh: in-progress and spawned no geojob
(status showed refreshing:true); after deleting the key the next request
spawned the job, which completed and released the lock.

Geojob: atomicReplaceFile now checks the staging file exists and skips
(with a warning) instead of failing hard on ENOENT, and the staging-cleanup
unlink no longer warns on ENOENT (moveFile already removed the file).
Reproduced empty/missing target directory runs — both complete successfully.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <[email protected]>
…the Redis lock

CI npm ci failed with ERESOLVE: [email protected] declares peerOptional
ioredis@^5.0.4 while the lock had ioredis@6. Downgrade to ^5.11.1 (the lock
API used — set NX PX / eval / exists / quit — is identical in v5); npm ci
now resolves cleanly and nest build passes. Verified at runtime: with the
generation lock key held, status reports refreshing:true in ~13ms and a
stale request serves with X-Occurrence-Data-Refresh: in-progress while
logging 'generation already running on another replica, skipping'; after
releasing the key the next request spawns the job and completes cleanly.

docs/occurrence-data-pipeline.md updated with the cross-replica Redis lock
semantics (background skip vs cold-start wait, lock TTL, 3s Redis guard).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <[email protected]>
@rogonion
rogonion merged commit bd8c647 into test Nov 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant