Skip to content

fix(reliability): keep recovery retry on one current-main clock - #428

Open
seonghobae wants to merge 2 commits into
mainfrom
fix/recovery-clock-main-20260813-r3
Open

fix(reliability): keep recovery retry on one current-main clock#428
seonghobae wants to merge 2 commits into
mainfrom
fix/recovery-clock-main-20260813-r3

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Objective

Use one caller-supplied recovery instant for both stale-processing eligibility and retry scheduling so recovery behavior is deterministic and cannot drift across hidden wall-clock reads.

Exact current-head evidence

  • protected base at reconstruction: 961f44a19c4ced87f8062b3ab9842a9a08d05235;
  • exact head: 02d01de14fb7435dccde137ce264539f2c520438;
  • changed paths: DefaultConversionWorker.java and two focused worker regressions only;
  • CI, Security Scan, SAST Semgrep, and fuzz: terminal-success;
  • mergeable: true;
  • unresolved inline findings: zero;
  • stale predecessor fix(reliability): keep recovery retry time on one clock #384 is superseded.

Recovery contract

recoverPendingJobs(now, processingLeaseTimeout) derives stale-processing eligibility and the queued retry timestamp from the same caller-owned now. The regression fixture derives stale and future timing from that deterministic clock and proves the state store receives the exact recovery instant rather than a second Instant.now() observation.

Scope boundary

Recovery timestamp consistency only. This does not implement durable scheduling, transactional outbox, distributed worker leases, backpressure, cancellation, OpenTelemetry export, or issue #312 completion.

Merge gate

Keep this exact head unchanged. Auto-merge may act only after all live required checks remain successful, zero valid unresolved findings remain, and a qualifying independent non-author approval is attached to this exact head. Protected main has advanced since reconstruction; any required base update must regenerate every exact-head check and review. No predecessor evidence transfers.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review available on request

  • 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2e310dea-d01f-4d01-9729-267d7a273f58


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae enabled auto-merge (squash) August 12, 2026 23:58

Copy link
Copy Markdown
Collaborator Author

Exact-current-head review request for 02d01de14fb7435dccde137ce264539f2c520438 on protected main 961f44a19c4ced87f8062b3ab9842a9a08d05235. CI, Security Scan, SAST Semgrep, and fuzz are terminal-success; the three-file deterministic recovery-clock delta is mergeable with zero unresolved threads. Verify that stale eligibility and retry scheduling use the same caller-supplied instant and that no hidden wall clock remains. Review read-only; do not mutate the branch or bypass independent approval.

@opencode-agent review
@cwl-noema-review review
@strix review
@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='961f44a19c4ced87f8062b3ab9842a9a08d05235'
head='02d01de14fb7435dccde137ce264539f2c520438'

echo '== Checked-out commit =='
git rev-parse HEAD
echo '== Commit ancestry =='
git merge-base --is-ancestor "$base" "$head" && echo 'base-is-ancestor=yes'
printf 'ahead='; git rev-list --count "${base}..${head}"
printf 'behind='; git rev-list --count "${head}..${base}"
echo '== Changed paths =='
git diff --name-status "$base" "$head"
echo '== Java outlines for changed paths =='
git diff --name-only "$base" "$head" -- '*.java' | while IFS= read -r f; do
  echo "--- $f"
  ast-grep outline "$f" --items all --type class,method --view compact || true
done
echo '== Exact diff =='
git diff --unified=80 "$base" "$head" -- '*.java'
echo '== Candidate wall-clock calls in production Java =='
rg -n -i -C 3 '(Instant\.now|LocalDateTime\.now|OffsetDateTime\.now|ZonedDateTime\.now|Clock\.system|System\.currentTimeMillis|new[[:space:]]+Date[[:space:]]*\()' \
  -g '*.java' -g '!**/test/**' -g '!**/tests/**' .

Length of output: 39675


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== Relevant type and method outlines =='
ast-grep outline src/main/java/com/clearfolio/viewer/repository/InMemoryConversionJobRepository.java \
  --items all --type class,method --view signatures
ast-grep outline src/main/java/com/clearfolio/viewer/model/ConversionJob.java \
  --items all --type class,method --view signatures
ast-grep outline src/main/java/com/clearfolio/viewer/service/DefaultConversionWorker.java \
  --items all --type class,method --view signatures

echo '== Recovery eligibility implementations and declarations =='
rg -n -C 12 '\bfindRecoverableJobs\s*\(' \
  src/main/java src/test/java

echo '== Job state transition and readiness implementations =='
rg -n -C 10 '\b(isReadyForProcessing|markProcessing|markRetryScheduled|scheduleRetry)\s*\(' \
  src/main/java/com/clearfolio/viewer/model/ConversionJob.java \
  src/main/java/com/clearfolio/viewer/repository/InMemoryConversionJobRepository.java \
  src/main/java/com/clearfolio/viewer/repository/ConversionJobStateStore.java

echo '== All recoverPendingJobs call sites =='
rg -n -C 6 '\brecoverPendingJobs\s*\(' -g '*.java' .

echo '== Recovery-focused test source =='
cat -n src/test/java/com/clearfolio/viewer/service/DefaultConversionWorkerRecoveryClockTest.java

echo '== Working tree status =='
git status --short

Length of output: 35320


@seonghobae: I found no blocking issues.

I verified HEAD is 02d01de14fb7435dccde137ce264539f2c520438. The branch is one commit ahead and zero commits behind 961f44a19c4ced87f8062b3ab9842a9a08d05235.

recoverPendingJobs(now, processingLeaseTimeout) derives staleProcessingBefore from now. It passes now to findRecoverableJobs for due-job eligibility. It also passes that same now to stateStore.scheduleRetry for stale processing jobs.

The new regression captures the retry timestamp and requires it to equal recoveryNow. The previous Instant.now() call would not satisfy this assertion.

process, failure backoff, and delayed execution still read the wall clock. These reads occur after recovery selection and requeueing. They do not affect stale eligibility or the recovery retry timestamp.

You are interacting with an AI system.

Copy link
Copy Markdown
Collaborator Author

@opencode-agent review
@coderabbitai review
@cwl-noema-review review
@strix review

Review exact head 02d01de14fb7435dccde137ce264539f2c520438 only. Exact-head CI, Security Scan, SAST Semgrep, and fuzz are terminal-success; unresolved inline findings are zero. Validate that one caller-supplied recovery instant controls both stale eligibility and retry scheduling and that tests derive all temporal expectations from that same clock.

Review read-only; preserve independent reviewer identity. Do not mutate the branch, synthesize author approval, weaken protection, or merge.

@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='961f44a19c4ced87f8062b3ab9842a9a08d05235'
head='02d01de14fb7435dccde137ce264539f2c520438'

echo '== commit identity and ancestry =='
printf 'HEAD='; git rev-parse HEAD
git cat-file -e "${head}^{commit}"
git merge-base --is-ancestor "$base" "$head" && echo 'base_is_ancestor=yes'
printf 'ahead='; git rev-list --count "${base}..${head}"
printf 'behind='; git rev-list --count "${head}..${base}"

echo '== changed paths =='
git diff --name-status "$base" "$head"

echo '== changed Java outlines =='
git diff --name-only "$base" "$head" -- '*.java' |
while IFS= read -r file; do
  echo "--- ${file}"
  ast-grep outline "$file" --items all --type class,method --view signatures || true
done

echo '== exact Java diff =='
git diff --unified=100 "$base" "$head" -- '*.java'

echo '== recovery method and relevant test call sites =='
rg -n -C 14 '\b(recoverPendingJobs|findRecoverableJobs|scheduleRetry)\s*\(' \
  -g '*.java' src

echo '== wall-clock calls in conversion-worker recovery context =='
rg -n -C 5 '(Instant\.now\s*\(|Clock\.system|System\.currentTimeMillis\s*\()' \
  src/main/java/com/clearfolio/viewer/service/DefaultConversionWorker.java \
  src/test/java/com/clearfolio/viewer/service/DefaultConversionWorkerRecoveryClockTest.java \
  src/test/java/com/clearfolio/viewer/service/DefaultConversionWorkerTest.java

echo '== status =='
git status --short

Length of output: 99335


@seonghobae I found no blocking issues on 02d01de14fb7435dccde137ce264539f2c520438.

I verified that HEAD matches the requested commit. The branch is one commit ahead and zero commits behind 961f44a19c4ced87f8062b3ab9842a9a08d05235.

recoverPendingJobs(now, processingLeaseTimeout) derives staleProcessingBefore from now. It passes now to findRecoverableJobs(now, staleProcessingBefore). For stale PROCESSING jobs, it passes the same now to stateStore.scheduleRetry(...).

DefaultConversionWorkerRecoveryClockTest captures the retry instant and requires equality with recoveryNow. This test fails with the former independent Instant.now() call.

The remaining wall-clock calls are outside this recovery invariant. They support startup entry, normal processing, failure backoff, or delayed execution.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Collaborator Author

Exact-current-head independent review request for 02d01de14fb7435dccde137ce264539f2c520438.

The three-file recovery-clock slice is mergeable; CI, Security Scan, SAST Semgrep, and fuzz are terminal-success; unresolved inline findings are zero. One caller-supplied instant now governs stale eligibility and retry scheduling without a hidden second wall-clock read.

@opencode-agent review
@coderabbitai review
@cwl-noema-review review
@strix review

Review this exact head read-only. Do not mutate the branch, synthesize author approval, weaken protection, or merge. Any base update must regenerate all exact-head evidence.

@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown

Rate Limit Exceeded

@seonghobae have exceeded the limit for the number of chat messages per hour. Please wait 20 minutes and 41 seconds before sending another message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant