Repository navigation
Add retry backoff for connection errors in OnyxUtils.retryOperation #87782
Description
Errors #1, #3, #5 from the storage error investigation account for 51.7% of all failures and are transient connection issues. OnyxUtils.retryOperation currently retries immediately with no delay, which is ineffective — the DB connection needs time to recover.
Explore adding exponential backoff or an alternative retry mechanism for these connection-class errors.
Parent issue: #85252
Activity
leshniak commentedon Apr 15, 2026
I am Maciej from Callstack - I can work on this.
leshniak commentedon Apr 17, 2026
VictoriaLogs Investigation — Key Findings
Assumption tested: Exponential backoff will improve retry success rate for connection errors.
What data showed:
0% recovery rate at 0ms delay — All 5 retry attempts fail at the same rate (~67K each). No attempt succeeds more than any other. Flat curve, not declining.
100% exhaustion — Every retry chain that starts, finishes all 5 attempts and gives up. 67,404 operations hit
MAX_STORAGE_OPERATION_RETRY_ATTEMPTS.IDB is degraded, not dead — Users who exhaust retries on one key continue successfully writing to other keys in the same session. Example: one user had 28K successful writes alongside exhausted retries.
What this means:
Tradeoff:
Honest framing: Backoff is a well-reasoned bet, not a proven fix. The PR adds instrumentation (connection error logging with delay info) so we can measure actual recovery rates in production.
Julesssss commentedon Apr 17, 2026
Thanks for investigating. It seems we should drop this and just stick with the main improvements. LMK if you dissagree
leshniak commentedon Apr 20, 2026
@Julesssss I think it's worth trying. With the current logs I can't say with confidence that it will fix the issue, but there is a chance. It doesn't break anything.
Julesssss commentedon Apr 20, 2026
Okay fair enough, lets test delay. Could you suggest a plan here for what/when we'll measure exactly. And what would lead us to rolling back.
leshniak commentedon Apr 30, 2026
Hey @Julesssss, sorry for the delay!
I just pushed observability improvements to the Onyx PR — the original version only logged retry attempts but not outcomes, which would have made the experiment unmeasurable. Now we have three distinct log signals:
Connection error detected, retrying with backoff ({delay}ms)— on each retry attempt (was already there)Connection error recovered after backoff on attempt {N}/5— when a retry succeeds after backoff (new)Connection error exhausted all retries with backoff— when all 5 retries fail (new, distinct from generic exhaustion)Measurement plan
What to measure (via VictoriaLogs queries on the 3 signals above):
When:
Rollback criteria:
Success criteria:
Julesssss commentedon Apr 30, 2026
No worries, plan looks good to me. Lets get the PR open for review and then we'll need to wait a bit while Onyx slowly gets bumped 👍
fabioh8010 commentedon May 12, 2026
@leshniak @Julesssss I propose we hold this idea/PR until we have proper DB healing mechanisms in Onyx as detailed here. I feel that if we have cases when the storage write is repeatedly and instantly failing (like
Internal error opening backing store for indexedDB.open.) the first and best goal is probably on focus on the healing strategy. Then we can think about implement retry backoff for specific cases.Julesssss commentedon May 12, 2026
Yeah that makes sense, thanks for jumping in, lets discuss further in the linked issue.
fabioh8010 commentedon Jul 8, 2026
@elirangoshen is taking over to verify if we need this retry backoff logic since he did a recent audit over the storage errors.
melvin-bot commentedon Jul 13, 2026
@leshniak, @Julesssss, @fabioh8010, this Monthly task hasn't been acted upon in 6 weeks; closing.
If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.