[Onyx] Investigate UnknownError: Internal error opening backing store for indexedDB.open. storage error

@fabioh8010

Coming from #87844.

Summary

Let's investigate the following Onyx storage error: UnknownError: Internal error opening backing store for indexedDB.open., which accounts for 884,955 (26.3%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.

Platforms	Top Methods
Desktop Chrome (664K), Desktop Edge (220K)	`setWithRetry` (682K), `multiSetWithRetry` (181K)

Retries Exhausted	Evictions	No Eviction	Quota Checks	>= 95% Full	Mean % Full	Quota Check Failed
147,944	0	0	0	—	—	0

No eviction or quota check events — this is an IndexedDB internal failure unrelated to storage space
This error is not classified as a storage capacity error in Onyx (STORAGE_ERRORS only includes QuotaExceededError for IDB), so it never triggers the eviction flow — it just retries immediately and fails repeatedly
Likely caused by corrupted IndexedDB state, browser extensions interfering, or Chromium-level bugs

Action Items

Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios where this happens — is there a common pattern (e.g., specific pages, actions, or browser versions)?
Consider adding retry backoff for this error. Currently retries are immediate with no delay, and 147K requests exhausted all retries. A delay could give the browser time to recover the backing store. (being handled here -> Add retry backoff for connection errors in OnyxUtils.retryOperation #87782)

Issue Owner

Current Issue Owner: @fabioh8010

I can work on this.

Investigation Report

Summary

Internal error opening backing store for indexedDB.open. is a Chromium-only error caused by corrupted LevelDB files backing IndexedDB. It affects a small number of users permanently — once corrupted, the DB never self-heals. Two bugs in Onyx's error handling make the impact catastrophic: the intended fallback to MemoryOnlyProvider never fires, and retries are futile against structural corruption.

Data Analysis (VictoriaLogs, 7-day window)

User concentration — not widespread, highly concentrated:

Metric	Value
Distinct affected users	~36
Top user's share	32% (21,844 errors)
Top 5 users' share	64.5%

Browser versions — not a regression, spans all versions:

Chrome Version	Errors
146	43,058 (63.5%)
147	12,472 (18.4%)
145	4,220 (6.2%)
144	4,075 (6.0%)
135	2,766 (4.1%)
143	1,118 (1.6%)

Session Analysis (top user: `[redacted]`)

Traced a full session from first log entry through error cascade:

14:44:49  OnyxUpdateManager applying HTTPS update — lastUpdateID: 0   ← IDB already unreadable
14:45:12  Onyx.set(networkRequestQueue) — PusherPing persist attempt
14:45:12  IDB write fails — retry 0/5 through 5/5 all within 1ms
14:45:12  [alrt] Storage operation failed after 5 retries
14:45:12  StoreUtils - Clearing all the stores (×2)
14:45:14  New browserGUID — user refreshes, cycle repeats

Key observations:

lastUpdateID: 0 on every session start — DB is dead before first write
App is functional (API calls, Pusher work) but nothing persists
browserGUID changes between log batches — user is refreshing trying to fix it
Retries complete in <1ms — all futile against corrupted backing store

Verification of Root Cause

Three falsification tests were run:

Test	Result
Any successful IDB writes mixed with errors?	No — `[Onyx] set called` logs are pre-write cache updates, not IDB confirmations. 100% of IDB writes fail.
Does `degradePerformance` ever fire?	No — zero `"Falling back to only using cache"` log entries found. Dead code confirmed.
Does any user recover without clearing browser data?	No — steady daily error volume, `lastUpdateID: 0` on every session.

Two Bugs in Onyx

Bug 1: degradePerformance is dead code for async errors (storage/index.ts:31-55)

try {
    resolve(fn());  // fn() returns a Promise — rejections escape the catch
} catch (error) {
    // This only catches synchronous throws.
    // IDB operations reject asynchronously — this block never executes.
    degradePerformance(error);
}

Bug 2: retryOperation can't help (OnyxUtils.ts:788-829)

The error name is UnknownError, not QuotaExceededError, so it doesn't match STORAGE_ERRORS. No eviction is attempted. The operation is retried as-is — but a corrupted LevelDB backing store won't recover from retries. After all retries exhaust, retryOperation resolves with Promise.resolve() — silently dropping the write.

Data Loss Assessment

These users' data is already lost. The proposed fix does not introduce any new data loss:

lastUpdateID: 0 on every session proves the app can't read any stored data — IDB is unreadable
Chromium's own recovery (leveldb::RepairDB()) already failed before Onyx code runs
Every write silently drops via Promise.resolve() after retries exhaust — nothing persists
On page refresh, all state is gone and the cycle repeats

The corrupted IDB is a write-only black hole: nothing goes in, nothing comes out. Calling deleteDatabase() removes files that are already inaccessible.

After healing, the app starts with a fresh empty IDB — the same state it already sees every session (lastUpdateID: 0). The server is the source of truth: all reports, transactions, and policies are re-fetched, same as a first login or new device. The only data that could theoretically be lost is optimistic writes in the SequentialQueue that hadn't reached the server — but those also failed to persist to IDB (same error), so they're already gone regardless.

Net data loss from the fix: zero. The alternative — doing nothing — is worse: users stay permanently broken with silent data loss on every session.

Chromium Context

The error originates in content/browser/indexed_db/indexed_db_factory.cc when IndexedDBBackingStore::Open() returns a non-OK leveldb::Status. Chromium has a built-in recovery (attempts leveldb::RepairDB(), then deletes and recreates) — but when that also fails, the error becomes permanent.

indexedDB.deleteDatabase() takes a different code path (DestroyDB() — filesystem-level removal) and can succeed even when open() fails, as long as the process has file permissions.

Proposed Fix

Key design decision: This error is permanent corruption, not a transient connection issue. It must be handled in the storage layer (tryOrDegradePerformance) before it ever reaches retryOperation. This keeps it cleanly separated from the retry backoff mechanism introduced in PR #775.

How the error flows today (broken)

IDB operation rejects async
  → tryOrDegradePerformance sync catch misses it
  → retryOperation catches it
  → not in STORAGE_ERRORS, not in CONNECTION_ERRORS (after fix below)
  → retries 5× with backoff
  → all fail → Promise.resolve() → silent data loss

How the error should flow (fixed)

IDB operation rejects async
  → tryOrDegradePerformance async catch detects "Internal error opening backing store"
  → deleteDatabase('OnyxDB')
  → re-init IDB provider, retry operation once
    → success: healed, app re-fetches from server (same as first login)
    → failure: degrade to MemoryOnlyProvider
  → retryOperation is never reached for this error

Coordination with backoff PR (#775)

The backoff PR currently classifies this error as a CONNECTION_ERROR in IDB_CONNECTION_ERRORS. It should be removed — it's not a transient connection issue, and including it would mean users wait ~3.1s of backoff for an error that provably never recovers. After removal, each error class has a single, clear handler:

Error class	Where handled	Strategy
Capacity errors (`QuotaExceeded`, `disk full`)	`retryOperation`	Immediate retry + eviction
Connection errors (Safari drops, DB closing)	`retryOperation`	Exponential backoff (100ms→1.6s)
Permanent corruption (backing store)	`tryOrDegradePerformance`	Heal IDB → fallback to memory

This also improves the backoff experiment's data quality — corruption errors (which always show 0% recovery) won't pollute the recovery rate measurements for actual transient connection errors.

Changes needed

File	Change
`lib/storage/index.ts`	Fix async catch in `tryOrDegradePerformance`, add heal-then-degrade logic for backing store error
`lib/storage/providers/IDBKeyValProvider/index.ts`	Expose re-init capability for healing
PR #775 (`lib/OnyxUtils.ts`)	Remove `'internal error opening backing store'` from `IDB_CONNECTION_ERRORS`

Observability

Distinct log messages for each outcome:

IDB corruption detected, attempting heal — entered healing path
IDB heal succeeded — deleteDatabase + re-init worked, operation retried successfully
IDB heal failed: <error>, falling back to memory — healing failed, degraded to MemoryOnlyProvider

Post-deploy monitoring via VictoriaLogs:

_msg:"IDB corruption detected" OR _msg:"IDB heal succeeded" OR _msg:"IDB heal failed"
| stats by (_time:1d) count() as events

And verify error volume drops:

_msg:"Internal error opening backing store" AND _msg:"retryAttempt: 0/5"
| stats by (_time:1d) count() as errors

Root Cause Confidence: ~85%

High confidence pieces:

Async catch bug in tryOrDegradePerformance — 100%. Code is provably dead for async rejects.
retryOperation futility — 100%. UnknownError not in STORAGE_ERRORS, retries can't fix corrupted LevelDB.
Permanent per-user, not transient — 95%. VictoriaLogs show lastUpdateID: 0 every session, zero recovery, steady daily volume.
Chromium-only — 95%. Error string matches Chromium source, browser distribution is all Chrome/Edge.
Small user concentration (~36) — 90%. VictoriaLogs data is clear.

Inferred (lower confidence):

LevelDB corruption as the trigger — 70%. Symptom matches Chromium's LevelDB corruption path, but we haven't confirmed actual corrupted files on disk. Could also be filesystem-level issues (permissions, antivirus locks, disk errors) producing identical symptoms.
deleteDatabase() will heal — 75%. Chromium source says it uses DestroyDB() (filesystem removal), different code path from open(). Should work, but untested on actual corrupted instances.
~36 users accounts for 884K errors — 80%. 7-day window extrapolated to 4-week figure. User set could drift.

What would increase confidence:

Reproducing corruption locally (trigger LevelDB corruption → confirm open() fails → confirm deleteDatabase() heals) → +10%
Fullstory session from affected user confirming broken UX matches our timeline → +3%
Chromium bug tracker entries describing this exact permanent state → +2%

Tomorrow I want to continue on this and maybe try simulate the corrupted DB. Hopefully the pattern will match.

Investigation Report — Chapter 2: Reproduction & Healing Simulation

Reproduction Setup

Used Playwright with a persistent Chrome profile (--user-data-dir) against staging.new.expensify.com. Logged in, verified OnyxDB working (99 keys in keyvaluepairs store), then located the LevelDB files on disk:

<profile>/Default/IndexedDB/https_staging.new.expensify.com_0.indexeddb.leveldb/
├── 000004.log    (153K — write-ahead log)
├── 000005.ldb    (3.5K — sorted table)
├── CURRENT       (16B  — points to active MANIFEST)
├── LOCK
├── LOG
└── MANIFEST-000001 (80B — database schema)

Experiment 1: Simple Corruption

Overwrote MANIFEST, .ldb, .log, and CURRENT with random bytes. Reloaded page.

Result: Chrome auto-repaired. indexedDB.open('OnyxDB') succeeded, 99 keys intact. Chromium's built-in leveldb::RepairDB() handled it.

Conclusion: Simple file corruption is insufficient to reproduce the permanent error. Chrome's recovery is robust against basic corruption.

Experiment 2: Corruption + Immutable Files

Same corruption, then set macOS immutable flags (chflags uchg) on all LevelDB files. This prevents Chrome from repairing, deleting, or recreating files — simulating the real-world condition where Chrome's full recovery pipeline (repair → delete → recreate) fails.

Reloaded page.

Result:

indexedDB.open('OnyxDB')
// → onerror: UnknownError: "Internal error opening backing store for indexedDB.open."

indexedDB.deleteDatabase('OnyxDB')
// → onerror: UnknownError: "Internal error opening backing store for indexedDB.open."

Exact error match with production. Both open() and deleteDatabase() fail with the identical error string.

Experiment 3: Chrome Process-Level Caching

Removed immutable flags (files now deletable but still corrupt). Tested from the same browser process:

indexedDB.open('OnyxDB')       // → STILL FAILS
indexedDB.deleteDatabase('OnyxDB')  // → STILL FAILS

Finding: Chrome caches the IDB failure state per-origin in browser process memory. Even after the underlying filesystem issue is resolved, operations continue to fail within the same process.

Experiment 4: Browser Restart with Corrupt (Deletable) Files

Killed Chrome process. Relaunched with the same profile.

indexedDB.open('OnyxDB')  // → STILL FAILS

Finding: Chrome's recovery pipeline also fails on restart. The corruption we created is deep enough that RepairDB() fails, and Chrome's fallback (delete directory + recreate) also doesn't succeed. Files on disk remain unchanged — Chrome gives up.

Experiment 5: External File Removal → Healing

Deleted the entire LevelDB directory from disk (rm -rf). Then, in the same browser process (no restart):

indexedDB.deleteDatabase('OnyxDB')  // → SUCCESS ✅
indexedDB.open('OnyxDB')            // → SUCCESS ✅ (triggers onupgradeneeded)

// Write + read roundtrip:
store.put('healed!', 'test_key')    // → SUCCESS ✅
store.get('test_key')               // → 'healed!' ✅

Full healing confirmed. Once the corrupted LevelDB directory is gone, deleteDatabase() completes (nothing to conflict with), and open() creates a fresh database. No browser restart needed.

Revised Understanding

What we assumed	What we found
`deleteDatabase()` uses a different code path (`DestroyDB`) and should work	`deleteDatabase()` also fails when corrupt files exist — it needs to access LevelDB to delete it
Chrome caches error in process memory	Partially true — but the root issue is that corrupt files block both `open()` AND `deleteDatabase()`
Browser restart clears the cache	No — if corrupt files persist, restart doesn't help
Removing files + `deleteDatabase()` needs restart	No — works in the same process!

What This Means for the Fix

deleteDatabase() alone won't heal from JavaScript — Chrome can't delete a corrupt LevelDB directory through the IDB API. The files physically block it.

Viable healing paths:

Approach	Mechanism	Pros	Cons
`Clear-Site-Data: "storage"` HTTP header	Server sends header, browser wipes all storage for origin	Clean, no user action, works at OS level	Requires server-side coordination, clears ALL storage (not just OnyxDB)
`navigator.storage.getDirectory()` + OPFS	Access filesystem directly	Targeted	OPFS is separate from IDB — can't access IDB's LevelDB files
User-guided "Clear site data"	Modal instructs user to clear via browser settings	Poor UX, user friction	Always works as a last resort
ServiceWorker `Cache-Control`	SW intercepts request, triggers server `Clear-Site-Data`	Can be triggered client-side	Requires SW registration before corruption

Recommended approach: Detect corruption → degrade to MemoryOnlyProvider (app keeps working) → show modal explaining the situation → trigger Clear-Site-Data: "storage" via a dedicated API endpoint → reload page → fresh IDB, app re-fetches from server.

Code-Level Test Results

Wrote tests in tests/unit/storage/StorageCorruptionTest.ts using fake-indexeddb:

Test	Status	What it proves
Async error escapes `tryOrDegradePerformance`	PASS	Bug 1 confirmed: sync catch misses async rejection
Non-corruption errors don't trigger healing	PASS	Error classification works
Detect corruption → heal → recover	FAIL (expected)	Fix not implemented yet
Heal fails → degrade to MemoryOnlyProvider	FAIL (expected)	Fix not implemented yet

The 2 passing tests confirm current bugs. The 2 failing tests define the target behavior for the fix.

Updated Confidence

Root cause confidence: ~85% → ~92%

Reproduction of exact error string: +5%
Chrome's recovery pipeline failure confirmed: +2%

deleteDatabase() healing confidence: 75% → 40% (revised down)

deleteDatabase() fails when corrupt files exist on disk
Need Clear-Site-Data or equivalent browser-level wipe instead

@leshniak

@leshniak Thanks for the investigations. I'm struggling however to understand WHY this are happening to the users, what's the pattern? I would like us to investigate first the patterns and the WHY, so I strongly suggest to focus on this action item before we jump into any kind of solutions

Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios where this happens — is there a common pattern (e.g., specific pages, actions, or browser versions)?

Let's take 5-10 of the most affected users and check the three tools together (Sentry, VictoriaLogs and Fullstory) to see what the were doing when these issues were happening, and try to find a pattern or understand the root cause.

@fabioh8010

@fabioh8010 I don't think it's our fault, looks more like some inconsistency in the users' filesystem / Chrome profile. But I'll pay more attention to Fullstory today.

Investigation Report — Chapter 3: Why Does Corruption Happen?

Approach

Analyzed the top 10 affected users across VictoriaLogs, Fullstory, and Sentry to find common patterns. Cross-referenced with Chromium bug tracker.

User Pattern Analysis

Classified users into two behavioral patterns based on error duration:

Pattern	Users	Behavior
Burst (< 2 hours)	[redacted] (1h22m, 191K), [redacted] (46m, 34K), [redacted] (1h15m, 23K)	Single session, massive error volume, then stops
Persistent (days)	[redacted] (2.5d), [redacted] (18h), [redacted] (3.4d), [redacted] (23h), [redacted] (25h), [redacted] (2.7d)	Errors spanning multiple sessions across days

VictoriaLogs: Pre-Error Context (4 users analyzed)

Queried server-side logs in the 5 minutes before each user's first IDB error.

User	First command before error	clientUpdateID	Client logs?
[redacted]	SAML login	`0`	No
[redacted]	ReconnectApp	`12693974043` (non-zero)	No
[redacted]	PusherPing	`12723667431` (non-zero)	Yes — `lastUpdateID: 0`
[redacted]	PusherPing	`12690985759` (non-zero)	No

Key findings:

3 of 4 users had working IDB before corruption — non-zero clientUpdateID proves they had a functioning database that stored real update IDs at some point.
No trigger visible in server logs — all users show normal API activity (PusherPing, ReconnectApp, Search) right before their first error. No unusual server-side errors, no failed requests, no timeouts.
[redacted] client logs confirm the broken state** — app functions normally (SequentialQueue processing, API calls succeeding), but OnyxUpdateManager reports lastUpdateID: 0 because IDB can't be read. The corruption predates the session.

Fullstory: User Experience During Corruption

[redacted] (Chrome/135.0.0.0, macOS):

No Fullstory session at the exact time of first error (19:44 UTC) — Fullstory may have failed to initialize due to broken storage
Session 3 hours later (5:28 PM local): "Uncaught (in promise) UnknownError: Internal error opening backing store for indexedDB.open." appears in the first second
Despite the error, the app renders normally — user navigates from Search page to homepage
No error modal, no banner, no degraded UI — the user has zero indication anything is wrong
Confirms Bug 1: the error is "Uncaught (in promise)" — tryOrDegradePerformance sync catch missed it

[redacted] — Chrome update as trigger:

Working sessions on Chrome 144
Chrome updates to Chrome 146
2-3 short clean sessions (5-10s, no errors) on v146
Then permanent IDB corruption begins — errors from that point forward
The short sessions themselves were clean — corruption happened between sessions, likely during Chrome's background profile migration or LevelDB compaction after the version update

[redacted] — no Fullstory session at the error time (gap in Chrome recordings), but VictoriaLogs shows 3 different browserGUIDs within 1 second — rapid tab open/close behavior.

[redacted] — no Fullstory data available.

Chromium Bug Tracker: Known Issue

This is a well-documented, recurring Chromium bug class. Key bugs:

Bug	Relevance
369636250	Exact error match. "Internal error opening backing store for indexedDB.open." Escalated from Dexie.js community. ~24,500 occurrences/month for one app alone. Chrome 107+.
340398745	LevelDB Lock contention during backing store initialization. Multiple fixes merged across release channels. Filed May 2024.
1303434	Chrome 99 QuotaManager regression caused widespread IDB failures. Fixed in M100/M101. Precedent for version-update corruption.
40570200	IDB backend store schema migration (v2→v3) failed to update on-disk version number. All platforms affected.
40275097	Unclean Windows shutdown causes IndexedDB corruption.
41328523	Repeated LevelDB recovery failures after system crash.
40253999	Active migration: IndexedDB from LevelDB to SQLite. Currently only in incognito mode. Persisted storage migration not yet shipped (flagged as "higher risk").

Chromium's recovery behavior: When LevelDB corruption is detected, Chrome attempts RepairDB(), then tries to delete and recreate the directory. When both fail, the error becomes permanent. Critically, Chromium does not use LevelDB's built-in recovery API — it just deletes the entire database. When deletion also fails (permissions, locks, cached error state), the database is permanently inaccessible.

Known Corruption Triggers (from Chromium bugs)

Chrome version upgrades — schema migration failures, QuotaManager regressions (our [redacted] case: v144→v146)
Unclean shutdown / crash — most commonly reported trigger
Low disk space — corrupts LevelDB during compaction
Lock contention — multiple tabs/processes competing for LevelDB lock files
Browser version downgrade — serialized values from newer Chrome cannot be read by older versions

Conclusion: Why Does Corruption Happen?

IDB corruption is a known, recurring Chromium-level issue that Expensify cannot prevent. It occurs at the LevelDB filesystem layer, below our application's observability boundary. Our Fullstory and VictoriaLogs analysis confirms:

No Expensify-app-level trigger is visible before corruption
Corruption predates the user's session (IDB is already broken when the app loads)
At least one case ([redacted]) correlates with a Chrome major version update (144→146)
The Chromium bug tracker has multiple open issues describing identical behavior
Other major apps (Signal Desktop, Dexie.js users, Electron apps) report the same problem

What Expensify can do: Since we can't prevent corruption, we must detect and recover from it. The fix proposed in Chapter 1 (detect → degrade to MemoryOnlyProvider → show user notification) remains the right approach. The Fullstory evidence adds urgency: currently, affected users see a silently broken app with no indication of the problem.

Updated Confidence

Assessment	Previous	Now	Change
Root cause (LevelDB corruption)	~92%	~97%	Chromium bugs confirm exact error path, [redacted] Chrome update correlation
Corruption is unpredictable/unpreventable	80%	~95%	No app-level trigger found across 4 users, known Chromium bug class
`deleteDatabase()` healing from JS	40%	40%	No new data (Chapter 2 findings still apply)
User impact (silent broken app)	90%	~99%	Fullstory confirms: app renders, no error UI, user unaware

@leshniak

Hey @leshniak I'm bit confused with the user reports, what kind of issues did they experience when having these errors? Was the app working normally and they doing everything without issues, or were they impacted/blocked by the storage errors?

Could you link some Fullstory sessions here so we can check?

@Julesssss

I will be OOO next week and will return on May 4th, tagging @Julesssss to have a look here.

Chapter 3 — Addendum: Fullstory Session Evidence

To verify whether IDB corruption directly harms the user experience, I located Fullstory sessions for 4 of the 5 top-affected users ([redacted] had no Fullstory data).

1. [redacted]

Session: Replay link
Device: Tepoztlán, MOR · OS X · Chrome

4× [Onyx] Storage operation failed after 5 retries at second 1. Inbox shows "Loading" state initially. User proceeds to Search, browses reports, edits submit tags (0–6) on multiple expenses, views transaction details — 79 minutes of productive work. Periodic error-clicks (dead taps on broken elements) and sporadic storage errors resurface at ~11 min and ~12 min. App never crashes, no error UI shown. User ends session normally.

Verdict: No direct harm. Silent errors, user unaware.

2. [redacted]

Session: Replay link
Device: La Jolla, CA · OS X · Chrome

Session starts with Uncaught (in promise) UnknownError: Internal error opening backing store on the first second, then app renders the Search page normally. User refreshes, navigates to homepage. No crash, no error screen.

Verdict: No direct harm. Error on first second, app renders normally after.

3. [redacted]

Session: Replay link
Device: Waynesboro, VA · OS X · Chrome

Very short session (~5–10 s). Chrome updated from v144 → v146 between sessions. Several short clean sessions on v146, then corruption began permanently. No errors visible in these short pre-corruption sessions.

Verdict: No direct harm visible. Corruption happened between sessions (likely during Chrome update), not during user action.

4. [redacted]

Session: Replay link
Device: Barcelona, CT · OS X · Chrome

12-minute session of fully normal usage: browses Search, views multiple reports (Bangkok Retreat 2026, Claude AI expense, co-working expenses), views receipt attachment, creates new tracked expense. No errors, no loading stalls. This session is likely post-corruption — user may have self-healed by clearing browser data.

Verdict: No harm observed in this session.

Overall Conclusion

IDB corruption does not crash or visibly break the app. The harm is silent and indirect:

Writes to IDB fail silently → data not persisted
retryOperation exhausts retries then resolves with undefined → caller never knows the write failed
User sees occasional "Loading" stalls and dead taps (error-clicks), but no crash or error screen
User likely loses draft data, Onyx state, or queued updates without knowing

This reinforces the need for a degradation + notification strategy — when IDB is unrecoverable, Onyx should degrade to MemoryOnlyProvider and surface a visible indicator so the user knows they're in a degraded state.

Chapter 3 — Addendum 2: Broader Fullstory Analysis (10 sessions with IDB backing store error)

Using Fullstory's segment search for sessions containing console-error: "Internal error opening backing store" in the last 30 days, I found 10 affected sessions. Every single one is Chromium-based.

#	Session	Location	OS	Browser	Length	Errors Start	Behavior Summary
1	Replay	Melbourne, VIC	OS X	Chrome	~10 min	2:14	Logs in, chats, searches, edits expense descriptions. Dead taps on "Create report" and "Add unreported expense" buttons
2	Replay	Surrey, BC	OS X	Chrome	~20 min	0:46	Opens Search, sits idle. No user interaction — pure error loop every 30–75 s burning Sentry quota
3	Replay	St-Bruno, QC	Windows	Chrome	~2 min	0:50	Signs in via email, lands on homepage, goes to Search. Loading indicators everywhere
4	Replay	Universal City, TX	Windows	Chrome	~67 min	~55 min	New user onboarding — creates account, onboarding flow, creates expenses, chats. Errors appear late in a long session
5	Replay	Greenwood Village, CO	OS X	Chrome	~1 min	0:15	Opens report, pinch-gesture, navigates away. Short session
6	Replay	Milwaukee, WI	OS X	Chrome	~59 min	13:03	First 13 min clean. Errors start on page return. Continues editing expenses, descriptions, and tags for 46 more minutes normally
7	Replay	Reno, NV	Windows	Chrome	idle 25+ min	0:18	Opens Concierge DM, sits idle forever. Error loop every 30–60 s. No interaction at all
8	Replay	Norfolk, VA	OS X	Chrome	~71 min	0:00	Errors from first frame. Despite that: searches, views reports, edits descriptions/tags/attendees for 71 min. Heavy dead taps on Save, View details, description fields
9	Replay	Scotts Valley, CA	OS X	Chrome	~1.5 min	0:00	Opens expense, views report. Short session with errors on first second
10	Replay	Las Vegas, NV	Windows	Edge	~7 min	0:13	Signs in, hits errors, browses Settings → Profile → Preferences → Help → Troubleshoot → "Switch to Expensify Classic". Direct evidence of user churn

Key Findings

100% Chromium-based — 9 Chrome + 1 Microsoft Edge (also Chromium engine). Zero Firefox or Safari. Confirms this is a Chromium/LevelDB-specific bug, not a general IDB issue.
Cross-platform: 6 macOS + 4 Windows. Not OS-specific — it's the browser engine that matters.
Session Begin porting the web POC to react native #10 — user churn evidence: User on Edge hits IDB errors, gets frustrated navigating Settings → Help → Troubleshoot, and clicks "Switch to Expensify Classic". This is a real user leaving New Expensify because of this bug.
Sessions Fix spaces #2 and Implement infinite sessions #7 — idle error loops: User opens the app, does nothing, and retryOperation fires storage errors every 30–60 seconds indefinitely. This burns Sentry quota on inactive tabs with zero user value.
Session Use specific lodash modules #8 — 71 min of silent broken state: Errors from first frame, user keeps working productively but experiences heavy dead taps (Save button, description fields, View details). User never sees an error message.
Session Moving files and folders a little #4 — new user onboarding: A brand-new user going through onboarding gets hit with IDB corruption mid-session. First impression of the product includes silent storage failures.

Conclusion

This data strongly confirms the Chromium/LevelDB filesystem hypothesis from Chapter 3. The error is:

Browser-engine-specific: 100% Chromium (Chrome + Edge), 0% Firefox/Safari
Not triggered by user action: Errors appear on idle tabs, on first frame, mid-session — no consistent user trigger
Cross-platform: Both macOS and Windows affected equally
Silently destructive: Users work for 60–70+ minutes without knowing their data isn't being persisted, or they churn to Classic

The fix should focus on detection and graceful degradation within Onyx, since we cannot fix the Chromium bug itself.

My recommendation is to add healing from memory in case of such an issue. It doesn't look that something in Onyx is actually broken - in my opinion the root cause is outside the app lifecycle. The best we can do is to improve resiliency if that happens (see the related PR).

I'll be back on May 13, so someone else will take care about pushing it further.

Reviewing label has been removed, please complete the "BugZero Checklist".

@fabioh8010

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.3.98-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

Bump Onyx to 3.0.79 #92351

If no regressions arise, payment will be issued on 2026-06-11. 🎊

For reference, here are some details about the assignees on this issue:

@fabioh8010 does not require payment (Contractor)

Still held here on the Onyx bump queue.

@fabioh8010

@fabioh8010 Huh... This is 4 days overdue. Who can take care of this?

The Onyx bump hit prod, no? Is there anything left here?

Yes the issue is done!

@fabioh8010

Thanks @fabioh8010! 🙏

[Onyx] Investigate UnknownError: Internal error opening backing store for indexedDB.open. storage error #87862

Description

Summary

Action Items

Activity

leshniak commented on Apr 20, 2026

leshniak commented on Apr 21, 2026

Investigation Report

Summary

Data Analysis (VictoriaLogs, 7-day window)

Session Analysis (top user: [redacted])

Verification of Root Cause

Two Bugs in Onyx

Data Loss Assessment

Chromium Context

Proposed Fix

How the error flows today (broken)

How the error should flow (fixed)

Coordination with backoff PR (#775)

Changes needed

Observability

leshniak commented on Apr 21, 2026

Root Cause Confidence: ~85%

leshniak commented on Apr 21, 2026

leshniak commented on Apr 22, 2026

Investigation Report — Chapter 2: Reproduction & Healing Simulation

Reproduction Setup

Experiment 1: Simple Corruption

Experiment 2: Corruption + Immutable Files

Experiment 3: Chrome Process-Level Caching

Experiment 4: Browser Restart with Corrupt (Deletable) Files

Experiment 5: External File Removal → Healing

Revised Understanding

What This Means for the Fix

Code-Level Test Results

Updated Confidence

fabioh8010 commented on Apr 23, 2026

leshniak commented on Apr 23, 2026

leshniak commented on Apr 23, 2026

Investigation Report — Chapter 3: Why Does Corruption Happen?

Approach

User Pattern Analysis

VictoriaLogs: Pre-Error Context (4 users analyzed)

Fullstory: User Experience During Corruption

Chromium Bug Tracker: Known Issue

Known Corruption Triggers (from Chromium bugs)

Conclusion: Why Does Corruption Happen?

Updated Confidence

fabioh8010 commented on Apr 24, 2026

fabioh8010 commented on Apr 24, 2026

leshniak commented on Apr 28, 2026

Chapter 3 — Addendum: Fullstory Session Evidence

1. [redacted]

2. [redacted]

3. [redacted]

4. [redacted]

Overall Conclusion

leshniak commented on Apr 28, 2026

Chapter 3 — Addendum 2: Broader Fullstory Analysis (10 sessions with IDB backing store error)

Key Findings

Conclusion

leshniak commented on May 1, 2026

37 remaining items

melvin-bot commented on Jun 4, 2026

melvin-bot commented on Jun 4, 2026

Julesssss commented on Jun 5, 2026

melvin-bot commented on Jun 9, 2026

cristipaval commented on Jun 10, 2026

fabioh8010 commented on Jun 10, 2026

cristipaval commented on Jun 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Onyx] Investigate `UnknownError: Internal error opening backing store for indexedDB.open.` storage error #87862

Session Analysis (top user: `[redacted]`)