Skip to content

Prevent offline flush from outliving storage - #1567

Open
bmehta001 wants to merge 1 commit into
microsoft:mainfrom
bmehta001:recover-offline-flush-lifecycle
Open

bmehta001 wants to merge 1 commit into
microsoft:mainfrom
bmehta001:recover-offline-flush-lifecycle

Conversation

@bmehta001

Copy link
Copy Markdown
Contributor

Summary

Restore the offline-flush safeguards from 33d05033 that were superseded during the merge into #1520, while retaining subsequent batching and provider-injection changes.

  • Drain admitted flush/store operations before closing storage, invalidate or cancel queued work by generation, and retry failed shutdown stages. Reinitialization discards closed storage before creating the new generation.
  • Cancel immediate worker-queue tasks as well as timed tasks and keep saved-record notifications outside the flush I/O lock.
  • Retain the original batch when a provider mutates its input, throws, or returns a partial count; cover lifecycle, dispatcher, batching, and retry cases.
  • Use MSVC /bigobj only for OfflineStorageTests.cpp, whose regression tests exceed the normal COFF section limit.

Validation

All selected tests had no disabled cases. UnitTests emitted one pre-existing uninteresting gMock DispatchEvent warning on both platforms.

Configuration and command Result
Windows VS 2026: cmake --build out-vs2026 --config Debug --target UnitTests --parallel 3 and cmake --build out-vs2026 --config Debug --target FuncTests --parallel 3 Passed
Windows: out-vs2026\tests\unittests\Debug\UnitTests.exe --gtest_filter=OfflineStorageLifecycleTests.*:OfflineStorageHandlerFlushTests.*:OfflineStorageTests.*:*OfflineStorageRetrievalExceptionTests*.*:*TaskDispatcher*CAPI*.*:PalTests.Cancel*:PalTests.Schedule*:PalTests.WorkerThread* --gtest_color=no 69 passed
Windows WinInet: out-vs2026\tests\functests\Debug\FuncTests.exe --gtest_filter=BasicFuncTests.doNothing:BasicFuncTests.killIsTemporary:BasicFuncTests.sendManyRequestsAndCancel --gtest_repeat=10 --gtest_color=no 30 passed; local collector, isolated TEMP/TMP, 600-second bound
Ubuntu/WSL: cmake -DMATSDK_BUILD_FUNC_TESTS=ON . and cmake --build . --target UnitTests FuncTests --parallel 2 in the Ninja build directory Passed
Ubuntu/WSL: timeout 120s env PATH=. TMPDIR=. UnitTests --gtest_filter=OfflineStorageLifecycleTests.*:OfflineStorageHandlerFlushTests.*:OfflineStorageTests.*:*OfflineStorageRetrievalExceptionTests*.*:*TaskDispatcher*CAPI*.*:PalTests.Cancel*:PalTests.Schedule*:PalTests.WorkerThread* --gtest_color=no 69 passed
Ubuntu/WSL curl: timeout 120s env PATH=. TMPDIR=. FuncTests --gtest_filter=BasicFuncTests.doNothing:BasicFuncTests.killIsTemporary --gtest_repeat=2 --gtest_color=no 4 passed; local collector. The cancellation stress test is Windows-only.
Windows no-exceptions: ctest --test-dir <no-exceptions-build> -C Debug --output-on-failure -R '^no-exceptions-smoke$' 1 passed; /EHs-c- and _HAS_EXCEPTIONS=0
Ubuntu/WSL no-exceptions: ctest --output-on-failure -R '^no-exceptions-smoke$' in the no-exceptions build directory 1 passed; -fno-exceptions
git diff --check Passed

An earlier WinInet stress attempt used a 300-second bound and timed out in its tenth iteration after 27 tests passed; the 600-second rerun above completed. The Linux unit build retains the existing -Wno-deprecated-declarations; the modified test source also passed -fsyntax-only with -Wdeprecated-declarations. Existing Windows bundled SQLite/zlib warning suppression was not changed or independently validated.

Trade-offs and existing limitations

  • Retrying a partially persisted batch is at-least-once and can duplicate records already written. Retaining a retry copy adds temporary memory and copying proportional to one batch (at most 2,000 records plus their payloads).
  • Storage-failure/drop callbacks are still delivered while the flush I/O lock is held; synchronously reentering Flush() from those callbacks can deadlock. Saved notifications were moved outside that lock, but callbacks must not synchronously initiate teardown while their own operation is active.
  • The existing IOfflineStorage lifecycle contract still requires initialization to complete before other methods are called and disallows other operations after shutdown.

Drain generation-scoped flushes before storage shutdown and retain batches when providers mutate their input, preventing stranded teardown and lost retries.

Files: lib\offline\OfflineStorageHandler.cpp, lib\offline\OfflineStorageHandler.hpp, lib\pal\WorkerThread.cpp, tests\unittests\CMakeLists.txt, tests\unittests\OfflineStorageTests.cpp, tests\unittests\PalTests.cpp

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f7e2f450-c2c4-43c5-9fb1-05fa4f7d8e9e
@bmehta001
bmehta001 requested a review from a team as a code owner November 19, 2026 20:06
@bmehta001
bmehta001 enabled auto-merge (squash) November 19, 2026 20:06
@bmehta001
bmehta001 disabled auto-merge November 19, 2026 20:06
@bmehta001
bmehta001 enabled auto-merge (squash) November 19, 2026 20:06
@bmehta001

Copy link
Copy Markdown
Contributor Author

CI note: Build runner contracts (ubuntu-latest) and Public header gate (GCC/Clang) currently fail on unchanged main as well. The failures are the build-runner test's child-process race and the missing uint8_t include in ICdsFactory.hpp; #1562 contains fixes for both. The macOS runner-contract job was cancelled after the matrix failure. These checks are not among the current required-status contexts; required checks are still running.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant