Repository navigation
Database: WAL, batched clears of deletion records, and giving free space back - #53
Open
ChakraFusion wants to merge 3 commits into
Open
ChakraFusion wants to merge 3 commits into
ChakraFusion wants to merge 3 commits into
Conversation
opensave.db had grown to 663 MB on one device, 262 MB of it pages that deleted rows had left: SQLite never shrinks the file on its own, and snapshots - each with a row per file of its save, a quarter of a million for one game - are merged, pruned and replaced all the time. The database is switched to incremental auto-vacuum, which takes one full VACUUM when it is opened (8 seconds for that database, 663 to 380 MB). After that its free pages are handed back a few at a time - when it is opened and hourly - so deleting a snapshot makes the file smaller. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Putting Project Zomboid back after its folder was emptied spent most of an hour in the database, the save untouched meanwhile: every file's deletion record was cleared with a statement of its own, a quarter of a million of them, and with SQLite's default rollback journal every one of them created, flushed and deleted a journal file. - Deletion records are cleared in one transaction per save location, on putting a save back and when files come back in a snapshot (ClearDeletedFiles): 50,000 take a third of a second. - The database runs in WAL mode with synchronous NORMAL: writes are appended to one log instead of each making a journal file of its own. The database stays whole whatever happens; a power cut can cost only the last moments' writes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
One statement per path, even inside one transaction, took 10s for 50,000 records under the race detector on CI's runners, and the test's limit failed. 500 paths to a statement: 50,000 records in about 0.3s here. The test's limit is generous now (30s): what it guards against took minutes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As offered in #21. Three commits:
VACUUMwhen the database is opened, then free pages handed back at open and hourly. On one PC, 663 MB became 380 MB.synchronous=NORMAL, and deletion records cleared in one transaction when a save is put back or files reappear. Putting a 240k-file save back had spent most of an hour creating and deleting a rollback journal per row.DELETE: 50,000 records in about 0.3 s, versus 10 s with one statement per path under the race detector.Nothing copies
opensave.dbdirectly; everything goes throughstore.Open, so the-walfile needs no special handling.Tests: the database uses WAL; compaction gives space back (using a table of the test's own); clearing 50,000 records is one transaction and fast.
🤖 Generated with Claude Code