Repository navigation
Yield NULL content for files over SQLite's length limit - #1642
Merged
Merged
Conversation
…th-limit-1104 # Conflicts: # packages/rust/src/vtab.rs
…th-limit-1104 # Conflicts: # packages/rust/src/vtab.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1584
A file longer than SQLite's string length limit (default 1e9 bytes) failed the whole query with
string or blob too big, after reading the file into memory. The documented contract is that a file whose content cannot be read yieldsNULLcontent.Fix: the path-table reads the connection's
SQLITE_LIMIT_LENGTHatxConnectand passes it to the content reader. A file whose stat size is over the limit is never opened; one that grew past it since the stat is read throughtake(limit + 1)and yieldsNULL. The issue repro now exits 0 with 12 MB RSS (was ~1 GB and exit 1).Related: #1585 (same
vtab.rsread_text; FIFO //dev/zerore-open not handled here, though the read cap now bounds the/dev/zerocase's memory).Red
Red run (tests only, head
492176ea): https://github.com/thekevinscott/dirsql/actions/runs/41922948504 --content_length_limit_e2efails withdirsql query: SQLite error: string or blob too big(Linux and Windows);vtab::content_over_the_length_limit_is_null_without_erroring_the_queryfails at the query withTooBig(Windows job, which runs the full suite).Changelog / Migrations
<root>/<pkg>/changelog.d/for each changed package (or:skip-changelogtrailer on a commit with reason)<root>/<pkg>/migrations.d/for each changed package (every subsection but Summary_None._when nothing breaks)E2E Verification
packages/python/e2e-attestations/<branch>.jsonreceipt written ifpackages/pythonchanged (just e2e-attest-python)packages/ts/e2e-attestations/<branch>.jsonreceipt written ifpackages/tschanged (just e2e-attest-ts)testing-conventions e2e attest 'uv run python -m pytest tests/e2e/query_contract_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_transcripts_test.py tests/e2e/fanout_test.py tests/e2e/root_at_construction_test.py tests/e2e/on_file_query_test.py -x -q'(aftermaturin develop --releaseandcargo build --release --features cli);testing-conventions e2e attest 'pnpm test:e2e -- tests/e2e/query-contract.test.ts tests/e2e/fanout.test.ts tests/e2e/root-at-construction.test.ts tests/e2e/query-params.test.ts';cargo test -p dirsql --features cli --test content_length_limit_e2e --test path_table_cli_e2e --test path_table_on_file_e2e --test vtab --test path_table_content_prefetchtruncate -s 1000000001 big.txt, issue query) prints[{"path":"big.txt","t":"null"},{"path":"small.txt","t":"text"}], exit 0.