Skip to content

Yield NULL content for files over SQLite's length limit - #1642

Merged
thekevinbot merged 8 commits into
mainfrom
bot/1584-content-length-limit-1104
Nov 5, 2026
Merged

thekevinbot merged 8 commits into
mainfrom
bot/1584-content-length-limit-1104

Conversation

@thekevinbot

@thekevinbot thekevinbot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1584

A file longer than SQLite's string length limit (default 1e9 bytes) failed the whole query with string or blob too big, after reading the file into memory. The documented contract is that a file whose content cannot be read yields NULL content.

Fix: the path-table reads the connection's SQLITE_LIMIT_LENGTH at xConnect and passes it to the content reader. A file whose stat size is over the limit is never opened; one that grew past it since the stat is read through take(limit + 1) and yields NULL. The issue repro now exits 0 with 12 MB RSS (was ~1 GB and exit 1).

Related: #1585 (same vtab.rs read_text; FIFO / /dev/zero re-open not handled here, though the read cap now bounds the /dev/zero case's memory).

Red

Red run (tests only, head 492176ea): https://github.com/thekevinscott/dirsql/actions/runs/41922948504 -- content_length_limit_e2e fails with dirsql query: SQLite error: string or blob too big (Linux and Windows); vtab::content_over_the_length_limit_is_null_without_erroring_the_query fails at the query with TooBig (Windows job, which runs the full suite).

Changelog / Migrations

  • Changelog fragment added under <root>/<pkg>/changelog.d/ for each changed package (or: skip-changelog trailer on a commit with reason)
  • Migration fragment with the same slug added under <root>/<pkg>/migrations.d/ for each changed package (every subsection but Summary _None._ when nothing breaks)

E2E Verification

  • Ran e2e suites locally for every affected SDK
  • Python SDK e2e: pass
  • TypeScript SDK e2e: pass
  • Rust core e2e (if applicable): pass
  • packages/python/e2e-attestations/<branch>.json receipt written if packages/python changed (just e2e-attest-python)
  • packages/ts/e2e-attestations/<branch>.json receipt written if packages/ts changed (just e2e-attest-ts)
  • Command(s) run: testing-conventions e2e attest 'uv run python -m pytest tests/e2e/query_contract_test.py tests/e2e/use_case_join_test.py tests/e2e/use_case_transcripts_test.py tests/e2e/fanout_test.py tests/e2e/root_at_construction_test.py tests/e2e/on_file_query_test.py -x -q' (after maturin develop --release and cargo build --release --features cli); testing-conventions e2e attest 'pnpm test:e2e -- tests/e2e/query-contract.test.ts tests/e2e/fanout.test.ts tests/e2e/root-at-construction.test.ts tests/e2e/query-params.test.ts'; cargo test -p dirsql --features cli --test content_length_limit_e2e --test path_table_cli_e2e --test path_table_on_file_e2e --test vtab --test path_table_content_prefetch
  • Result summary: Python 18 passed; TS 14 passed; Rust all passed. Hand repro (truncate -s 1000000001 big.txt, issue query) prints [{"path":"big.txt","t":"null"},{"path":"small.txt","t":"text"}], exit 0.

@thekevinbot
thekevinbot merged commit 99885a6 into main Nov 5, 2026
89 checks passed
@thekevinbot
thekevinbot deleted the bot/1584-content-length-limit-1104 branch November 5, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Yield NULL content for files over SQLite's length limit, not a query error

1 participant