sqlite: add virtual table support via createModule() by byteforge38 · Pull Request #61544 · nodejs/node

byteforge38 · 2026-01-27T13:58:57Z

Expose SQLite's virtual table API through a new database.createModule(name, options) method, wrapping sqlite3_create_module_v2().

This enables users to create read-only virtual tables backed by JavaScript data sources. The registered module can be used in two ways:

Eponymous table: Query the module name directly (e.g., SELECT * FROM module_name).
Named virtual table: Use CREATE VIRTUAL TABLE t USING module_name.

Hidden columns can be used to pass parameters via table-valued function syntax (e.g., SELECT * FROM module_name(param1, param2)).

The options object accepts:

columns: Array of column definitions (name, type, optional hidden)
rows: Function returning an iterable of row arrays
directOnly: Restrict usage to top-level SQL (default: false)
useBigIntArguments: Pass integer parameters as BigInts (default: false)

Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY. Column names are quoted to prevent SQL injection. An idxNum bitmask is used to correctly map hidden column constraints between xBestIndex and xFilter.

Fixes: #61539

nodejs-github-bot · 2026-01-27T13:59:03Z

Review requested:

@nodejs/sqlite

byteforge38 · 2026-01-27T14:11:38Z

@araujogui Could you please review my pr?

byteforge38 · 2026-01-27T23:41:02Z

Please re-run CI check

byteforge38 · 2026-01-30T14:38:40Z

anyone could review my pr please?

github-actions · 2026-07-28T00:19:01Z

This pull request has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

turbocrime · 2026-08-08T07:33:43Z

i'm interested in this feature. unstale

trivikr · 2026-08-08T12:36:48Z

@byteforge38 Can you sign the first commit using rebase?

It's required as per updated instructions

Your commit must contain the Signed-off-by line with your name and email address as an acknowledgement that you agree to the Developer Certificate of Origin.

It can be done using interactive rebase, picking the specific commit and running git commit --amend --no-edit -s for it.

Once that's done, can you please rebase from main, resolve merge conflicts and force push?

You can also consider closing this PR and creating new one if that's easy.
@TrevorBurnham and I are looking into existing sqlite PRs and helping get reviews/land them.

Expose SQLite's virtual table API through a new `database.createModule(name, options)` method, wrapping `sqlite3_create_module_v2()`. This enables read-only virtual tables backed by JavaScript data sources, usable either as an eponymous table (`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING module_name`. Hidden columns pass parameters using table-valued function syntax (`SELECT * FROM module_name(param1, param2)`). `options` accepts `columns`, `rows`, `directOnly`, and `useBigIntArguments`. Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY, and column names are quoted when building the `sqlite3_declare_vtab()` schema. Rebased from nodejs#61544, which was opened by byteforge38 and became inactive. Changes on top of that work: - xBestIndex passes the constrained hidden-column indices to xFilter through idxStr rather than an int bitmask, which previously aliased for parameter indices at or above the width of an int. - VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match UserDefinedFunction instead of a raw pointer. - xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it close() from inside rows(), an iterator's next(), or a row getter finalized the statement that SQLite was still stepping, crashing the process. - createModule() rejects being called from an authorizer callback. - xClose calls the iterator's return() method so generator `finally` blocks run when SQLite stops stepping early, as it does for LIMIT. Skipped when an exception is already pending, so the original error still reaches the caller. Refs: nodejs#61544 Fixes: nodejs#61539 Co-authored-by: byteforge38 <[email protected]> Signed-off-by: Trevor Burnham <[email protected]> Assisted-by: Claude Opus 5

Expose SQLite's virtual table API through a new `database.createModule(name, options)` method, wrapping `sqlite3_create_module_v2()`. This enables read-only virtual tables backed by JavaScript data sources, usable either as an eponymous table (`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING module_name`. Hidden columns pass parameters using table-valued function syntax (`SELECT * FROM module_name(param1, param2)`). `options` accepts `columns`, `rows`, `directOnly`, and `useBigIntArguments`. Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY, and column names are quoted when building the `sqlite3_declare_vtab()` schema. Rebased from nodejs#61544, which was opened by byteforge38 and became inactive. Changes on top of that work: - xBestIndex passes the constrained hidden-column indices to xFilter through idxStr rather than an int bitmask, which previously aliased for parameter indices at or above the width of an int. - VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match UserDefinedFunction instead of a raw pointer. - xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it close() from inside rows(), an iterator's next(), or a row getter finalized the statement that SQLite was still stepping, crashing the process. - createModule() rejects being called from an authorizer callback. - xClose calls the iterator's return() method so generator `finally` blocks run when SQLite stops stepping early, as it does for LIMIT. Skipped when an exception is already pending, so the original error still reaches the caller. - Documents that values yielded by rows() follow the usual conversion rules, so a number is stored as REAL and a BigInt as INTEGER even when a column declares INTEGER, since virtual tables do not apply column affinity to the values they return. Refs: nodejs#61544 Refs: nodejs#63826 Fixes: nodejs#61539 Co-authored-by: byteforge38 <[email protected]> Signed-off-by: Trevor Burnham <[email protected]> Assisted-by: Claude Opus 5

Expose SQLite's virtual table API through a new `database.createModule(name, options)` method, wrapping `sqlite3_create_module_v2()`. This enables read-only virtual tables backed by JavaScript data sources, usable either as an eponymous table (`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING module_name`. Hidden columns pass parameters using table-valued function syntax (`SELECT * FROM module_name(param1, param2)`). `options` accepts `columns`, `rows`, `directOnly`, and `useBigIntArguments`. Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY, and column names are quoted when building the `sqlite3_declare_vtab()` schema. Rebased from nodejs#61544, which was opened by byteforge38 and became inactive. Changes on top of that work: - xColumn reports the value each hidden column was constrained to, rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it may recheck a constraint it already handed to xFilter; against NULL that recheck rejected every row, and `gs(1, 3) WHERE start = 1` returned no rows. - xBestIndex lowers estimatedCost as it consumes constraints. With a constant cost the planner was free to pick the unconstrained plan and recheck afterwards, so a correlated parameter such as `FROM t, gs(t.a, t.a + 1)` also returned no rows. - Violations of the iteration protocol report a SQLite error instead of calling PropagateJSError with no JavaScript exception pending. That left `.all()` returning undefined and `exec()` reporting success. - xBestIndex passes the constrained hidden-column indices to xFilter through idxStr rather than an int bitmask, which previously aliased for parameter indices at or above the width of an int. - xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it close() from inside rows(), an iterator's next(), or a row getter finalized the statement that SQLite was still stepping, crashing the process. - xClose calls the iterator's return() method so generator `finally` blocks run when SQLite stops stepping early, as it does for LIMIT or a `break` out of a for...of loop. It is skipped while tearing down from ~StatementSync or ~DatabaseSync, which run from garbage collection callbacks where JavaScript cannot be executed; an abandoned generator does not run `finally` in JavaScript either. It is also skipped when an error is already pending, so that error still reaches the caller. - VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match UserDefinedFunction instead of a raw pointer. - createModule() rejects being called from an authorizer callback. - Documents that values yielded by rows() follow the usual conversion rules, so a number is stored as REAL and a BigInt as INTEGER even when a column declares INTEGER, since virtual tables do not apply column affinity to the values they return. Refs: nodejs#61544 Refs: nodejs#63826 Fixes: nodejs#61539 Co-authored-by: byteforge38 <[email protected]> Signed-off-by: Trevor Burnham <[email protected]> Assisted-by: Claude Opus 5

TrevorBurnham · 2026-09-04T23:18:19Z

I've fixed up this PR and re-submitted it as #65787.

trivikr · 2026-09-05T03:35:29Z

Closing as updated PR is posted at #65787

sqlite: add virtual table support via createModule()

7b6bae9

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem. labels Jan 27, 2026

louwers reviewed Jan 27, 2026

View reviewed changes

Comment thread src/node_sqlite.cc Outdated

byteforge38 added 2 commits January 27, 2026 10:24

sqlite: heap-allocate sqlite3_module per virtual table module

04c015f

sqlite: fix clang-format in virtual table code

cf6f98e

github-actions Bot added the stale Issues and PRs marked stale due to inactivity and scheduled for automatic closure. label Jul 28, 2026

github-actions Bot removed the stale Issues and PRs marked stale due to inactivity and scheduled for automatic closure. label Aug 9, 2026

TrevorBurnham mentioned this pull request Sep 4, 2026

sqlite: add virtual table support via createModule() #65787

Open

trivikr closed this Sep 5, 2026

Navigation Menu

Uh oh!

Uh oh!

sqlite: add virtual table support via createModule() - #61544

sqlite: add virtual table support via createModule()#61544
byteforge38 wants to merge 3 commits into
nodejs:mainfrom
byteforge38:sqlite-virtual-table-api

byteforge38 commented Jan 27, 2026

Uh oh!

nodejs-github-bot commented Jan 27, 2026

Uh oh!

byteforge38 commented Jan 27, 2026

Uh oh!

Uh oh!

byteforge38 commented Jan 27, 2026

Uh oh!

byteforge38 commented Jan 30, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

turbocrime commented Aug 8, 2026 •

edited

Loading

Uh oh!

trivikr commented Aug 8, 2026

Uh oh!

TrevorBurnham commented Sep 4, 2026

Uh oh!

trivikr commented Sep 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Uh oh!

Conversation

byteforge38 commented Jan 27, 2026

Uh oh!

nodejs-github-bot commented Jan 27, 2026

Uh oh!

byteforge38 commented Jan 27, 2026

Uh oh!

Uh oh!

byteforge38 commented Jan 27, 2026

Uh oh!

byteforge38 commented Jan 30, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

turbocrime commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trivikr commented Aug 8, 2026

Uh oh!

TrevorBurnham commented Sep 4, 2026

Uh oh!

trivikr commented Sep 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

turbocrime commented Aug 8, 2026 •

edited

Loading