sqlite3: keep connection handle for in-memory databases across transactions

[BartWaardenburg]

Summary

Fix silent data loss when calling client.transaction() against an in-memory URL (:memory:, file::memory:, file::memory:?cache=private). After the transaction starts, subsequent client.execute(...) calls silently lose all schema and data because the client opens a brand-new, empty in-memory database.

Fixes #229. Fixes #140.

Repro (broken on main)

import { createClient } from "@libsql/client";

const client = createClient({ url: ":memory:" });
await client.execute("CREATE TABLE t (id INTEGER, name TEXT)");
await client.execute("INSERT INTO t VALUES (1, 'a')");

const tx = await client.transaction("write");
await tx.execute("INSERT INTO t VALUES (2, 'b')");
await tx.commit();

await client.execute("SELECT * FROM t");
// LibsqlError: SQLITE_ERROR: no such table: t

Confirmed against @libsql/[email protected] and the current main branch.

Root cause

In packages/libsql-client/src/sqlite3.ts, Sqlite3Client.transaction() unconditionally sets this.#db = null after starting the BEGIN, so the next call to #getDb() opens a new Database. For file-backed databases this is fine — reopening the file gives you the same data. For in-memory URLs the "database" only exists on the open connection; opening a second connection at :memory: gives you a fresh, empty database. Everything from the original connection becomes unreachable.

PR #220 introduced file::memory:?cache=shared as a workaround, but the documented :memory: URL remains broken on main.

Fix

Adopt the model the http and ws clients already use.

There, every client call does openStream() → work → closeGracefully(), and transaction() hands a stream to the transaction, which closes it on commit, rollback and close alike. Nothing is shared between an open transaction and the rest of the client, and no resource outlives the call that borrowed it.

Sqlite3Client now owns a ConnectionPool, sized by config.concurrency — already computed in expandConfig, already used by the remote clients, and until now ignored here. Client calls borrow a connection and return it in a finally; a transaction borrows one and returns it when it settles. A connection is rolled back before it goes back into the pool, so a borrower can never inherit someone else's transaction.

 async transaction(mode: TransactionMode = "write"): Promise<Transaction> {
-    const db = this.#getDb();
+    this.#checkNotClosed();
+    const db = await this.#pool.acquire(true);
     executeStmt(db, transactionModeToBegin(mode), this.#intMode);
-    this.#db = null; // A new connection will be lazily created on next use
-    return new Sqlite3Transaction(db, this.#intMode);
+    return new Sqlite3Transaction(db, this.#intMode, (used) =>
+        this.#pool.release(used),
+    );
 }

This removes special cases rather than adding them: no #db = null, no #getDb(), no isInMemory flag on the client, and no rewriting of the URL the caller asked for.

Why not just keep the handle for :memory:

The first commit does exactly that, and it fixes the reported bug. But this.#db = null is not only wrong for in-memory — Sqlite3Transaction never closes the connection it was handed and the client has already dropped the reference, so every transaction against a file leaks one:

open fds after 50 committed transactions
  before: 50
  after:   1

Keeping the handle for :memory: alone leaves that leak in place and puts the two database kinds on different models.

Single-connection databases

An in-memory database exists only on the connection that opened it, so a second connection would be a second, empty database rather than another way into the same one. An in-memory client therefore has a pool of exactly one. Embedded replicas likewise, as each connection carries its own sync state.

A single connection cannot serve a client call while a transaction holds it, and waiting would hang — only the caller can end that transaction. So the pool tracks which borrows are transactions and refuses such a call immediately:

LibsqlError: TRANSACTION_ACTIVE: This client has a single connection, which an
open transaction is holding. In-memory databases and embedded replicas cannot
have more than one. Commit or roll back the transaction before using the
client again.

Every other borrow is a short synchronous call that returns the connection before the caller regains control, so those still queue — Promise.all([...reads]) against :memory: keeps working.

Behaviour

Measured, during an open write transaction:

	:memory:	file
read an untouched table	TRANSACTION_ACTIVE	ok
read the written table	TRANSACTION_ACTIVE	ok — pre-transaction snapshot
write, batch, second transaction	TRANSACTION_ACTIVE	SQLITE_BUSY
executeMultiple	TRANSACTION_ACTIVE	ok

File-backed matches the hrana clients: concurrent isolated reads, serialized writers. An in-memory client refuses overlapping work rather than pretending to a concurrency the database does not have.

Changed behaviour to note in the release notes: client.execute(...) during an open transaction no longer joins that transaction. For files it runs on another connection, as it did before. For :memory: it is refused, where previously it silently ran against a throwaway empty database.

Tests

The tests run against an in-memory and a file-backed client from one table (describe.each); both helpers ignore $URL, so they run in every CI configuration. Cases that need a second connection are targeted at file-backed, since an in-memory client has only one.

Both kinds:

• client work during a transaction survives that transaction's rollback
• 50 sequential transactions complete — only possible if each returns its connection
• a connection left mid-transaction is not handed on in that state
• a bare BEGIN through client.execute does not span calls
• client.close() with an open transaction does not crash
• client.close() and client.reconnect() settle operations that are waiting for a connection, rather than leaving them pending forever

File-backed:

• client.execute during a transaction runs outside it
• connections held at the same time all see the same database
• a committed write is visible on the other pooled connections
• a read during a transaction sees the pre-transaction state

In-memory:

• client calls during an open transaction are refused, not hung, and the caller's transaction is untouched by the refusal
• client calls with no transaction involved still queue behind each other
• file::memory:?cache=private survives a transaction
• separate in-memory clients do not share a database

Verification

Rebased onto main at 0.17.4.

npm run typecheck                            # passed
npm run build                                # passed
prettier --check                             # clean
URL=":memory:"         npx jest --runInBand  # 199 passed, 9 skipped
URL="file:/tmp/x.db"   npx jest --runInBand  # 197 passed, 11 skipped

Skipped tests need the Python hrana server.

Notes

• A pool buys no throughput here — the bindings are synchronous, so four concurrent queries against a file take 4× one query (345ms → 1354ms). Connections exist for isolation, not parallelism.
• Local databases are not in WAL mode (journal_mode is delete), which is why a file-backed write blocks even against a deferred read transaction. Changing that is its own PR.

Follow-ups (not in this PR)

• packages/libsql-client-wasm/src/wasm.ts:199 has the identical this.#db = null line and the identical leak.
• reconnect() on an in-memory database still opens a fresh connection.
• Running the whole suite with URL="file::memory:?cache=shared" fails on table t already exists, because that database is process-global and tests collide. Pre-existing on main; not a CI configuration.

[penberg]

Thanks @BartWaardenburg! I reworked the fix a bit to keep the in-memory and file-based paths symmetric.

[BartWaardenburg]

Thanks for having a look @penberg. Keep up the lovely work with Turso!

[penberg]

I now release 0.18.0 with this fix. Thanks again @BartWaardenburg!