Skip to content

Stream data from Memory functionality - #90

Open
psharma-1909 wants to merge 2 commits into
masterfrom
psharma/add-load-data-support
Open

psharma-1909 wants to merge 2 commits into
masterfrom
psharma/add-load-data-support

Conversation

@psharma-1909

@psharma-1909 psharma-1909 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Aligns singlestore-nodejs with SingleStore Load Data from Local Files: LOAD DATA LOCAL INFILE already worked via infileStreamFactory; this PR adds in-memory streaming, an explicit local_infile connection option, fixes factory inheritance for prepared statements and pool cluster, and documents local-file vs in-memory loads.

Changes

Features

  • infileStream (query option) — Stream bytes from a Node.js Readable for LOAD DATA LOCAL INFILE. The filename in SQL is ignored. Takes precedence over infileStreamFactory.
  • local_infile (connection option) — Controls CLIENT_LOCAL_FILES (default true). When false, file and stream loads fail with a clear client error; relevant server errors (1148 / 3948 / 4166) are rewritten when appropriate.
  • lib/commands/local_infile.js — Shared helpers for option assignment, disabled errors, and error rewriting.
  • Query / Execute — Shared local-infile handling; connection-level factory used when the query does not set one.
  • PreparedStatementInfo.execute — Passes connection infileStreamFactory into Execute.
  • PoolCluster.query — Inherits connection-level infileStreamFactory when omitted on the query.

Types

  • ConnectionOptions: local_infile
  • QueryOptions: infileStream

Tests

  • Unit: test/unit/commands/test-local-infile.test.cjs, test/unit/connection/test-connection_config.test.cjs, test/unit/pool-cluster/test-load-infile-factory.test.cjs
  • Integration: extended test/integration/connection/test-load-infile.test.cjs (memory stream, local_infile: false)

Documentation

  • README, site docs (extras, quickstart, API/config, indexes), Changelog unreleased 1.0.2, QueryOptions embed fixtures

Note

Medium Risk
Changes LOAD DATA LOCAL INFILE behavior (new in-memory path and client-side disable), which affects bulk ingest and a historically sensitive protocol feature, though defaults stay permissive and opt-out is explicit.

Overview
Adds in-memory bulk load for LOAD DATA LOCAL INFILE via a new query option infileStream (a Node Readable). When set, uploaded bytes come from that stream and the filename in SQL is ignored; infileStream wins over infileStreamFactory.

Introduces local_infile on the connection (default true), wired to CLIENT_LOCAL_FILES, with string/boolean parsing and interaction with flags. When disabled, loads fail early with a clear connector message, and selected server errors (1148 / 3948 / 4166) are rewritten for LOCAL INFILE SQL via shared lib/commands/local_infile.js.

Query and Execute share the same option assignment and streaming path (connection-level factory when the query omits one). Prepared statement execute and PoolCluster.query now propagate connection infileStreamFactory where it was missing before.

TypeScript adds local_infile and infileStream; README and site docs document file vs memory loads and fix singlestore-nodejs require examples. Unit and integration tests cover memory loads, disabled infile, config flags, and pool cluster inheritance.

Reviewed by Cursor Bugbot for commit ea2ab39. Bugbot is set up for automated code reviews on this repo. Configure here.

@psharma-1909
psharma-1909 requested a review from a team as a code owner October 28, 2026 12:15

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ea2ab39. Configure here.

Comment thread lib/commands/execute.js
this._localStream = null;
this._unpipeStream = function () {};
this._streamFactory = options.infileStreamFactory;
assignLocalInfileOptions(this, options);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Execute skips LOCAL INFILE completion

High Severity

Execute now accepts infileStream and a connection infileStreamFactory, but it never copies infileOk from Query. After _streamLocalInfile sends file bytes it returns undefined, so the command emits end before the server OK. That desynchronizes the connection and can release a pooled connection while the stream is still writing.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ea2ab39. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Cursor Bugbot was skipped and Cursor Security Agent did not run, so those signals were not used; no approval policy required human review and this automation had no prior approval. No reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant