Skip to content

Remote files: send the content as raw bytes instead of base64 (~1.3x faster) - #186

Merged
bertysentry merged 1 commit into
mainfrom
175-remote-file-access-send-file-content-as-raw-bytes-instead-of-base64-33-per-stream
Sep 29, 2026
Merged

bertysentry merged 1 commit into
mainfrom
175-remote-file-access-send-file-content-as-raw-bytes-instead-of-base64-33-per-stream

Conversation

@bertysentry

Copy link
Copy Markdown
Contributor

Closes #175.

What changes

  • RemoteFiles: the read script writes the file's bytes raw to [Console]::OpenStandardOutput(), one Write per 64 KiB block (two full 32 KiB reads of the WinRM service). The digest and download-probe scripts write their few bytes raw too.
  • RemoteProcess: a package-private readStdoutChunk() returns stdout as the undecoded chunks the CommandCursor delivers. Stderr is still decoded text, so error messages keep their accents and the PowerShell 2.0 BOM handling.
  • The base64 line decoder (DecodingStream) is gone, replaced by ContentStream, which passes chunks on as they arrive. Errors still come from the exit code, checked at the end of the output: a failure midway still fails the read instead of returning a short one.
  • No API change.

Byte-exactness

I probed raw output with a stand-alone script before writing any code. After the change, the library itself passed these checks on Windows Server 2008 R2 / PowerShell 2.0 (anaxagore), 2019 (dev-hv-01) and 2022 (tc-win2022):

  • every read terminal (readBytes, readText, openStream, openReader, downloadTo) plus digest(), compared with a SHA-256 computed on the host by a separate plain PowerShell command;
  • the test files: every byte value 0–255 plus random data ending in \r, a lone \n, a UTF-8 BOM file ending in \r\n, an empty file, and ranges across the 64 KiB block boundary (a 100-byte range, a tail, a two-block stream range).

The WinRS shell translates nothing: no CR/LF conversion, no code-page effect. PowerShell 2.0 puts no BOM on the raw stream, since it only adds one through its own text writer, which these scripts never use. No host needed base64, so it is dropped entirely rather than kept as a fallback.

Measured (HTTP + NTLM encryption, 20 MiB openStream(), SHA-verified, raw and base64 runs alternated)

Host raw (this PR) base64 (main)
2022 10.6–11.0 s (1.91–1.98 MB/s) 14.1–14.7 s (1.43–1.48 MB/s)
2019 11.0–12.1 s (1.74–1.90 MB/s) 14.6–16.6 s (1.26–1.44 MB/s)
2008 R2 10.8–11.9 s (1.77–1.94 MB/s) 13.7–17.1 s (1.23–1.53 MB/s)

That is about 1.3× faster everywhere. The ≥ 1.9 MB/s target is met on 2022 and in the best 2008 R2 run, and missed by up to 9% in the other runs on 2019 and 2008 R2. The totals include the time to the first byte (0.5–0.75 s, mostly PowerShell startup). From there on, the transfer runs at 1.84–2.04 MB/s on those two hosts. That is the host's ceiling of 32 KiB per read at about 60 reads per second, so no block shape can do better on one stream (#176 is the way past it).

Downloads (probe included): 20 MiB in 11.6–12.2 s on 2022 and 2008 R2, against 16.2 s and 16.8 s with base64 the same day (one of four 2008 R2 runs was an outlier at 14.1 s). 13.0 s on 2019. 64 MiB in 34.6 s on 2022, where the docs measured 46 s with base64.

Notes

  • Deviation from the issue: the issue says "digest() is unchanged". Its output format did change (raw instead of one base64 line), and so did the probe's. With every content script writing raw bytes, the client needs only one output path, and the base64 decoder could be deleted outright.
  • The read script is 80 characters shorter, so the path limit of a read rises from 1,461 to 1,524 characters (docs: "about 1,500"). Digest and probe barely move (1,656 and 1,617), and both stay above the read's limit.
  • unexpectedOutputIsReported is deleted. Raw output has no format that stray text could break. A download still verifies size and SHA-256, and aMalformedProbeIsReported still covers a probe of the wrong length.
  • New or updated tests: chunks cut in the middle of a block, with an empty Receive in between (RemoteFileTest), and the same cut in the CLI's cat test. The local-PowerShell script test now reads raw stdout, and its file ends with a CR.
  • Docs: files.md (mechanism, requirements, path limit, Read performance), file-transfers.md (Download performance), cli.md (cat/get speed and the 60 s download size). README unchanged, per AGENTS.md.

mvn clean verify site green on JDK 17.

🤖 Generated with Claude Code

The read, digest and download-probe scripts now write the bytes raw to
[Console]::OpenStandardOutput(), 64 KiB per write, and the client takes
stdout as undecoded chunks through a package-private
RemoteProcess.readStdoutChunk(). The base64 line decoder is gone.

Raw output was verified byte-exact on Windows Server 2008 R2
(PowerShell 2.0), 2019 and 2022: every byte value, files ending in CR
or LF, a UTF-8 BOM file, an empty file, and ranges across a block
boundary, through every read terminal and against a SHA-256 computed on
the host. PowerShell 2.0 adds no byte order mark to the raw stream.

A 20 MiB openStream() now takes 10.6-12 s instead of 14-17 s (about
1.3x faster) on all three hosts. The shorter read script also raises
the path limit of a read from about 1,450 to about 1,500 characters.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T22:09:46.263764Z cfc4cd7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bertysentry
bertysentry merged commit 36757ae into main Sep 29, 2026
5 checks passed
@bertysentry
bertysentry deleted the 175-remote-file-access-send-file-content-as-raw-bytes-instead-of-base64-33-per-stream branch September 29, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote file access: send file content as raw bytes instead of base64 (~+33% per stream)

1 participant