Repository navigation
Remote files: send the content as raw bytes instead of base64 (~1.3x faster) - #186
Merged
bertysentry merged 1 commit intoSep 29, 2026
Conversation
The read, digest and download-probe scripts now write the bytes raw to [Console]::OpenStandardOutput(), 64 KiB per write, and the client takes stdout as undecoded chunks through a package-private RemoteProcess.readStdoutChunk(). The base64 line decoder is gone. Raw output was verified byte-exact on Windows Server 2008 R2 (PowerShell 2.0), 2019 and 2022: every byte value, files ending in CR or LF, a UTF-8 BOM file, an empty file, and ranges across a block boundary, through every read terminal and against a SHA-256 computed on the host. PowerShell 2.0 adds no byte order mark to the raw stream. A 20 MiB openStream() now takes 10.6-12 s instead of 14-17 s (about 1.3x faster) on all three hosts. The shorter read script also raises the path limit of a read from about 1,450 to about 1,500 characters. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
bertysentry
deleted the
175-remote-file-access-send-file-content-as-raw-bytes-instead-of-base64-33-per-stream
branch
September 29, 2026 09:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #175.
What changes
RemoteFiles: the read script writes the file's bytes raw to[Console]::OpenStandardOutput(), oneWriteper 64 KiB block (two full 32 KiB reads of the WinRM service). The digest and download-probe scripts write their few bytes raw too.RemoteProcess: a package-privatereadStdoutChunk()returns stdout as the undecoded chunks theCommandCursordelivers. Stderr is still decoded text, so error messages keep their accents and the PowerShell 2.0 BOM handling.DecodingStream) is gone, replaced byContentStream, which passes chunks on as they arrive. Errors still come from the exit code, checked at the end of the output: a failure midway still fails the read instead of returning a short one.Byte-exactness
I probed raw output with a stand-alone script before writing any code. After the change, the library itself passed these checks on Windows Server 2008 R2 / PowerShell 2.0 (anaxagore), 2019 (dev-hv-01) and 2022 (tc-win2022):
readBytes,readText,openStream,openReader,downloadTo) plusdigest(), compared with a SHA-256 computed on the host by a separate plain PowerShell command;\r, a lone\n, a UTF-8 BOM file ending in\r\n, an empty file, and ranges across the 64 KiB block boundary (a 100-byte range, a tail, a two-block stream range).The WinRS shell translates nothing: no CR/LF conversion, no code-page effect. PowerShell 2.0 puts no BOM on the raw stream, since it only adds one through its own text writer, which these scripts never use. No host needed base64, so it is dropped entirely rather than kept as a fallback.
Measured (HTTP + NTLM encryption, 20 MiB
openStream(), SHA-verified, raw and base64 runs alternated)That is about 1.3× faster everywhere. The ≥ 1.9 MB/s target is met on 2022 and in the best 2008 R2 run, and missed by up to 9% in the other runs on 2019 and 2008 R2. The totals include the time to the first byte (0.5–0.75 s, mostly PowerShell startup). From there on, the transfer runs at 1.84–2.04 MB/s on those two hosts. That is the host's ceiling of 32 KiB per read at about 60 reads per second, so no block shape can do better on one stream (#176 is the way past it).
Downloads (probe included): 20 MiB in 11.6–12.2 s on 2022 and 2008 R2, against 16.2 s and 16.8 s with base64 the same day (one of four 2008 R2 runs was an outlier at 14.1 s). 13.0 s on 2019. 64 MiB in 34.6 s on 2022, where the docs measured 46 s with base64.
Notes
digest()is unchanged". Its output format did change (raw instead of one base64 line), and so did the probe's. With every content script writing raw bytes, the client needs only one output path, and the base64 decoder could be deleted outright.unexpectedOutputIsReportedis deleted. Raw output has no format that stray text could break. A download still verifies size and SHA-256, andaMalformedProbeIsReportedstill covers a probe of the wrong length.RemoteFileTest), and the same cut in the CLI'scattest. The local-PowerShell script test now reads raw stdout, and its file ends with a CR.files.md(mechanism, requirements, path limit, Read performance),file-transfers.md(Download performance),cli.md(cat/getspeed and the 60 s download size). README unchanged, per AGENTS.md.mvn clean verify sitegreen on JDK 17.🤖 Generated with Claude Code