Remote files: send the content as raw bytes instead of base64 (~1.3x faster)

src/main/java/org/metricshub/winrm/RemoteFile.java

@@ -72,15 +72,16 @@
  * }</pre>
  * <p>
  * The content travels through the WinRM connection itself (no SMB, no extra port): a small
- * PowerShell script on the host writes the bytes base64-encoded, which is binary-safe and
- * independent of the remote console code page. It is designed for configuration files, logs and
- * small data files — <b>not a bulk transport</b>: base64 through a command shell is far slower
- * than SMB. The host needs PowerShell (2.0 or later) in {@code FullLanguage} mode.
+ * PowerShell script on the host writes the file's raw bytes to its standard output stream, past
+ * any text conversion, so every byte arrives as stored whatever the remote console code page. It
+ * is designed for configuration files, logs and small data files — <b>not a bulk transport</b>:
+ * the WinRM service forwards a command's output at about 2 MB/s, far slower than SMB. The host
+ * needs PowerShell (2.0 or later) in {@code FullLanguage} mode.
  * <p>
  * The file is opened with a share mode that tolerates other writers, so a log being written by a
  * running service can be read; a file held with an exclusive lock (e.g. {@code pagefile.sys})
  * fails with a sharing violation. A read writes nothing on the host: its script travels on the
- * command line, which limits the path to about 1,450 characters (fewer with non-Latin
+ * command line, which limits the path to about 1,500 characters (fewer with non-Latin
  * characters) — a longer path fails before anything is sent. Ranges are <b>byte</b> ranges, and
  * reads are not snapshots: a
  * file that grows or shrinks between two reads is read as it is at each read.
@@ -257,10 +258,10 @@ public String readText(final Charset charset) {
 	}
 
 	/**
-	 * Open the content (the whole file, or the configured range) as a stream, decoded as it
-	 * arrives: memory is bounded by one transfer block, not by the file, and there is no size cap.
-	 * The file is opened before this method returns, so a file that cannot be read fails here; a
-	 * failure midway is reported by {@code read()} (never as a silently short read).
+	 * Open the content (the whole file, or the configured range) as a stream, passed on as it
+	 * arrives: memory is bounded by one protocol response, not by the file, and there is no size
+	 * cap. The file is opened before this method returns, so a file that cannot be read fails here;
+	 * a failure midway is reported by {@code read()} (never as a silently short read).
 	 * <p>
 	 * <b>The stream must be closed</b> — use try-with-resources. It holds the client's connection
 	 * until it reaches its end or is closed; closing it early stops the remote read. Failures are
@@ -314,7 +315,7 @@ public BufferedReader openReader(final Charset charset) {
 	 * {@link #maxBytes(long)} settings do not apply.
 	 * <p>
 	 * The timeout is a wall-clock deadline for the whole download, and the transfer runs at about
-	 * 1.5 MB/s: a large file needs a raised {@link #timeout(Duration)}. A deadline that fires while
+	 * 1.8 MB/s: a large file needs a raised {@link #timeout(Duration)}. A deadline that fires while
 	 * the verified file is being moved onto the destination lets that move complete: the download
 	 * then succeeds. Downloads are not resumable — one that fails or times out starts over.
 	 *

src/main/java/org/metricshub/winrm/RemoteFiles.java

@@ -38,15 +38,17 @@
 
 /**
  * The remote file access primitive: small PowerShell scripts that open a remote file and write
- * what the caller asked for as <b>base64 lines</b> on stdout, and the {@link InputStream} that
- * decodes those lines as the output chunks arrive. The metadata scripts ({@link #infoScript},
- * {@link #listScript}) write one ASCII record per entry instead, parsed by {@link #parseEntry}
- * and {@link #parseInaccessible}: plain integers, and only the path base64-encoded.
+ * what the caller asked for (its bytes, its digest) <b>raw</b> on stdout, and the
+ * {@link InputStream} that passes those bytes on as the output chunks arrive. The metadata scripts
+ * ({@link #infoScript}, {@link #listScript}) write one ASCII record per entry instead, parsed by
+ * {@link #parseEntry} and {@link #parseInaccessible}: plain integers, and only the path
+ * base64-encoded.
  * <p>
- * Base64 is what makes a text console a binary-safe channel: every byte value survives, and the
- * output is pure ASCII, so recovering the exact bytes never depends on the remote console code
- * page. Each line is an independent base64 block, decoded on its own. The caller-supplied path is
- * embedded base64-encoded too (UTF-8), so no path ever needs quoting or escaping.
+ * The bytes are written to {@code [Console]::OpenStandardOutput()}, a stream no text writer, code
+ * page or byte order mark touches, and the WinRM service forwards a command's stdout as it is:
+ * every byte value arrives as written (verified on Windows Server 2008 R2 with PowerShell 2.0,
+ * 2019 and 2022). The caller-supplied path is embedded base64-encoded (UTF-8), so no path ever
+ * needs quoting or escaping.
  * <p>
  * Failures are reported by the script's exit code, checked when the output ends (see the
  * {@code EXIT_*} constants), and turned into a {@link WinRMClientException} naming the cause.
@@ -77,14 +79,11 @@ private RemoteFiles() {}
 	static final int EXIT_COMMAND_NOT_FOUND = 9009;
 
 	/**
-	 * Bytes read and written per base64 line: 49,149, a multiple of 3 so a full block encodes
-	 * without padding, into 65,532 characters — with the newline, a 65,533-byte line that fits in
-	 * exactly two of the WinRM service's output reads. The service's shell plugin reads a
-	 * command's stdout pipe with at most one 32 KiB read per timer tick (64 per second at the
-	 * default 15.625 ms), so each line should fill whole reads: a 57 KiB block (a 77,825-byte
-	 * line) needs three, and is measurably slower.
+	 * Bytes read and written per block: 64 KiB, exactly two of the WinRM service's output reads.
+	 * The service's shell plugin reads a command's stdout pipe with at most one 32 KiB read per
+	 * timer tick (64 per second at the default 15.625 ms), so each write should fill whole reads.
 	 */
-	static final int BLOCK_SIZE = 49_149;
+	static final int BLOCK_SIZE = 64 * 1024;
 
 	/**
 	 * The hash algorithms {@link #digestScript(String, String)} accepts: the names .NET's
@@ -193,37 +192,36 @@ private RemoteFiles() {}
 	/**
 	 * Read a byte range: resolve a negative offset from the size of the <i>open</i> stream (so a
 	 * growing log is tailed from its current end), seek, and write at most {@code length} bytes
-	 * ({@code -1}: to the end) as one base64 line per block. Seeking past the end is legal and
-	 * reads nothing. {@code %d} are the offset, the length and the block size.
+	 * ({@code -1}: to the end), block by block. Seeking past the end is legal and reads nothing.
+	 * {@code %d} are the offset, the length and the block size.
 	 * <p>
-	 * Each line goes to the raw stdout stream as ONE write: the WinRM service reads the pipe at
+	 * Each block goes to the raw stdout stream as ONE write: the WinRM service reads the pipe at
 	 * most once per timer tick, taking only what is in the pipe, and {@code [Console]::Out} (an
-	 * auto-flushing writer with a small buffer) cuts a line into 256-byte writes that leave each
+	 * auto-flushing writer with a small buffer) cuts its output into 256-byte writes that leave each
 	 * read with about 4 KiB — 5 to 6 times slower (measured on Windows 2008 R2 and 2022).
 	 */
 	private static final String READ_RANGE = "try{$n=[long]%d;$l=[long]%d;" +
 		"if($n -lt 0){$n=[Math]::Max([long]0,$f.Length+$n)};" +
 		"$f.Position=$n;$b=New-Object byte[] %d;$o=[Console]::OpenStandardOutput();" +
 		"while($l -ne 0){$c=$b.Length;if($l -gt 0 -and $l -lt $c){$c=[int]$l};" +
 		"$r=$f.Read($b,0,$c);if($r -le 0){break};" +
-		"$a=[Text.Encoding]::ASCII.GetBytes([Convert]::ToBase64String($b,0,$r)+\"`n\");" +
-		"$o.Write($a,0,$a.Length);if($l -gt 0){$l-=$r}};" +
+		"$o.Write($b,0,$r);if($l -gt 0){$l-=$r}};" +
 		"$o.Flush()}catch{fail $_.Exception}finally{$f.Close()}";
 
-	/** Hash the whole file and write the raw digest as one base64 line. {@code %s} is the algorithm. */
+	/** Hash the whole file and write the raw digest. {@code %s} is the algorithm. */
 	private static final String DIGEST = "try{$h=[Security.Cryptography.HashAlgorithm]::Create('%s');" +
-		"[Console]::Out.WriteLine([Convert]::ToBase64String($h.ComputeHash($f)))}catch{fail $_.Exception}finally{$f.Close()}";
+		"$y=$h.ComputeHash($f);$o=[Console]::OpenStandardOutput();$o.Write($y,0,$y.Length)}" +
+		"catch{fail $_.Exception}finally{$f.Close()}";
 
 	/**
-	 * What a download is verified against, as one base64 line of {@link #PROBE_LENGTH} bytes: the
-	 * number of bytes hashed (8 bytes, little-endian), then their SHA-256 digest. The count is the
-	 * stream position after hashing, not the file length read beforehand, so both describe the same
-	 * bytes even when the file changes during the probe.
+	 * What a download is verified against, {@link #PROBE_LENGTH} raw bytes: the number of bytes
+	 * hashed (8 bytes, little-endian), then their SHA-256 digest. The count is the stream position
+	 * after hashing, not the file length read beforehand, so both describe the same bytes even when
+	 * the file changes during the probe.
 	 */
-	private static final String PROBE = "try{$h=[Security.Cryptography.HashAlgorithm]::Create('SHA256');$d=$h.ComputeHash($f);"
-		+
-		"[Console]::Out.WriteLine([Convert]::ToBase64String([byte[]]([BitConverter]::GetBytes($f.Position)+$d)))}" +
-		"catch{fail $_.Exception}finally{$f.Close()}";
+	private static final String PROBE = "try{$h=[Security.Cryptography.HashAlgorithm]::Create('SHA256');" +
+		"$d=$h.ComputeHash($f);$y=[byte[]]([BitConverter]::GetBytes($f.Position)+$d);" +
+		"$o=[Console]::OpenStandardOutput();$o.Write($y,0,$y.Length)}catch{fail $_.Exception}finally{$f.Close()}";
 
 	/** The length of the {@link #probeScript(String)} output: an 8-byte size and a 32-byte SHA-256 digest. */
 	static final int PROBE_LENGTH = 40;
@@ -432,19 +430,19 @@ private static WinRMClientException malformed(final String line, final Throwable
 	}
 
 	/**
-	 * Start the script and return the stream of the bytes it writes. The first line is fetched
+	 * Start the script and return the stream of the bytes it writes. The first bytes are fetched
 	 * before returning, so a file that cannot be opened fails here, not on the first read.
 	 *
 	 * @param client the client to run the script on
 	 * @param path the remote file, for the error messages
-	 * @param script the script, from {@link #readScript} or {@link #digestScript}
+	 * @param script the script, from {@link #readScript}, {@link #digestScript} or {@link #probeScript}
 	 * @param timeout the inactivity timeout of the stream
-	 * @return the decoded stream; it must be closed
+	 * @return the stream; it must be closed
 	 */
 	static InputStream open(final WinRMClient client, final String path, final String script, final Duration timeout) {
 		final RemoteProcess process = start(client, path, script, timeout);
 		try {
-			return new DecodingStream(process, path, client.hostname());
+			return new ContentStream(process, path, client.hostname());
 		} catch (final RuntimeException e) {
 			process.close();
 			throw e;
@@ -641,23 +639,21 @@ static WinRMClientException failure(
 	}
 
 	/**
-	 * The bytes a script writes, decoded line by line as the output arrives: memory is bounded by
-	 * one line, not by the file. The exit code is checked when the output ends, so a failure
-	 * midway is reported instead of looking like a short file.
+	 * The bytes a script writes, passed on chunk by chunk as the output arrives: memory is bounded
+	 * by one protocol response, not by the file. The exit code is checked when the output ends, so
+	 * a failure midway is reported instead of looking like a short file.
 	 */
-	private static final class DecodingStream extends InputStream {
+	private static final class ContentStream extends InputStream {
 
 		private final RemoteProcess process;
-		private final BufferedReader stdout;
 		private final String path;
 		private final String hostname;
 		private byte[] block = new byte[0];
 		private int position;
 		private boolean ended;
 
-		private DecodingStream(final RemoteProcess process, final String path, final String hostname) {
+		private ContentStream(final RemoteProcess process, final String path, final String hostname) {
 			this.process = process;
-			this.stdout = process.stdout();
 			this.path = path;
 			this.hostname = hostname;
 			fill();
@@ -690,35 +686,16 @@ public int available() {
 
 		/** Make sure unread bytes are buffered: {@code false} at the end of a successful output. */
 		private boolean fill() {
-			while (position == block.length) {
-				if (ended) {
-					return false;
-				}
-				final String line = readLine(stdout);
-				if (line == null) {
+			if (position == block.length && !ended) {
+				final byte[] chunk = process.readStdoutChunk();
+				if (chunk == null) {
 					end();
-					return false;
-				}
-				final String data = line.strip();
-				if (data.isEmpty()) {
-					continue;
-				}
-				try {
-					block = Base64.getDecoder().decode(data);
-				} catch (final IllegalArgumentException e) {
-					throw new WinRMClientException(
-						String.format(
-							"Unexpected output while reading remote file %s on %s: %s",
-							path,
-							hostname,
-							data.length() > 80 ? data.substring(0, 80) + "..." : data
-						),
-						e
-					);
+				} else {
+					block = chunk;
+					position = 0;
 				}
-				position = 0;
 			}
-			return true;
+			return position < block.length;
 		}
 
 		/** The output ended: collect the exit code, release the connection, report a failure. */

src/main/java/org/metricshub/winrm/RemoteProcess.java

@@ -29,6 +29,7 @@
 import java.io.Writer;
 import java.nio.charset.Charset;
 import java.time.Duration;
+import java.util.ArrayDeque;
 import java.util.concurrent.TimeoutException;
 import org.metricshub.winrm.exceptions.WinRMTimeoutException;
 import org.metricshub.winrm.exceptions.WindowsRemoteException;
@@ -98,6 +99,10 @@ public final class RemoteProcess implements AutoCloseable {
 	private final StringBuilder stdoutPending = new StringBuilder();
 	private final StringBuilder stderrPending = new StringBuilder();
 
+	// Undecoded stdout chunks not read yet, once readStdoutChunk() took stdout as raw bytes: null
+	// while stdout is decoded text.
+	private ArrayDeque<byte[]> stdoutChunks;
+
 	// Written input that has not been flushed to the host yet.
 	private final StringBuilder stdinPending = new StringBuilder();
 
@@ -166,6 +171,26 @@ public BufferedReader stderr() {
 		return stderr;
 	}
 
+	/**
+	 * Read the next chunk of the standard output as raw bytes, bypassing the charset decoder: for
+	 * output that is not text, like the content of a remote file (see {@link RemoteFiles}). The
+	 * first call switches stdout to raw bytes for good, so make it before anything reads or waits
+	 * for output; the {@link #stdout()} reader then stays empty.
+	 *
+	 * @return the bytes as they arrived, never empty, or {@code null} at the end of the output
+	 * @throws WinRMTimeoutException when the command stays silent for a whole inactivity timeout
+	 * @throws org.metricshub.winrm.exceptions.WinRMClientException for any other failure
+	 */
+	synchronized byte[] readStdoutChunk() {
+		if (stdoutChunks == null) {
+			stdoutChunks = new ArrayDeque<>();
+		}
+		while (stdoutChunks.isEmpty() && !finished) {
+			fetchOnce();
+		}
+		return stdoutChunks.poll();
+	}
+
 	/**
 	 * Get the standard input of the remote command. Written text is buffered locally until
 	 * {@code flush()}, which carries it to the host as one WSMan Send (encoded with the request's
@@ -410,7 +435,11 @@ private void absorb(final CommandCursor.Chunk chunk) {
 			stdoutPending.append(stdoutDecoder.finish());
 			stderrPending.append(stderrDecoder.finish());
 		} else {
-			stdoutPending.append(stdoutDecoder.decode(chunk.stdout()));
+			if (stdoutChunks == null) {
+				stdoutPending.append(stdoutDecoder.decode(chunk.stdout()));
+			} else if (chunk.stdout().length > 0) {
+				stdoutChunks.add(chunk.stdout());
+			}
 			stderrPending.append(stderrDecoder.decode(chunk.stderr()));
 		}
 	}

src/site/markdown/cli.md

@@ -310,7 +310,7 @@ there, or run the redirection in cmd.exe. PowerShell 7.4 and later keep the byte
 
 When the output is closed early — `... cat 'D:\logs\huge.log' | head` — `cat` stops the remote
 read instead of transferring the rest of the file for nobody, and exits with `74`. The transfer
-runs at about 1.5 MB/s (see [Read performance](files.html#read-performance)): logs and
+runs at about 2 MB/s (see [Read performance](files.html#read-performance)): logs and
 configuration files, not bulk data.
 
 ### `get`
@@ -319,8 +319,8 @@ configuration files, not bulk data.
 [digest-verified, atomic, and skipped when the local copy is already identical](file-transfers.html#downloading-a-file).
 Without a local path, the file is written in the current directory under its remote name; an
 existing directory receives it under its remote name too. Nothing is printed on success.
-`--timeout` is the deadline of the whole download: at about 1.5 MB/s, the default 60 seconds
-covers files up to about 80 MB — raise it for larger ones.
+`--timeout` is the deadline of the whole download: at about 1.8 MB/s, the default 60 seconds
+covers files up to about 100 MB — raise it for larger ones.
 
 ### Quoting remote paths
 

src/site/markdown/file-transfers.md

@@ -243,7 +243,7 @@ reads need — PowerShell 2.0 or later in `FullLanguage` mode, and read access t
 
 The timeout of a download is a **wall-clock deadline for the whole transfer**: the client's
 timeout for `downloadFile(...)` (30 seconds by default), or `timeout(Duration)` on the request.
-A large file needs a raised timeout — at the speed below, 30 seconds is about 40 MB. When the
+A large file needs a raised timeout — at the speed below, 30 seconds is about 50 MB. When the
 deadline fires, the exception says how far the transfer got:
 
 ```text
@@ -257,9 +257,9 @@ place lets the move complete, and the download succeeds.
 
 ### Download performance
 
-Measured over HTTP with NTLM encryption, a download runs at about **1.35–1.45 MB/s**: 20 MiB in
-14.7 seconds on Windows Server 2022, 15.6 seconds on Windows Server 2008 R2 (PowerShell 2.0), and
-64 MiB in 46 seconds on 2022 — probe included. A small file costs under a second (two PowerShell
+Measured over HTTP with NTLM encryption, a download runs at about **1.7–1.9 MB/s**: 20 MiB in
+12 seconds on Windows Server 2022 and 2008 R2 (PowerShell 2.0), 13 seconds on 2019, and 64 MiB in
+35 seconds on 2022 — probe included. A small file costs under a second (two PowerShell
 invocations: the probe and the read), and skipping an identical copy about 0.4 seconds. The limit
 is on the host, in the way the WinRM service forwards a command's output — see
 [Read performance](files.html#read-performance).