Issue #196: Retire the shell when a command's terminate Signal does not go through

CHANGELOG.md

@@ -158,6 +158,20 @@ Consequences:
 
 ### Fixed
 
+- **A long-lived client no longer exhausts the host's operation quota** (#196). Every command
+  run in a shell holds one of the user's WSMan operations until the shell is deleted, even once
+  terminated, and the client reused its shell forever: a client polling indefinitely ended up
+  with every command refused with WSManFault 2150859174 (*the maximum number of concurrent
+  operations for this user has been exceeded*), after 15 commands on Windows Server 2008 R2 and
+  1500 on later versions, until it was recreated. The client now replaces its shell every 10
+  commands (the new `WinRMClient.Builder.maxCommandsPerShell(int)` changes that number) and never
+  reuses a shell holding a command it could not terminate (its terminate `Signal` failed or was
+  skipped). On Windows Server 2008 R2, whose quota fault carries its WSManFault code, a command
+  the quota refuses in a shell that already ran commands is also retried once in a new shell;
+  later versions send that fault with no code, so nothing reliable identifies it. The new shell
+  gets the same working directory, environment and profile; deleting the old one ends any process
+  a previous command left running in it, as closing the client always did.
+
 - **A streaming read resumed after a long pause no longer times out spuriously** (#198). When a
   streaming consumer (e.g. a `RemoteProcess` read slowly) paused longer than the inactivity
   timeout and the host dropped the idle connection meanwhile, the reconnection inherited the

src/main/java/org/metricshub/winrm/CommandRequest.java

@@ -192,8 +192,9 @@ private String powerShellFromFile(final String script, final long timeoutMillis,
 
 	/**
 	 * Set the working directory of the remote process. The remote command shell is created on
-	 * the first command a client executes and is reused afterward, so this setting takes effect
-	 * only when this is the client's first command.
+	 * the first command a client executes and is reused afterward (every shell that replaces it
+	 * gets the same settings), so this setting takes effect only when this is the client's first
+	 * command.
 	 *
 	 * @param workingDirectory the working directory path on the remote host
 	 * @return this request
@@ -209,7 +210,8 @@ public CommandRequest workingDirectory(final String workingDirectory) {
 	 * be called several times; insertion order is preserved, and setting the same name again
 	 * replaces its value. Like {@link #workingDirectory(String)}, the environment is shell-scoped:
 	 * the remote command shell is created on the first command a client executes and is reused
-	 * afterward, so this setting takes effect only when this is the client's first command.
+	 * afterward (every shell that replaces it gets the same settings), so this setting takes
+	 * effect only when this is the client's first command.
 	 *
 	 * <pre>{@code
 	 * CommandResult result = client.command("build.cmd")

src/main/java/org/metricshub/winrm/ShellFileCopy.java

@@ -118,8 +118,10 @@ private ShellFileCopy() {}
 
 	/**
 	 * Base delay before retrying after an operation-quota rejection; each retry waits one step
-	 * longer. Measured on Windows 2008 R2 (quota 15 per user): the budget fully recovers within
-	 * 30 seconds, so the escalating delays (5+10+15+20 s) comfortably bridge it.
+	 * longer (5+10+15+20 s). Time does not release the operations a shell holds, only
+	 * deleting the shell does, and the client already replaces its own shell when the quota
+	 * refuses a command (see {@code WsmanClient}): these delays give the user's other
+	 * connections time to release theirs.
 	 */
 	static final long QUOTA_RETRY_DELAY_MILLIS = 5_000L;
 
@@ -796,8 +798,9 @@ private static WindowsRemoteCommandResult run(
 
 				// The quota rejection happened while the operation was being CREATED — before the
 				// command could run — so retrying cannot duplicate a side effect. Old Windows
-				// versions cap concurrent operations very low (15 per user on 2008 R2) and reap
-				// completed ones lazily: give the server increasingly more time to recover.
+				// versions cap concurrent operations very low (15 per user on 2008 R2), and the
+				// client already replaced its own shell: give the user's other connections
+				// increasingly more time to release theirs.
 				try {
 					Utils.sleep(
 						Math.min(

src/main/java/org/metricshub/winrm/WinRMClient.java

@@ -331,6 +331,7 @@ public static final class Builder {
 		private int consoleCodePage;
 		private boolean loadUserProfile;
 		private String arraySeparator = LightWinRMService.DEFAULT_ARRAY_SEPARATOR;
+		private int maxCommandsPerShell = LightWinRMService.DEFAULT_MAX_COMMANDS_PER_SHELL;
 		private SSLContext sslContext;
 		private Duration timeout = DEFAULT_TIMEOUT;
 		private int retries;
@@ -624,6 +625,38 @@ public Builder arraySeparator(final String arraySeparator) {
 			return this;
 		}
 
+		/**
+		 * Set how many commands the remote command shell runs before the client replaces it.
+		 * Default: 10.
+		 * <p>
+		 * The client reuses its command shell for commands, file transfers and remote file
+		 * operations, but every command run in a shell holds one of the user's WSMan operations
+		 * until the shell is deleted, even after it completed. The host caps them per user,
+		 * across all of that user's connections ({@code MaxConcurrentOperationsPerUser}: 15 on
+		 * Windows Server 2008 R2, 1500 later), so a shell reused forever ends up having every
+		 * command refused. Replacing the shell (one Delete and one Create, about 100 ms) releases
+		 * them. The new shell gets the same working directory, environment and profile, but
+		 * deleting the old one ends any process a previous command left running in it, as
+		 * {@link WinRMClient#close()} does. Besides, on Windows Server 2008 R2, whose quota fault
+		 * carries its WSManFault code, a command the quota refuses in a shell that already ran
+		 * commands is retried once in a new shell; in a fresh shell, which holds nothing to
+		 * release, the fault is reported. Later versions send that fault with no code, so this
+		 * setting is what keeps a client under the quota there.
+		 * <p>
+		 * A lower value leaves more of the quota to the user's other connections at the cost of
+		 * more frequent replacements; 1 runs every command in a shell of its own, like {@code winrs}.
+		 *
+		 * @param maxCommandsPerShell how many commands a shell runs before it is replaced (at least 1)
+		 * @return this builder
+		 */
+		public Builder maxCommandsPerShell(final int maxCommandsPerShell) {
+			if (maxCommandsPerShell < 1) {
+				throw new IllegalArgumentException("maxCommandsPerShell must be at least 1.");
+			}
+			this.maxCommandsPerShell = maxCommandsPerShell;
+			return this;
+		}
+
 		/**
 		 * Build the client. This does not connect yet: the connection is established and
 		 * authenticated by the first operation.
@@ -672,6 +705,7 @@ public WinRMClient build() {
 					consoleCodePage,
 					loadUserProfile,
 					arraySeparator,
+					maxCommandsPerShell,
 					retries,
 					toMillis(retryDelay)
 				);

src/main/java/org/metricshub/winrm/light/LightWinRMService.java

@@ -62,6 +62,13 @@ public final class LightWinRMService implements WindowsRemoteExecutor {
 	/** The default string joining the elements of a WMI array property in a WQL row. */
 	public static final String DEFAULT_ARRAY_SEPARATOR = "|";
 
+	/**
+	 * The default number of commands a remote command shell runs before the client replaces it.
+	 * Each command holds one of the user's WSMan operations until its shell is deleted, and
+	 * Windows Server 2008 R2 allows only 15 per user ({@code MaxConcurrentOperationsPerUser}).
+	 */
+	public static final int DEFAULT_MAX_COMMANDS_PER_SHELL = 10;
+
 	private final WinRMEndpoint winRMEndpoint;
 	private final WsmanClient client;
 	private final AtomicBoolean closed = new AtomicBoolean(false);
@@ -275,6 +282,71 @@ public static LightWinRMService createInstance(
 	 * Create a light WinRM executor that may delegate the caller's Kerberos credentials to the host,
 	 * so remote commands can authenticate onward as the caller (the second hop), may load the
 	 * user profile in the command shell, and joins WMI array properties with a custom separator.
+	 * The command shell is replaced every {@link #DEFAULT_MAX_COMMANDS_PER_SHELL} commands.
+	 *
+	 * @param winRMEndpoint endpoint with credentials (mandatory)
+	 * @param timeout timeout in milliseconds (must be > 0)
+	 * @param ticketCache Kerberos ticket cache path (used by the Kerberos scheme; {@code null} logs
+	 *        in with the password)
+	 * @param authentications requested authentication schemes, tried in order (NTLM, Kerberos, and/or Basic);
+	 *        {@code null}/empty means NTLM only
+	 * @param allowDelegation whether Kerberos forwards the caller's ticket-granting ticket to the
+	 *        host (which must then be forwardable); requires Kerberos among {@code authentications}
+	 * @param sslContext the {@link SSLContext} providing the HTTPS socket factory (hostname
+	 *        verification stays on); {@code null} uses the default configuration
+	 * @param trustAllCertificates when {@code true} (and no {@code sslContext} is given), trust every
+	 *        server certificate and skip hostname verification — insecure, testing only
+	 * @param consoleCodePage the console code page of the command shell; 0 keeps the default 65001,
+	 *        which makes command output UTF-8 whatever the remote locale
+	 * @param loadUserProfile whether the command shell loads the user profile (registry hive,
+	 *        per-user environment variables); {@code false} is the historical behavior
+	 * @param arraySeparator the string joining the elements of a WMI array property in a WQL row;
+	 *        see {@link #DEFAULT_ARRAY_SEPARATOR}
+	 * @param connectRetries how many times one round trip may re-attempt to connect and authenticate
+	 *        (must be >= 0); 0 keeps the historical fail-fast behavior
+	 * @param retryDelay the pause in milliseconds before each retry (must be >= 0)
+	 * @return a new {@code LightWinRMService}
+	 * @throws WinRMException on invalid arguments or an unsupported authentication request
+	 */
+	// CPD-OFF — a compatibility overload: its parameter list is the next overload's minus the
+	// shell bound, and reordering the parameters to fool the detector would break callers.
+	public static LightWinRMService createInstance(
+		final WinRMEndpoint winRMEndpoint,
+		final long timeout,
+		final java.nio.file.Path ticketCache,
+		final List<AuthenticationEnum> authentications,
+		final boolean allowDelegation,
+		final SSLContext sslContext,
+		final boolean trustAllCertificates,
+		final int consoleCodePage,
+		final boolean loadUserProfile,
+		final String arraySeparator,
+		final int connectRetries,
+		final long retryDelay
+	) throws WinRMException {
+		return createInstance(
+			winRMEndpoint,
+			timeout,
+			ticketCache,
+			authentications,
+			allowDelegation,
+			sslContext,
+			trustAllCertificates,
+			consoleCodePage,
+			loadUserProfile,
+			arraySeparator,
+			DEFAULT_MAX_COMMANDS_PER_SHELL,
+			connectRetries,
+			retryDelay
+		);
+		// CPD-ON
+	}
+
+	/**
+	 * Create a light WinRM executor that may delegate the caller's Kerberos credentials to the host,
+	 * so remote commands can authenticate onward as the caller (the second hop), may load the
+	 * user profile in the command shell, joins WMI array properties with a custom separator, and
+	 * replaces the command shell after a given number of commands.
 	 *
 	 * @param winRMEndpoint endpoint with credentials (mandatory)
 	 * @param timeout timeout in milliseconds (must be &gt; 0)
@@ -294,6 +366,8 @@ public static LightWinRMService createInstance(
 	 *        per-user environment variables); {@code false} is the historical behavior
 	 * @param arraySeparator the string joining the elements of a WMI array property in a WQL row;
 	 *        see {@link #DEFAULT_ARRAY_SEPARATOR}
+	 * @param maxCommandsPerShell how many commands a command shell runs before it is replaced (must
+	 *        be &gt; 0); see {@link #DEFAULT_MAX_COMMANDS_PER_SHELL}
 	 * @param connectRetries how many times one round trip may re-attempt to connect and authenticate
 	 *        (must be &gt;= 0); 0 keeps the historical fail-fast behavior
 	 * @param retryDelay the pause in milliseconds before each retry (must be &gt;= 0)
@@ -311,12 +385,16 @@ public static LightWinRMService createInstance(
 		final int consoleCodePage,
 		final boolean loadUserProfile,
 		final String arraySeparator,
+		final int maxCommandsPerShell,
 		final int connectRetries,
 		final long retryDelay
 	) throws WinRMException {
 		Utils.checkNonNull(winRMEndpoint, "winRMEndpoint");
 		Utils.checkNonNull(arraySeparator, "arraySeparator");
 		Utils.checkArgumentNotZeroOrNegative(timeout, "timeout");
+		if (maxCommandsPerShell < 1) {
+			throw new IllegalArgumentException("maxCommandsPerShell must be at least 1.");
+		}
 		if (connectRetries < 0) {
 			throw new IllegalArgumentException("connectRetries must not be negative.");
 		}
@@ -366,6 +444,7 @@ public static LightWinRMService createInstance(
 			consoleCodePage,
 			loadUserProfile,
 			arraySeparator,
+			maxCommandsPerShell,
 			connectRetries,
 			retryDelay
 		);