From ad4c84322816ba6cf5f9ec31db96b99392e33018 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 16 Oct 2026 20:34:40 +0200
Subject: [PATCH 1/5] docs: the reverse tunnel of the ssh shadow

A section in the SSH tunnel guide (how it works, what the site has to
allow, the annotations, SOCKS for undeclared endpoints, who else on the
node can use the forwards, limits, troubleshooting), the Reverse options
in the options table, and the interlink.eu/reverse-* annotations in the
pod annotations reference. The note saying the ssh shadow gives the pod
no way back into the cluster is updated.

Part of #548

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 docs/docs/guides/09-pod-annotations.mdx       |  96 +++++++++++++
 .../guides/14-ssh-tunnel-configuration.mdx    | 128 +++++++++++++++++-
 2 files changed, 220 insertions(+), 4 deletions(-)

diff --git a/docs/docs/guides/09-pod-annotations.mdx b/docs/docs/guides/09-pod-annotations.mdx
index 33fa91cd..9bbcb776 100644
--- a/docs/docs/guides/09-pod-annotations.mdx
+++ b/docs/docs/guides/09-pod-annotations.mdx
@@ -16,6 +16,7 @@ InterLink supports several categories of annotations:
 - **[Mesh Networking](#mesh-networking)**: Control bidirectional mesh networking for offloaded pods
 - **[Wstunnel Integration](#wstunnel-integration)**: Control websocket tunneling for exposed ports
 - **[SSH Shadow](#ssh-shadow)**: Unix socket ports and namespace placement for shadow pods
+- **[SSH Reverse Tunnel](#ssh-reverse-tunnel)**: Let an offloaded pod reach endpoints inside the cluster over the SSH shadow
 - **[Job Management](#job-management)**: Handle remote job execution and tracking
 - **[System Annotations](#system-annotations)**: Internal annotations used by interLink components
 
@@ -308,6 +309,101 @@ metadata:
 
 ---
 
+## SSH Reverse Tunnel
+
+With the [SSH shadow](./14-ssh-tunnel-configuration.mdx) and
+`Network.SSH.Reverse.Enabled: true`, these annotations let an offloaded pod open
+connections to endpoints inside the cluster. How it works, what the site has to allow
+and the limits are in
+[Pod to cluster: the reverse tunnel](./14-ssh-tunnel-configuration.mdx#pod-to-cluster-the-reverse-tunnel).
+
+### `interlink.eu/reverse-forwards`
+
+Lists the cluster endpoints the pod may connect to.
+
+**Usage:**
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: trainer
+  annotations:
+    interlink.eu/reverse-forwards: "minio.user.svc.cluster.local:9000 6443:kubernetes.default.svc.cluster.local:443"
+spec:
+  containers:
+  - name: trainer
+    image: docker://python:3.12
+```
+
+**Format:**
+- **Endpoint**: `host:port` - the job connects to the same port
+- **Remapped**: `listen-port:host:port` - the job connects to `listen-port`; needed when `port` is below 1024
+- **Several**: space-separated
+
+**Behavior:**
+- The virtual kubelet adds the names to the container's `/etc/hosts` and waits for the tunnel before the workload starts
+- A Service FQDN also resolves as `name.namespace.svc` and `name.namespace`, and as `name` in the pod's own namespace
+- An entry that does not parse, or a listen port below 1024, is refused when the pod is created
+- Needs `Network.SSH.Reverse.Enabled: true`; otherwise the pod is refused rather than run without its tunnel
+
+**Default:** Not set (no connections back into the cluster)
+
+---
+
+### `interlink.eu/reverse-socks-port`
+
+Adds a SOCKS5 proxy for endpoints not listed in `interlink.eu/reverse-forwards`.
+
+**Usage:**
+```yaml
+metadata:
+  annotations:
+    interlink.eu/reverse-socks-port: "1080"
+```
+
+**Behavior:**
+- Any cluster address, resolved inside the cluster at connect time, for programs told to use the proxy (`ALL_PROXY=socks5h://<bind>:1080`)
+- A plain loopback listener: every user of the compute node can use it
+- Must be above 1023 and not clash with a forward's listen port
+
+**Default:** Not set (no proxy)
+
+---
+
+### `interlink.eu/reverse-bind`
+
+The `127.x.y.z` address the forwards listen on inside the job.
+
+**Behavior:**
+- Derived from the pod UID when unset, so two pods on one node never share it
+- Must be in `127.0.0.0/8` and not `127.0.0.1`
+
+**Default:** Derived from the pod UID
+
+---
+
+### `interlink.eu/reverse-node-exec`
+
+How the shadow runs a command on the compute node from the login node, for this pod.
+
+**Format:** `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`)
+
+**Default:** `Network.SSH.Reverse.NodeExec`
+
+---
+
+### `interlink.eu/reverse-owner-only`
+
+Whether the forwards can be used by the job's user only.
+
+**Behavior:**
+- `true`: each forward ends in a unix socket only that user can open, with a relay in front that checks the connecting user; needs `python3` on the compute node
+- `false`: the ports are bound directly, open to every user of the compute node
+
+**Default:** `true`
+
+---
+
 ## Job Management
 
 ### `JobID`
diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index ef19c0bd..0dbe3339 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -25,10 +25,11 @@ configuration, no pre-exec injection.
 
 :::info
 The SSH shadow replaces **wstunnel**, not [full mesh](./13-mesh-network-configuration.mdx).
-It exposes the offloaded pod's ports to the cluster; it does not give the pod access
-back into the cluster, so an offloaded workload still cannot reach in-cluster object
-storage or message buses. Combining the two is rejected at startup and tracked in
-[#548](https://github.com/interlink-hq/interLink/issues/548).
+It exposes the offloaded pod's ports to the cluster. A workload that needs the other
+direction, in-cluster object storage or a message bus for example, gets it over the same
+login node with the [reverse tunnel](#pod-to-cluster-the-reverse-tunnel), without
+WireGuard and without a network namespace. Combining the SSH shadow with full mesh is
+rejected at startup.
 :::
 
 ---
@@ -128,6 +129,7 @@ warning in the virtual kubelet log.
 | --- | --- | --- | --- |
 | `wstunnel` (default) | cluster → pod | `EnableTunnel` + exposed ports | wstunnel client, dials out |
 | `ssh` | cluster → pod | `EnableTunnel` + exposed ports | nothing |
+| `ssh` reverse | pod → cluster | `SSH.Reverse.Enabled` + `interlink.eu/reverse-forwards` | nothing: the batch script waits for the tunnel |
 | full mesh | bidirectional | `FullMesh: true`, every pod | `mesh.sh` (slirp4netns + WireGuard) |
 
 ---
@@ -220,6 +222,10 @@ in the chart repository for a complete deployment.
 | `ReplicateCredentials` | `true` | Copy the credential into each shadow's namespace |
 | `NodeWaitTimeout` | `2h` | How long the shadow waits for the compute node |
 | `ExtraOptions` | `[]` | Extra ssh options, each passed as `-o <option>` on every hop, after the shadow's own. ssh keeps the first value it sees, so they add settings but cannot override the shadow's. One exception: `GSSAPIDelegateCredentials=yes` turns Kerberos delegation back on for the login hop |
+| `Reverse.Enabled` | `false` | Let pods ask for a [tunnel back into the cluster](#pod-to-cluster-the-reverse-tunnel) |
+| `Reverse.NodeExec` | `ssh` | How the shadow runs a command on the compute node from the login node: `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`, several times slower, see [Performance](#performance)) |
+| `Reverse.SshdPath` | `/usr/sbin/sshd` | The sshd binary on the compute node |
+| `Reverse.PythonPath` | `python3` | The python3 on the compute node, for owner-only forwards |
 
 Misconfiguration fails at **startup**, not on the first offloaded pod, so a missing
 `LoginHost` or a `kerberos` block without a `Principal` is caught immediately.
@@ -710,6 +716,110 @@ kubelet log.
 
 ---
 
+## Pod to cluster: the reverse tunnel
+
+Everything above carries connections from the cluster to the pod. A workload that
+needs the other direction, an object store, a message bus or the Kubernetes API that
+only exist inside the cluster, gets it over the same login node. The site switches it
+on with `Network.SSH.Reverse.Enabled: true`; the pod lists what it needs:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: trainer
+  annotations:
+    interlink.eu/reverse-forwards: "minio.user.svc.cluster.local:9000 6443:kubernetes.default.svc.cluster.local:443"
+```
+
+The workload then uses the Service name and port as it would in the cluster,
+`http://minio.user.svc.cluster.local:9000` here. The compute node needs no route to the
+cluster, no WireGuard and no network namespace, so its own interfaces, InfiniBand
+included, stay visible to the job.
+
+### How it works
+
+```
+job container ──connect name:port──> 127.x.y.z:port on the compute node (sshd, -R)
+                                           │ carried back over the shadow's ssh session
+                                           ▼
+                     shadow pod (ssh-reverse) ──dials──> Service inside the cluster
+```
+
+1. The shadow runs a second container, `ssh-reverse`. Once the compute node is known
+   it starts a user-space `sshd -i` there, as the job's user, through a command run
+   on the login node (`ssh <node>` or `srun --overlap`, see `NodeExec`). The
+   command's stdin and stdout are that sshd's transport: it listens on no port. Its
+   host key and the one client key it accepts are generated for each start and
+   never leave the shadow.
+2. The shadow connects to it with `ssh -R`. Each forward is bound on the compute
+   node, on a loopback address derived from the pod UID (`127.x.y.z`), and every
+   connection made there is dialled from the shadow pod, inside the cluster.
+3. The virtual kubelet prepends a snippet to the pod's `slurm-job.vk.io/pre-exec`.
+   It maps the forwarded names to that address in the container's `/etc/hosts`
+   (through `SINGULARITY_BIND` and `APPTAINER_BIND`) and holds the workload until the
+   tunnel answers, for at most ten minutes. The workload itself is unchanged.
+
+The tunnel is rebuilt if the job moves to another node, like the forwards.
+
+### What the site has to allow
+
+- A command on the compute node, run from the login node as the job's user. Either
+  `ssh <node>` into a node where that user has a running job, which sites with
+  `pam_slurm_adopt` allow, or `srun --jobid=<job> --overlap`, which needs nothing
+  beyond Slurm. `NodeExec` picks one; a pod can override it.
+- `sshd` on the compute node (`Reverse.SshdPath`) and, for owner-only forwards,
+  `python3`. Both are stock on most distributions and neither needs root.
+- Nothing in the login node's `sshd_config`: the forwards are bound by the sshd the
+  shadow starts, not by the site's.
+
+### Annotations
+
+| Annotation | Value |
+| --- | --- |
+| `interlink.eu/reverse-forwards` | Space-separated endpoints, each `host:port` or `listen-port:host:port` |
+| `interlink.eu/reverse-socks-port` | Port of a SOCKS5 proxy for endpoints not listed; off when unset |
+| `interlink.eu/reverse-bind` | The `127.x.y.z` address to use instead of the derived one |
+| `interlink.eu/reverse-node-exec` | `ssh` or `srun`, overriding `Reverse.NodeExec` |
+| `interlink.eu/reverse-owner-only` | `true` (default) or `false`, see below |
+
+A Service port below 1024 cannot be bound by an unprivileged sshd on the compute
+node: give it a listen port, `6443:kubernetes.default.svc.cluster.local:443`, and the
+job connects to port 6443. The name resolves to the right address either way. An
+annotation that does not parse is refused when the pod is created, with the reason in
+the virtual kubelet log and the pod's events.
+
+### Endpoints not known in advance
+
+`interlink.eu/reverse-socks-port: "1080"` adds a SOCKS5 proxy on the same address. A
+program told to use it, `ALL_PROXY=socks5h://127.x.y.z:1080` for curl and Python
+requests, reaches any cluster address by name, resolved inside the cluster at connect
+time. It is off unless asked for, and it is a plain loopback listener that every user
+of that compute node can also use, so declare what you can and keep SOCKS for the rest.
+
+### Who else can use the forwards
+
+A port on a loopback address is reachable by every user of the compute node. By
+default each forward therefore ends in a unix socket that only the job's user can
+open, with a small relay in front, run by the sshd session on the node, that accepts
+a connection only when the connecting socket belongs to that user (read from
+`/proc/net/tcp`). `interlink.eu/reverse-owner-only: "false"` binds the ports directly
+instead: no relay, no `python3` on the node, and no protection from other users.
+
+### Limits
+
+- TCP only.
+- The listeners exist on the node that runs the batch script. Ranks of a multi-node
+  job on other nodes have no tunnel.
+- The tunnel comes up once the shadow has learned the compute node, a status poll or
+  two after the job starts; the pre-exec waits for it.
+- The pre-exec assumes the Slurm plugin with Singularity or Apptainer, as the mesh
+  pre-exec does.
+- `/tmp/il-rev-<shadow>` on the compute node holds the keys and sockets while the
+  tunnel is up; the session removes it when the tunnel ends.
+
+---
+
 ## Troubleshooting
 
 **Shadow stuck in `Init:0/1`.** Normal while the job is queued. If it persists,
@@ -784,6 +894,15 @@ the value it produced is quoted in the message.
 already contains a different object under the credential's name. Rename the
 credential, or set `ReplicateCredentials: false` and provision it yourself.
 
+**`interLink: the reverse tunnel did not come up within 600s` in the job log.** The
+shadow's `ssh-reverse` container never got through; read its log. `could not prepare
+... on <node>` means the command to the compute node was refused, so try
+`NodeExec: srun`; `no running job named ... on <node>` in `srun` mode means `squeue`
+on the login node did not list a running job under the pod's UID.
+
+**Pod creation fails with `interlink.eu/reverse-forwards: ...`.** The annotation did
+not parse; the message names the entry and what is wrong with it.
+
 **An nginx wstunnel Deployment appears instead of the SSH shadow.** A custom template
 file is overriding the built-in one. The virtual kubelet prefers
 `Network.WstunnelTemplatePath` over its own templates; the Helm chart stops mounting
@@ -795,5 +914,6 @@ it in `ssh` mode, but a hand-written config may still point at one.
 
 - [Discussion #535](https://github.com/interlink-hq/interLink/discussions/535) — the original proposal
 - [Issue #545](https://github.com/interlink-hq/interLink/issues/545) — in-cluster SSH deployment
+- [Issue #548](https://github.com/interlink-hq/interLink/issues/548) — pod to cluster connectivity without WireGuard
 - [Wstunnel Configuration](./10-wstunnel-configuration.mdx)
 - [Mesh Network Configuration](./13-mesh-network-configuration.mdx)

From d92864a3041cb6060ad6ee833ca04e714af58ad7 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 16 Oct 2026 20:34:40 +0200
Subject: [PATCH 2/5] docs: performance of the reverse tunnel carriers

The NodeExec row linked to a Performance section that did not exist. Add it:
srun --overlap carries the tunnel through Slurm's I/O forwarding, which has
lower throughput than ssh to the node, so srun is for nodes the login node
cannot reach over ssh. The owner-only relay adds no measurable cost.

Part of #548

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 docs/docs/guides/14-ssh-tunnel-configuration.mdx | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 0dbe3339..729df3fc 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -223,7 +223,7 @@ in the chart repository for a complete deployment.
 | `NodeWaitTimeout` | `2h` | How long the shadow waits for the compute node |
 | `ExtraOptions` | `[]` | Extra ssh options, each passed as `-o <option>` on every hop, after the shadow's own. ssh keeps the first value it sees, so they add settings but cannot override the shadow's. One exception: `GSSAPIDelegateCredentials=yes` turns Kerberos delegation back on for the login hop |
 | `Reverse.Enabled` | `false` | Let pods ask for a [tunnel back into the cluster](#pod-to-cluster-the-reverse-tunnel) |
-| `Reverse.NodeExec` | `ssh` | How the shadow runs a command on the compute node from the login node: `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`, several times slower, see [Performance](#performance)) |
+| `Reverse.NodeExec` | `ssh` | How the shadow runs a command on the compute node from the login node: `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`, lower throughput, see [Performance](#performance)) |
 | `Reverse.SshdPath` | `/usr/sbin/sshd` | The sshd binary on the compute node |
 | `Reverse.PythonPath` | `python3` | The python3 on the compute node, for owner-only forwards |
 
@@ -818,6 +818,12 @@ instead: no relay, no `python3` on the node, and no protection from other users.
 - `/tmp/il-rev-<shadow>` on the compute node holds the keys and sockets while the
   tunnel is up; the session removes it when the tunnel ends.
 
+### Performance
+
+`NodeExec: srun` carries the tunnel through Slurm's I/O forwarding, which has lower
+throughput than `ssh <node>`. Prefer `ssh` and keep `srun` for nodes the login node
+cannot reach over ssh. The owner-only relay adds no measurable cost.
+
 ---
 
 ## Troubleshooting

From d466b508afe50d825a2749c7ddc65f707b6d5d64 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 16 Oct 2026 20:34:40 +0200
Subject: [PATCH 3/5] docs: reverse tunnel on the node's sshd, with a job-owned
 relay

The reverse tunnel now ends on the compute node's own sshd, reached
through the login node with the same host key check as ForwardTarget:
node, and the owner-only relay is started by the job and holds the
pod's loopback address until the job ends. The guide described a
user-space sshd started for every tunnel and a relay that lived only
as long as the session, so it is rewritten to match: how it works,
what the site has to allow (forwarding on the node's sshd, a plugin
socket directory, python3 on the nodes), and the user-space sshd as
the fallback for nodes whose sshd refuses forwarding.

Owner-only is always on, so its annotation and opt-out go, and so does
reverse-node-exec: the carrier is the site's decision and is set only
in the virtual kubelet configuration, with srun documented as the last
resort. The SOCKS proxy gets a warning box in both guides: it listens
on 127.0.0.1 of the node, is open to every user there, and is refused
unless the virtual kubelet sets Reverse.AllowSocks.

The tunnel-only key section gains the wrapper lines the reverse tunnel
needs, and says that the user-space sshd cannot work with such a key,
since it runs a script the shadow sends. Troubleshooting covers the new
job log messages and the refused SOCKS pod.

Part of #548

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 docs/docs/guides/09-pod-annotations.mdx       |  39 +--
 .../guides/14-ssh-tunnel-configuration.mdx    | 304 +++++++++++++-----
 2 files changed, 239 insertions(+), 104 deletions(-)

diff --git a/docs/docs/guides/09-pod-annotations.mdx b/docs/docs/guides/09-pod-annotations.mdx
index 9bbcb776..d842502e 100644
--- a/docs/docs/guides/09-pod-annotations.mdx
+++ b/docs/docs/guides/09-pod-annotations.mdx
@@ -345,6 +345,7 @@ spec:
 - A Service FQDN also resolves as `name.namespace.svc` and `name.namespace`, and as `name` in the pod's own namespace
 - An entry that does not parse, or a listen port below 1024, is refused when the pod is created
 - Needs `Network.SSH.Reverse.Enabled: true`; otherwise the pod is refused rather than run without its tunnel
+- Needs a plugin that gives the job a socket directory (`SocketDirRoot` in the Slurm plugin); otherwise the job fails at start
 
 **Default:** Not set (no connections back into the cluster)
 
@@ -354,6 +355,13 @@ spec:
 
 Adds a SOCKS5 proxy for endpoints not listed in `interlink.eu/reverse-forwards`.
 
+:::warning
+The proxy listens on `127.0.0.1` of the compute node, where every user of that node
+can use it to reach any endpoint the shadow reaches in the cluster. The virtual
+kubelet refuses it unless `Network.SSH.Reverse.AllowSocks` is true. A site that agreed
+to fixed endpoints and no proxy has not agreed to this: ask before turning it on.
+:::
+
 **Usage:**
 ```yaml
 metadata:
@@ -362,8 +370,10 @@ metadata:
 ```
 
 **Behavior:**
-- Any cluster address, resolved inside the cluster at connect time, for programs told to use the proxy (`ALL_PROXY=socks5h://<bind>:1080`)
-- A plain loopback listener: every user of the compute node can use it
+- Any cluster address, resolved inside the cluster at connect time, for programs told to use the proxy (`ALL_PROXY=socks5h://127.0.0.1:1080`)
+- Pod creation is refused unless the virtual kubelet sets `Network.SSH.Reverse.AllowSocks: true`; when it does, the pod gets a `ReverseSocksProxy` warning event
+- A plain listener on `127.0.0.1`, with no owner check: every user of the compute node can use it
+- Two pods on one compute node cannot use the same port; the job fails at start when another user already listens on it, and the shadow's tunnel fails when another pod of the same account does
 - Must be above 1023 and not clash with a forward's listen port
 
 **Default:** Not set (no proxy)
@@ -372,38 +382,17 @@ metadata:
 
 ### `interlink.eu/reverse-bind`
 
-The `127.x.y.z` address the forwards listen on inside the job.
+The `127.x.y.z` address of the owner-only relay: the forwards listen there, inside the job.
 
 **Behavior:**
 - Derived from the pod UID when unset, so two pods on one node never share it
 - Must be in `127.0.0.0/8` and not `127.0.0.1`
+- The job fails at start if something else already listens on that address and a forward's port
 
 **Default:** Derived from the pod UID
 
 ---
 
-### `interlink.eu/reverse-node-exec`
-
-How the shadow runs a command on the compute node from the login node, for this pod.
-
-**Format:** `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`)
-
-**Default:** `Network.SSH.Reverse.NodeExec`
-
----
-
-### `interlink.eu/reverse-owner-only`
-
-Whether the forwards can be used by the job's user only.
-
-**Behavior:**
-- `true`: each forward ends in a unix socket only that user can open, with a relay in front that checks the connecting user; needs `python3` on the compute node
-- `false`: the ports are bound directly, open to every user of the compute node
-
-**Default:** `true`
-
----
-
 ## Job Management
 
 ### `JobID`
diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 729df3fc..39ee8411 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -129,7 +129,7 @@ warning in the virtual kubelet log.
 | --- | --- | --- | --- |
 | `wstunnel` (default) | cluster → pod | `EnableTunnel` + exposed ports | wstunnel client, dials out |
 | `ssh` | cluster → pod | `EnableTunnel` + exposed ports | nothing |
-| `ssh` reverse | pod → cluster | `SSH.Reverse.Enabled` + `interlink.eu/reverse-forwards` | nothing: the batch script waits for the tunnel |
+| `ssh` reverse | pod → cluster | `SSH.Reverse.Enabled` + `interlink.eu/reverse-forwards` | nothing: the batch script starts a relay and waits for the tunnel |
 | full mesh | bidirectional | `FullMesh: true`, every pod | `mesh.sh` (slirp4netns + WireGuard) |
 
 ---
@@ -218,14 +218,16 @@ in the chart repository for a complete deployment.
 | `Principal` | none | Kerberos principal (`kerberos`) |
 | `Krb5ConfigMap` | none | ConfigMap with a `krb5.conf`, mounted at `/etc/krb5.conf` |
 | `KnownHostsConfigMap` | none | ConfigMap with a `known_hosts` file: the login node's key, plus compute node keys, host patterns or `@cert-authority` lines |
-| `NodeHostKeys` | `verify` | How compute node host keys are checked with `ForwardTarget: node`: `verify` or `accept-new`. See [Compute nodes](#compute-nodes) |
+| `NodeHostKeys` | `verify` | How compute node host keys are checked, with `ForwardTarget: node` and by the reverse tunnel, except a user-space sshd reached with `Reverse.NodeExec: srun`: `verify` or `accept-new`. See [Compute nodes](#compute-nodes) |
 | `ReplicateCredentials` | `true` | Copy the credential into each shadow's namespace |
 | `NodeWaitTimeout` | `2h` | How long the shadow waits for the compute node |
 | `ExtraOptions` | `[]` | Extra ssh options, each passed as `-o <option>` on every hop, after the shadow's own. ssh keeps the first value it sees, so they add settings but cannot override the shadow's. One exception: `GSSAPIDelegateCredentials=yes` turns Kerberos delegation back on for the login hop |
 | `Reverse.Enabled` | `false` | Let pods ask for a [tunnel back into the cluster](#pod-to-cluster-the-reverse-tunnel) |
-| `Reverse.NodeExec` | `ssh` | How the shadow runs a command on the compute node from the login node: `ssh` (`ssh <node>`) or `srun` (`srun --jobid=<job> --overlap`, lower throughput, see [Performance](#performance)) |
-| `Reverse.SshdPath` | `/usr/sbin/sshd` | The sshd binary on the compute node |
-| `Reverse.PythonPath` | `python3` | The python3 on the compute node, for owner-only forwards |
+| `Reverse.Sshd` | `node` | Which sshd makes the reverse forwards: `node` (the compute node's own) or `userspace` (one the shadow starts in the job, for nodes whose sshd refuses forwarding). See [A user-space sshd](#a-user-space-sshd) |
+| `Reverse.NodeExec` | `ssh` | How the shadow reaches port 22 of the compute node: `ssh` (through the login node, as `ForwardMode` says) or `srun` (`srun --jobid=<job> --overlap` on the login node, a last resort with lower throughput, see [Performance](#performance)). Set here only: pods cannot change it |
+| `Reverse.AllowSocks` | `false` | Let pods ask for a SOCKS proxy on the compute node, open to every user of that node. See [Endpoints not known in advance](#endpoints-not-known-in-advance) |
+| `Reverse.SshdPath` | `/usr/sbin/sshd` | The sshd binary on the compute node, for `Reverse.Sshd: userspace` only |
+| `Reverse.PythonPath` | `python3` | The python3 on the compute node, which runs the relay |
 
 Misconfiguration fails at **startup**, not on the first offloaded pod, so a missing
 `LoginHost` or a `kerberos` block without a `Principal` is caught immediately.
@@ -345,11 +347,12 @@ restrict,port-forwarding,from="<cluster egress>,<login nodes>",command="$HOME/.i
 
 - `restrict` turns off port, agent and X11 forwarding and terminal allocation.
 - `port-forwarding` turns port forwarding back on, which `ForwardTarget: node`
-  needs in both modes (the forwards are `-L` on the compute node) and
-  `ForwardMode: portforward` needs on the login node. Only a deployment with
-  `ForwardTarget: login` and `ForwardMode: exec` can drop it: sshd then forwards
-  nothing for this key, but the wrapper's `nc` lines still relay to any host and
-  port the login node reaches, so `from=` stays the main limit.
+  needs in both modes (the forwards are `-L` on the compute node), as do
+  `ForwardMode: portforward` on the login node and the
+  [reverse tunnel](#pod-to-cluster-the-reverse-tunnel). Only a deployment with
+  `ForwardTarget: login`, `ForwardMode: exec` and no reverse tunnel can drop it:
+  sshd then forwards nothing for this key, but the wrapper's `nc` lines still relay
+  to any host and port the login node reaches, so `from=` stays the main limit.
 - `from=` lists where the key may be used from: the cluster's egress addresses for
   the login hop, and the login nodes, as the compute nodes see them, for the hop to
   a compute node. This is the main limit. A key that may forward, or run the
@@ -382,14 +385,16 @@ It allows the netcat relays of `exec` mode and the read of the login node's host
 list that [`NodeHostKeys: verify`](#compute-nodes) does, and nothing else: no shell,
 no `scp`, no job submission. Set `NC` to the virtual kubelet's `ExecConnectCommand`.
 The newline check matters: `grep -x` matches line by line, so a second line could
-otherwise ride along.
+otherwise ride along. The reverse tunnel needs
+[a few more lines](#with-a-key-for-the-tunnel-only).
 
 As written, `h` accepts any host, so the relays reach whatever the login node
 reaches. A site can narrow them by setting `h` to its node-name pattern, for example
 `h="'?node[0-9]+\.nodes\.example\.org'?"`.
 
 This works for `Auth: publickey` only. A keytab is the account's own credential and
-cannot be restricted this way.
+cannot be restricted this way. It does not work with
+[`Reverse.Sshd: userspace`](#a-user-space-sshd) either.
 
 ### Kerberos
 
@@ -415,7 +420,9 @@ expiry.
 The ticket is not forwarded to the login node or to compute nodes
 (`GSSAPIDelegateCredentials=no`): nothing the shadow runs there needs it. If a site
 requires it on the login node, add `GSSAPIDelegateCredentials=yes` to
-`ExtraOptions`.
+`ExtraOptions`. The one exception is the reverse tunnel's last-resort carrier,
+`Reverse.NodeExec: srun`: `srun` runs on the login node and needs the ticket there,
+so the login hop then delegates it. The hop to a compute node never does.
 
 ### Host key verification
 
@@ -442,8 +449,11 @@ This switches the shadow to `StrictHostKeyChecking=yes`.
 
 #### Compute nodes
 
-With `ForwardTarget: node` the shadow opens a second session, through the login
-node, to the compute node's own sshd, and checks that node's host key too.
+With `ForwardTarget: node`, and for the
+[reverse tunnel](#pod-to-cluster-the-reverse-tunnel) whatever the target, the shadow
+opens a second session, through the login node, to the compute node's own sshd, and
+checks that node's host key too. The one exception is a
+[user-space sshd](#a-user-space-sshd) reached with `Reverse.NodeExec: srun`.
 `NodeHostKeys` decides how:
 
 - `verify` (default): the key must be listed, and ssh runs with
@@ -505,8 +515,9 @@ scheduled anything — so at creation time there is no node to forward to yet.
    the pod spec, so kubelet refreshes it in place — restarting the shadow would
    change its pod IP, which Kubernetes has already been told is the offloaded pod's IP.
 4. The shadow's `wait-for-node` init container polls the mounted node name, and the
-   socket directory when the pod has [socket ports](#where-the-workload-listens),
-   then `ssh-forward` starts the tunnel.
+   socket directory when the pod has [socket ports](#where-the-workload-listens) or a
+   [reverse tunnel](#pod-to-cluster-the-reverse-tunnel). Then the tunnels start:
+   `ssh-forward` for exposed ports, `ssh-reverse` for a reverse tunnel.
 
 While the job is queued the shadow sits in `Init:0/1`. That is the expected state,
 not a failure — it reads as "waiting for the allocation" rather than a crash-looping
@@ -740,89 +751,188 @@ included, stay visible to the job.
 ### How it works
 
 ```
-job container ──connect name:port──> 127.x.y.z:port on the compute node (sshd, -R)
+job container ──connect name:port──> relay on 127.x.y.z:port (started by the job)
+                                           │ checks the caller's user
+                                           ▼
+                            <socket dir>/rev-<port>.sock (compute node's sshd, -R)
                                            │ carried back over the shadow's ssh session
                                            ▼
                      shadow pod (ssh-reverse) ──dials──> Service inside the cluster
 ```
 
-1. The shadow runs a second container, `ssh-reverse`. Once the compute node is known
-   it starts a user-space `sshd -i` there, as the job's user, through a command run
-   on the login node (`ssh <node>` or `srun --overlap`, see `NodeExec`). The
-   command's stdin and stdout are that sshd's transport: it listens on no port. Its
-   host key and the one client key it accepts are generated for each start and
-   never leave the shadow.
-2. The shadow connects to it with `ssh -R`. Each forward is bound on the compute
-   node, on a loopback address derived from the pod UID (`127.x.y.z`), and every
-   connection made there is dialled from the shadow pod, inside the cluster.
-3. The virtual kubelet prepends a snippet to the pod's `slurm-job.vk.io/pre-exec`.
-   It maps the forwarded names to that address in the container's `/etc/hosts`
-   (through `SINGULARITY_BIND` and `APPTAINER_BIND`) and holds the workload until the
-   tunnel answers, for at most ten minutes. The workload itself is unchanged.
-
-The tunnel is rebuilt if the job moves to another node, like the forwards.
+1. The virtual kubelet prepends a snippet to the pod's `slurm-job.vk.io/pre-exec`.
+   When the job starts, it launches a small Python relay on the compute node that
+   listens on a loopback address derived from the pod UID (`127.x.y.z`), one port per
+   forward, for as long as the job runs. It maps the forwarded names to that address
+   in the container's `/etc/hosts` (through `SINGULARITY_BIND` and `APPTAINER_BIND`)
+   and holds the workload until the tunnel is up, for at most ten minutes. The
+   workload itself is unchanged.
+2. The shadow runs a second container, `ssh-reverse`. Once the compute node and the
+   pod's [socket directory](#the-socket-directory) are known, it logs in to the
+   compute node's own sshd through the login node, the same way and with the same
+   [host key check](#compute-nodes) as `ForwardTarget: node`, and asks it with
+   `ssh -R` for one unix socket per forward, `rev-<port>.sock` in the socket
+   directory. The session runs nothing but a wait that ends when the job removes
+   that directory.
+3. For each connection, the relay checks that it comes from the job's user and hands
+   it to the matching socket. sshd carries it back over the session, and the shadow
+   pod dials the endpoint from inside the cluster.
+
+The tunnel is rebuilt if the job moves to another node, like the forwards. While it is
+down, the relay keeps its address and closes new connections at once.
 
 ### What the site has to allow
 
-- A command on the compute node, run from the login node as the job's user. Either
-  `ssh <node>` into a node where that user has a running job, which sites with
-  `pam_slurm_adopt` allow, or `srun --jobid=<job> --overlap`, which needs nothing
-  beyond Slurm. `NodeExec` picks one; a pod can override it.
-- `sshd` on the compute node (`Reverse.SshdPath`) and, for owner-only forwards,
-  `python3`. Both are stock on most distributions and neither needs root.
-- Nothing in the login node's `sshd_config`: the forwards are bound by the sshd the
-  shadow starts, not by the site's.
+The reverse tunnel always ends on the compute node, whatever `ForwardTarget` says.
+With the default `Reverse.NodeExec: ssh` it gets there the way
+[`ForwardTarget: node`](#with-forwardtarget-node) does, so it needs the same: `-W` or
+`nc <node> 22` on the login node, as `ForwardMode` says, and an ssh login to a node
+where the account has a running job. With `Reverse.NodeExec: srun` it needs `squeue`
+and `srun --jobid=<job> --overlap` on the login node instead and, with
+`Reverse.Sshd: node`, a netcat on the compute nodes (`ExecConnectCommand`) and a login
+to the node's sshd from `127.0.0.1`. On top of that:
+
+- **Remote forwarding to a unix socket on the compute node's sshd**,
+  `AllowStreamLocalForwarding yes` (the default) or `remote`, in both forward modes. A
+  [SOCKS proxy](#endpoints-not-known-in-advance) also needs `AllowTcpForwarding yes`
+  or `remote` there.
+- **A socket directory** from the plugin: `SocketDirRoot` in the Slurm plugin, see
+  [The socket directory](#the-socket-directory). The relay and the sockets live there.
+- **`python3` on the compute nodes** (`Reverse.PythonPath`), which runs the relay. It is
+  stock on most distributions and needs no root.
+
+Where the compute node's sshd refuses that forwarding, an sshd the shadow starts in
+the job can take its place, see [A user-space sshd](#a-user-space-sshd).
 
 ### Annotations
 
 | Annotation | Value |
 | --- | --- |
 | `interlink.eu/reverse-forwards` | Space-separated endpoints, each `host:port` or `listen-port:host:port` |
-| `interlink.eu/reverse-socks-port` | Port of a SOCKS5 proxy for endpoints not listed; off when unset |
-| `interlink.eu/reverse-bind` | The `127.x.y.z` address to use instead of the derived one |
-| `interlink.eu/reverse-node-exec` | `ssh` or `srun`, overriding `Reverse.NodeExec` |
-| `interlink.eu/reverse-owner-only` | `true` (default) or `false`, see below |
-
-A Service port below 1024 cannot be bound by an unprivileged sshd on the compute
-node: give it a listen port, `6443:kubernetes.default.svc.cluster.local:443`, and the
-job connects to port 6443. The name resolves to the right address either way. An
+| `interlink.eu/reverse-socks-port` | Port of a SOCKS5 proxy for endpoints not listed; off when unset, refused unless the virtual kubelet sets `Reverse.AllowSocks` |
+| `interlink.eu/reverse-bind` | The `127.x.y.z` address the relay listens on, instead of the derived one |
+
+The relay runs as the job's user, so it cannot bind a port below 1024: give such an
+endpoint a listen port, `6443:kubernetes.default.svc.cluster.local:443`, and the job
+connects to port 6443. The name resolves to the right address either way. An
 annotation that does not parse is refused when the pod is created, with the reason in
 the virtual kubelet log and the pod's events.
 
+The carrier to the compute node and the sshd that makes the forwards are the site's
+choice, set in the virtual kubelet configuration (`Reverse.NodeExec`,
+`Reverse.Sshd`); a pod cannot change them.
+
 ### Endpoints not known in advance
 
-`interlink.eu/reverse-socks-port: "1080"` adds a SOCKS5 proxy on the same address. A
-program told to use it, `ALL_PROXY=socks5h://127.x.y.z:1080` for curl and Python
-requests, reaches any cluster address by name, resolved inside the cluster at connect
-time. It is off unless asked for, and it is a plain loopback listener that every user
-of that compute node can also use, so declare what you can and keep SOCKS for the rest.
+:::warning
+A SOCKS proxy lets every user of the compute node reach, through it, any endpoint the
+shadow reaches in the cluster. ssh has no unix socket form of it, so it listens on
+`127.0.0.1` of the node and neither the relay nor the socket directory protects it.
+It is off unless the virtual kubelet sets `Reverse.AllowSocks: true`. A site that
+agreed to a reverse tunnel with fixed endpoints and no proxy has not agreed to this:
+ask before turning it on.
+:::
+
+With `AllowSocks`, `interlink.eu/reverse-socks-port: "1080"` adds a SOCKS5 proxy on
+`127.0.0.1:1080` of the compute node. A program told to use it,
+`ALL_PROXY=socks5h://127.0.0.1:1080` for curl and Python requests, reaches any cluster
+address by name, resolved inside the cluster at connect time. Declare what you can in
+`interlink.eu/reverse-forwards` and keep SOCKS for the rest.
+
+- Without `AllowSocks`, a pod that sets the annotation is refused when it is created.
+- With it, the pod gets a `ReverseSocksProxy` warning event, and the virtual kubelet
+  and the shadow log a warning.
+- Every pod's proxy is on `127.0.0.1`, so two pods on one compute node cannot use the
+  same port. If another user already listens on it, the job fails at start. If
+  another pod of the same account does, the job cannot tell, and the shadow's tunnel
+  fails instead, see [Troubleshooting](#troubleshooting).
 
 ### Who else can use the forwards
 
-A port on a loopback address is reachable by every user of the compute node. By
-default each forward therefore ends in a unix socket that only the job's user can
-open, with a small relay in front, run by the sshd session on the node, that accepts
-a connection only when the connecting socket belongs to that user (read from
-`/proc/net/tcp`). `interlink.eu/reverse-owner-only: "false"` binds the ports directly
-instead: no relay, no `python3` on the node, and no protection from other users.
+Only the job's user. A port on a loopback address is reachable by every user of the
+compute node, so the relay accepts a connection only when the connecting socket
+belongs to the job's user (read from `/proc/net/tcp`), and the socket behind it is
+created by sshd with mode 0600 inside the 0700 socket directory. This is always on.
+
+The job holds the relay's address from its start to its end, so nobody else can take
+it later, not even while the tunnel is being rebuilt. If something already listens
+there when the job starts, the relay cannot bind and the job fails rather than run
+with a stranger's listener in its place.
+
+It does not keep out other jobs of the same account, or the node's administrators.
+
+### A user-space sshd
+
+With `Reverse.Sshd: userspace` the forwards are made by an sshd the shadow starts on
+the compute node, as the job's user, instead of the node's own. It runs as `sshd -i`,
+with an ssh session's stdin and stdout as its transport, so it listens on no port. Its
+host key and the one client key it accepts are generated for each start and never
+leave the shadow, and its configuration, written to the socket directory, allows
+remote forwarding and nothing else. The relay, the sockets and the owner check are the
+same as above.
+
+It needs the sshd binary on the node (`Reverse.SshdPath`), and a key allowed to run
+any command on the node, or on the login node with `Reverse.NodeExec: srun`, because
+the shadow sends the script that prepares it. It does not work with a
+[key for the tunnel only](#a-key-for-the-tunnel-only).
+
+With `Reverse.NodeExec: srun` no session reaches the node's own sshd, so
+`NodeHostKeys` does not apply: the shadow checks only the host key it generated,
+which it sends through the login node.
+
+### With a key for the tunnel only
+
+With the default `Reverse.Sshd: node`, a
+[key for the tunnel only](#a-key-for-the-tunnel-only) works. Keep `port-forwarding`
+in its `authorized_keys` line, since `-R` needs it in both forward modes. In the
+wrapper, define the directory pattern next to `h=`:
+
+```sh
+d='/[A-Za-z0-9._/-]+'
+```
+
+and insert these lines into its `if`, before its last condition, the
+`ok 'cat /etc/ssh/ssh_known_hosts ...'; then` line:
+
+```sh
+   ok "rm -f $d/rev-\*\.sock" ||
+   ok "while \[ -d $d \]; do sleep 10; done" ||
+   ok "squeue -h --me -t R -n [0-9a-f-]{36} -o %i" ||
+   ok "srun --jobid=[0-9]+ --overlap -u -n1 --cpus-per-task=1 $NC 127\.0\.0\.1 22" ||
+```
+
+The first two run on the compute node: one clears the sockets a previous session left
+behind, before the forwards are requested, and the other is the wait that holds the
+session. The last two are the `srun` carrier, run on the login node. With it the
+compute node's sshd sees the session come from `127.0.0.1`, so add that address to
+`from=`.
 
 ### Limits
 
 - TCP only.
-- The listeners exist on the node that runs the batch script. Ranks of a multi-node
-  job on other nodes have no tunnel.
+- The relay and the sockets exist only on the batch node, the one the plugin reports
+  as `NodeName`, and so does the file that maps the names, as long as `TMPDIR` is
+  node-local. Ranks of a multi-node job on other nodes then fail instead of reaching
+  anything. With `TMPDIR` on a shared filesystem they resolve the names to an address
+  with no relay behind it, where another user of their node could listen.
+  Multi-node jobs are out of scope.
 - The tunnel comes up once the shadow has learned the compute node, a status poll or
   two after the job starts; the pre-exec waits for it.
 - The pre-exec assumes the Slurm plugin with Singularity or Apptainer, as the mesh
-  pre-exec does.
-- `/tmp/il-rev-<shadow>` on the compute node holds the keys and sockets while the
-  tunnel is up; the session removes it when the tunnel ends.
+  pre-exec does, and a plugin that reports a socket directory.
+- The sockets, the relay and, with a user-space sshd, its keys and configuration live
+  in the pod's socket directory. The job removes it when it ends, and the tunnel
+  session and the relay end with it.
 
 ### Performance
 
-`NodeExec: srun` carries the tunnel through Slurm's I/O forwarding, which has lower
-throughput than `ssh <node>`. Prefer `ssh` and keep `srun` for nodes the login node
-cannot reach over ssh. The owner-only relay adds no measurable cost.
+`Reverse.NodeExec: srun` reaches the compute node with `srun --jobid=<job> --overlap`
+on the login node, running inside the allocation `ExecConnectCommand` to
+`127.0.0.1:22` with `Reverse.Sshd: node`, or the user-space sshd itself with
+`userspace`, so the tunnel goes through Slurm's I/O forwarding, which has lower
+throughput than ssh. It is a last resort, for nodes the login node cannot reach over
+ssh. Under Kerberos it is also the one case where the login hop delegates the ticket,
+see [Kerberos](#kerberos). The owner-only relay adds no measurable cost.
 
 ---
 
@@ -838,10 +948,10 @@ kubectl get configmap <shadow>-node -o jsonpath='{.data.compute-node}'
 Empty means no plugin has reported one yet.
 
 **`waiting for the plugin to report a socket directory` in the `wait-for-node`
-log.** The pod lists socket ports, but the plugin has not reported a socket
-directory, and one without socket support never will: set `SocketDirRoot` in the
-Slurm plugin. After `NodeWaitTimeout` the init container fails with
-`timed out waiting for a socket directory`. What has been reported so far:
+log.** The pod lists socket ports or asks for a reverse tunnel, but the plugin has
+not reported a socket directory, and one without socket support never will: set
+`SocketDirRoot` in the Slurm plugin. After `NodeWaitTimeout` the init container fails
+with `timed out waiting for a socket directory`. What has been reported so far:
 
 ```bash
 kubectl get configmap <shadow>-node -o jsonpath='{.data.socket-dir}'
@@ -879,8 +989,8 @@ on the compute node. If the compute node refuses, have the site allow it or use
 `KnownHostsConfigMap` does not contain the login node's current key. Re-run
 `ssh-keyscan` and update the ConfigMap.
 
-With `node` it can also be the compute node's key. A line
-`no host key for <node>: ...` just before it means nothing in the list matches the
+With `node`, and in the `ssh-reverse` log, it can also be the compute node's key. A
+line `no host key for <node>: ...` just before it means nothing in the list matches the
 node's name: add its key, a host pattern or an `@cert-authority` line, as in
 [Compute nodes](#compute-nodes). That line is not printed when the list holds any
 `@cert-authority` line, so without it, first check that a line covers the reported
@@ -901,14 +1011,50 @@ already contains a different object under the credential's name. Rename the
 credential, or set `ReplicateCredentials: false` and provision it yourself.
 
 **`interLink: the reverse tunnel did not come up within 600s` in the job log.** The
-shadow's `ssh-reverse` container never got through; read its log. `could not prepare
-... on <node>` means the command to the compute node was refused, so try
-`NodeExec: srun`; `no running job named ... on <node>` in `srun` mode means `squeue`
-on the login node did not list a running job under the pod's UID.
+shadow's `ssh-reverse` container never got through. In its log each failed attempt
+ends in `the reverse tunnel to <node> is down; retrying in 30s`, and the lines before
+it say why:
+
+- `could not clear <dir> on <node>`: the session to the compute node failed. It goes
+  the same way as with `ForwardTarget: node`, so the lines before it read as in the
+  entries above: a host key the list does not accept, or a login the node refuses.
+- `remote port forwarding failed for listen path <dir>/rev-<port>.sock`: the compute
+  node's sshd refuses forwarding to a unix socket (`AllowStreamLocalForwarding`), or
+  the key lacks `port-forwarding`. Have the site allow it, or set
+  `Reverse.Sshd: userspace`. For the SOCKS port the line ends in `listen port <port>`:
+  the setting is `AllowTcpForwarding`, or something already listens on
+  `127.0.0.1:<port>` on that node (`ss -ltnp`), possibly another pod of the same
+  account. Pick another `interlink.eu/reverse-socks-port`.
+- `could not prepare <dir> on <node>`, with `Reverse.Sshd: userspace`: the session
+  that writes the user-space sshd's files into the socket directory failed (the login
+  to the node, `srun`, or `cd` into the directory). A wrong `Reverse.SshdPath` shows
+  up instead as the inner ssh failing to connect, just before the retry line.
+- `no running job named ...`, with `Reverse.NodeExec: srun`: `squeue` on the login
+  node did not list a running job under the pod's UID.
+
+**`interLink: the reverse tunnel needs a plugin with SocketDirRoot set` in the job
+log.** The plugin gave the job no socket directory. Set `SocketDirRoot` in the Slurm
+plugin, see [The socket directory](#the-socket-directory).
+
+**`interLink: the reverse relay stopped, see above` in the job log.** The relay did
+not start. `relay: cannot listen on 127.x.y.z:<port>: ...` just before it means
+something already listens on the pod's relay address; `ss -ltnp` on the node shows
+what. If it belongs to another user, the job failed on purpose rather than run behind
+it: report it to the site. Without that line, check `Reverse.PythonPath`.
+
+**`interLink: another user listens on the SOCKS port` in the job log.** Another user
+holds `127.0.0.1:<port>` on the compute node. Pick another
+`interlink.eu/reverse-socks-port`.
 
 **Pod creation fails with `interlink.eu/reverse-forwards: ...`.** The annotation did
 not parse; the message names the entry and what is wrong with it.
 
+**Pod creation fails with `... does not allow a SOCKS proxy`.** The pod sets
+`interlink.eu/reverse-socks-port` and the virtual kubelet does not set
+`Reverse.AllowSocks`. List the endpoints in `interlink.eu/reverse-forwards` instead,
+or read [Endpoints not known in advance](#endpoints-not-known-in-advance) before
+asking the site.
+
 **An nginx wstunnel Deployment appears instead of the SSH shadow.** A custom template
 file is overriding the built-in one. The virtual kubelet prefers
 `Network.WstunnelTemplatePath` over its own templates; the Helm chart stops mounting
@@ -920,6 +1066,6 @@ it in `ssh` mode, but a hand-written config may still point at one.
 
 - [Discussion #535](https://github.com/interlink-hq/interLink/discussions/535) — the original proposal
 - [Issue #545](https://github.com/interlink-hq/interLink/issues/545) — in-cluster SSH deployment
-- [Issue #548](https://github.com/interlink-hq/interLink/issues/548) — pod to cluster connectivity without WireGuard
+- [Issue #548](https://github.com/interlink-hq/interLink/issues/548): pod to cluster connectivity without WireGuard
 - [Wstunnel Configuration](./10-wstunnel-configuration.mdx)
 - [Mesh Network Configuration](./13-mesh-network-configuration.mdx)

From fd2368fc8c1cafd8c3d508c4993df8271b0f3ef0 Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 16 Oct 2026 21:34:40 +0200
Subject: [PATCH 4/5] docs: wrapper line for the reverse tunnel's hold command

The hold command the shadow runs on the node ends with "echo;", so the
wrapper pattern given for a tunnel-only key refused it and the reverse
tunnel never came up with such a key. Match the command as rendered.

Part of #548

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 docs/docs/guides/14-ssh-tunnel-configuration.mdx | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 39ee8411..627aeadd 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -896,7 +896,7 @@ and insert these lines into its `if`, before its last condition, the
 
 ```sh
    ok "rm -f $d/rev-\*\.sock" ||
-   ok "while \[ -d $d \]; do sleep 10; done" ||
+   ok "while \[ -d $d \]; do sleep 10; echo; done" ||
    ok "squeue -h --me -t R -n [0-9a-f-]{36} -o %i" ||
    ok "srun --jobid=[0-9]+ --overlap -u -n1 --cpus-per-task=1 $NC 127\.0\.0\.1 22" ||
 ```

From f3ab33625947b54f5238896037c63bee1b6e9ecb Mon Sep 17 00:00:00 2001
From: Matteo Bunino <48362942+matbun@users.noreply.github.com>
Date: Fri, 30 Oct 2026 13:28:31 +0100
Subject: [PATCH 5/5] docs: reverse tunnel review fixes

The derived reverse-bind address makes a collision unlikely, not
impossible, and the annotation resolves one. The user-space sshd's host
key and the client's public key are written to the socket directory;
only the client's private key stays in the shadow. The tunnel-only key
wrapper's directory pattern is narrowed to SocketDirRoot plus the pod's
12-digit directory, since its rm -f line could otherwise delete any
rev-*.sock the account owns, and its hold, squeue and srun lines follow
the commands the shadow now sends. Also documented: pod-vpn refuses the
reverse annotations, UDP-only pods get no ssh shadow, and the srun
carrier runs on the batch host.

Part of #548

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
---
 docs/docs/guides/09-pod-annotations.mdx       |  4 +--
 .../guides/14-ssh-tunnel-configuration.mdx    | 34 ++++++++++++-------
 2 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/docs/docs/guides/09-pod-annotations.mdx b/docs/docs/guides/09-pod-annotations.mdx
index d842502e..83e6985d 100644
--- a/docs/docs/guides/09-pod-annotations.mdx
+++ b/docs/docs/guides/09-pod-annotations.mdx
@@ -344,7 +344,7 @@ spec:
 - The virtual kubelet adds the names to the container's `/etc/hosts` and waits for the tunnel before the workload starts
 - A Service FQDN also resolves as `name.namespace.svc` and `name.namespace`, and as `name` in the pod's own namespace
 - An entry that does not parse, or a listen port below 1024, is refused when the pod is created
-- Needs `Network.SSH.Reverse.Enabled: true`; otherwise the pod is refused rather than run without its tunnel
+- Needs `Network.SSH.Reverse.Enabled: true` and no `interlink.eu/pod-vpn` on the pod; otherwise the pod is refused rather than run without its tunnel
 - Needs a plugin that gives the job a socket directory (`SocketDirRoot` in the Slurm plugin); otherwise the job fails at start
 
 **Default:** Not set (no connections back into the cluster)
@@ -385,7 +385,7 @@ metadata:
 The `127.x.y.z` address of the owner-only relay: the forwards listen there, inside the job.
 
 **Behavior:**
-- Derived from the pod UID when unset, so two pods on one node never share it
+- Derived from the pod UID when unset, which makes two pods on one node unlikely to share it, though not impossible: if they do and forward the same listen port, the second job fails at start, and setting this annotation on one of them resolves it
 - Must be in `127.0.0.0/8` and not `127.0.0.1`
 - The job fails at start if something else already listens on that address and a forward's port
 
diff --git a/docs/docs/guides/14-ssh-tunnel-configuration.mdx b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
index 627aeadd..6f34145b 100644
--- a/docs/docs/guides/14-ssh-tunnel-configuration.mdx
+++ b/docs/docs/guides/14-ssh-tunnel-configuration.mdx
@@ -119,7 +119,8 @@ The login node needs no route to the pod's own ports: the last hop happens insid
 the compute node.
 
 `ssh -L` carries TCP only. UDP ports on the offloaded pod are skipped, with a
-warning in the virtual kubelet log.
+warning in the virtual kubelet log, and a pod that exposes only UDP ports gets no
+shadow unless it asks for the [reverse tunnel](#pod-to-cluster-the-reverse-tunnel).
 
 ---
 
@@ -866,10 +867,11 @@ It does not keep out other jobs of the same account, or the node's administrator
 With `Reverse.Sshd: userspace` the forwards are made by an sshd the shadow starts on
 the compute node, as the job's user, instead of the node's own. It runs as `sshd -i`,
 with an ssh session's stdin and stdout as its transport, so it listens on no port. Its
-host key and the one client key it accepts are generated for each start and never
-leave the shadow, and its configuration, written to the socket directory, allows
-remote forwarding and nothing else. The relay, the sockets and the owner check are the
-same as above.
+host key and the one client key it accepts are generated for each start. The client's
+private key never leaves the shadow; the host's private key and the client's public
+key are written to the socket directory, next to a configuration that allows remote
+forwarding and nothing else, and go away with it. The relay, the sockets and the
+owner check are the same as above.
 
 It needs the sshd binary on the node (`Reverse.SshdPath`), and a key allowed to run
 any command on the node, or on the login node with `Reverse.NodeExec: srun`, because
@@ -885,20 +887,25 @@ which it sends through the login node.
 With the default `Reverse.Sshd: node`, a
 [key for the tunnel only](#a-key-for-the-tunnel-only) works. Keep `port-forwarding`
 in its `authorized_keys` line, since `-R` needs it in both forward modes. In the
-wrapper, define the directory pattern next to `h=`:
+wrapper, define the directory pattern next to `h=`, with the plugin's
+`SocketDirRoot` in place of `/gpfs/projects/example/interlink`, its regex
+metacharacters escaped:
 
 ```sh
-d='/[A-Za-z0-9._/-]+'
+d='/gpfs/projects/example/interlink/[0-9a-f]{12}'
 ```
 
+Keep it this narrow: the `rm -f` line below lets the key delete any `rev-*.sock` the
+account owns under a directory `d` matches.
+
 and insert these lines into its `if`, before its last condition, the
 `ok 'cat /etc/ssh/ssh_known_hosts ...'; then` line:
 
 ```sh
    ok "rm -f $d/rev-\*\.sock" ||
-   ok "while \[ -d $d \]; do sleep 10; echo; done" ||
-   ok "squeue -h --me -t R -n [0-9a-f-]{36} -o %i" ||
-   ok "srun --jobid=[0-9]+ --overlap -u -n1 --cpus-per-task=1 $NC 127\.0\.0\.1 22" ||
+   ok "while \[ -d $d \]( && \[ -S $d/rev-[0-9]{1,5}\.sock \])*; do sleep 10; echo; done" ||
+   ok "squeue -h --me -t R -n [0-9a-f-]{36} -o '%i %B'" ||
+   ok "srun --jobid=[0-9]+ --overlap -u -n1 -w [A-Za-z0-9][A-Za-z0-9._-]* --cpus-per-task=1 $NC 127\.0\.0\.1 22" ||
 ```
 
 The first two run on the compute node: one clears the sockets a previous session left
@@ -926,9 +933,10 @@ compute node's sshd sees the session come from `127.0.0.1`, so add that address
 
 ### Performance
 
-`Reverse.NodeExec: srun` reaches the compute node with `srun --jobid=<job> --overlap`
-on the login node, running inside the allocation `ExecConnectCommand` to
-`127.0.0.1:22` with `Reverse.Sshd: node`, or the user-space sshd itself with
+`Reverse.NodeExec: srun` reaches the compute node with
+`srun --jobid=<job> --overlap -w <batch host>` on the login node, running inside the
+allocation `ExecConnectCommand` to `127.0.0.1:22` with `Reverse.Sshd: node`, or the
+user-space sshd itself with
 `userspace`, so the tunnel goes through Slurm's I/O forwarding, which has lower
 throughput than ssh. It is a last resort, for nodes the login node cannot reach over
 ssh. Under Kerberos it is also the one case where the login hop delegates the ticket,