Repository navigation
Conversation
✅ Deploy Preview for interlink-dev canceled.
|
This was referenced Oct 8, 2026
matbun
force-pushed
the
feat/ssh-reverse-tunnel
branch
2 times, most recently
from
October 8, 2026 21:02
d1b5136 to
402dab5
Compare
matbun
force-pushed
the
feat/ssh-reverse-tunnel
branch
2 times, most recently
from
October 23, 2026 13:57
0d998f2 to
d82cc54
Compare
matbun
added a commit
to matbun/interLink
that referenced
this pull request
Oct 23, 2026
…terlink-hq#577 The branch carried early copies of interlink-hq#580, interlink-hq#583, interlink-hq#584 and interlink-hq#585 folded into one commit. It now holds main as of interlink-hq#571 (interlink-hq#580, interlink-hq#581, interlink-hq#582, interlink-hq#583, interlink-hq#585 as merged), interlink-hq#584 as reviewed and the rebased reverse tunnel, with the demo's own commits re-applied: TenantNamespace, RefuseForeignPods, kept init containers on confined nodes and their logs. Fixes found while porting: - interlink-hq#571 refreshed the pod IP through computeShadowResourceIdentity, which ignores TenantNamespace; on a confined node it looked for the shadow in <ns>-shadow and never moved the IP. It uses the provider's identity now. - The completed kept-init wait saves the init containers' logs before it takes the wait over, which cancels its context. - RefuseForeignPods handles the errors of SetWatchErrorHandler and AddEventHandler (errcheck). - The node ConfigMap patch is a struct, so interlink-hq#584 and interlink-hq#577 together do not trip goconst on "metadata". Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
matbun
force-pushed
the
feat/ssh-reverse-tunnel
branch
from
October 23, 2026 16:23
d82cc54 to
f4777a9
Compare
With ShadowMode ssh a pod can now reach cluster endpoints from the job, without WireGuard and without a network namespace. The site enables it with Network.SSH.Reverse.Enabled; the pod lists its endpoints in the interlink.eu/reverse-forwards annotation. The shadow gets an ssh-reverse container. Once the compute node is known it starts a user-space sshd -i there through the login node (ssh <node>, or srun --jobid --overlap with Reverse.NodeExec srun), as the job's user and with the command's stdio as transport, and binds the forwards on it with ssh -R on a loopback address derived from the pod UID. A snippet prepended to slurm-job.vk.io/pre-exec maps the names in the container's /etc/hosts and waits for the tunnel, so the workload uses the Service name and port unchanged. Each forward ends in a unix socket only the job's user can open, with a relay checking the connecting uid, unless interlink.eu/reverse-owner-only is false. interlink.eu/reverse-socks-port adds a SOCKS5 proxy for endpoints not declared, off unless asked for. The annotations are parsed and validated in Go, and a pod asking for the tunnel where it is not enabled is refused at creation rather than run without it. shouldCreateShadow is also true for a pod with forwards and no exposed port; the template then renders neither ssh-forward nor a Service. Under Kerberos the login hop delegates the ticket for pods with a reverse tunnel, because ssh or srun on the login node needs it. Part of interlink-hq#548 Co-Authored-By: Claude Fable 5.1 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
The pod controller's event broadcaster had no sink, so every Event recorded through it was dropped. It now records to the API server, and the provider gets the same recorder, so the provider can also put Events on the pods it runs. The helm chart's RBAC already lets the virtual kubelet create Events. Part of interlink-hq#548 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
The first version of the reverse tunnel left gaps a site review would not accept. The shadow reached the node with an unchecked `ssh <node>` from the login node, so the cluster endpoints went to whatever host answered to that name. The owner-only relay ran as the tunnel's session command, so on every reconnect its address was free for another user of the node to take. Owner-only and the carrier could be switched per pod, and any pod could open a SOCKS proxy that every user of the node can use. The forwards are now made by the compute node's own sshd (Network.SSH.Reverse.Sshd "node", the default), reached through the same checked node hop as ForwardTarget "node", so NodeHostKeys applies to the reverse tunnel too. Each one is `-R` to rev-<port>.sock in the pod's socket directory. The session holds only while that directory exists, and it writes to its channel so that it also ends on the node when the connection drops. The user-space sshd stays as Sshd "userspace" for sites whose node sshd refuses forwarding: it keeps its files in the socket directory, drops GatewayPorts and ForceCommand, and is started through the same node hop or srun. The relay moves out of the shadow and into the job. The pre-exec writes it to the socket directory and starts it with python3 -I before the workload; it binds every port first and fails the job if one is taken, signals readiness with rev-relay.ready, and exits when the job's shell or the directory goes. The job therefore holds its address for its whole life and a tunnel outage refuses connections instead of handing them to someone else. Since it now lives as long as the job, an accept or thread error no longer ends a listener, and each kind of failure is logged once and then every 1000th time: the log is the job's output, which any user of the node could otherwise grow by connecting in a loop. The /etc/hosts copy is a mktemp file in node-local temporary space, because only the batch node has the relay. Owner-only is no longer optional, and the carrier is the site's decision: interlink.eu/reverse-owner-only and interlink.eu/reverse-node-exec are gone (never released; pods setting them are not refused). A SOCKS proxy needs Reverse.AllowSocks; a pod asking for one without it is refused at creation. With it the proxy binds 127.0.0.1 on the node, in a session of its own so that a port another user takes while the tunnel is down cannot take the listed forwards down too; the job waits until the listener is its own user's, and the pod gets a VK warning, a Warning Event ReverseSocksProxy and a WARNING line in the shadow log. Kerberos delegation on the login hop is on only with Reverse.NodeExec srun, where srun on the login node needs the ticket. The reverse tunnel needs a plugin that sets SocketDirRoot; without it the job fails at start with a message saying so. Part of interlink-hq#548 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
matbun
force-pushed
the
feat/ssh-reverse-tunnel
branch
from
October 28, 2026 13:53
f4777a9 to
c219af0
Compare
matbun
marked this pull request as ready for review
October 28, 2026 13:55
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can skip requested reverse-forward setup, discard existing bind mounts, and generate invalid Deployments for UDP-only pods.
6 open issues
Reject reverse annotations with pod VPN annotation · New Preserve legacy bindings when setting APPTAINER_BIND · New Skip empty tunnel shadows for UDP-only pods · New Check IPv4-mapped clients in the tcp6 ownership table · New Pin carrier steps to the allocation batch host · New Exit relay when forwarded sockets disappear · New
What changed in this PR
Adds opt-in SSH reverse tunnels so offloaded pods can reach cluster endpoints without WireGuard or a network namespace, extending the existing SSH shadow networking.
Changes:
- Adds reverse-forward annotations, configuration validation, and workload setup.
- Introduces an owner-only relay, node/userspace SSH forwarding, and Slurm carrier support.
- Adds gated SOCKS access with warnings, Kubernetes Event delivery, and reverse-tunnel tests.
| File | Description |
|---|---|
pkg/virtualkubelet/virtualkubelet.go |
Integrates reverse tunnels into shadow creation and pod validation. |
pkg/virtualkubelet/templates/shadow-ssh-template.yaml |
Renders reverse SSH sessions and optional forwarding containers. |
pkg/virtualkubelet/templates/reverse-relay.py |
Relays same-user loopback connections to Unix sockets. |
pkg/virtualkubelet/shadow_ssh.go |
Normalizes reverse configuration and host-key requirements. |
pkg/virtualkubelet/shadow_ssh_reverse.go |
Parses annotations and generates workload tunnel setup. |
pkg/virtualkubelet/shadow_ssh_reverse_test.go |
Tests validation, rendered scripts, warnings, and relay behavior. |
pkg/virtualkubelet/config.go |
Defines SSH reverse-tunnel settings. |
cmd/virtual-kubelet/main.go |
Connects event recording to the Kubernetes API. |
Review effort: Balanced
- A pod with reverse annotations and interlink.eu/pod-vpn is refused at creation: pod-vpn disables the shadow, so the job would start without its forwards. - The hosts file bind no longer drops the job's own binds under Apptainer: APPTAINER_BIND starts from SINGULARITY_BIND when only the legacy variable is set, since Apptainer reads it only while APPTAINER_BIND is unset. - An ssh shadow is not created for a pod that exposes only UDP ports and asks for no reverse tunnel. ssh carries TCP only, and since the forward container is skipped without a TCP port, the Deployment had no container left with public-key auth. - The relay also looks up the client in /proc/net/tcp6: a client on an IPv6 socket, as a JVM opens by default, reaches the IPv4 listener through a v4-mapped address and was refused. - The srun carrier steps run on the batch host (squeue %B, srun -w), where the relay and the socket directory are, rather than on any node of a multi-node allocation. - The session holding the forwards also ends when one of its sockets is gone, as after a requeue on the same node recreated the socket directory under the same path; the SOCKS session still holds on the directory alone. Part of interlink-hq#548 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
errcheck on the echo server's io.Copy, goconst on the forward and the pod-vpn annotation (now test constants), and gosec G204 on the two commands the tests build themselves. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Matteo Bunino <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Part of #548.
With
ShadowMode: sshan offloaded pod can open connections back into the cluster, without WireGuard and without a network namespace. The site enables it withNetwork.SSH.Reverse.Enabled: true; the pod lists what it needs:The shadow's
ssh-reversecontainer opens a session to the compute node's own sshd through the login node, the same hop as the forward of #580, and asks it for onessh -Rper endpoint, each ending in a unix socket in the pod's socket directory. A relay started by the job holds a loopback address derived from the pod UID for the whole job and passes on only connections from the job's own user, and the job's/etc/hostsmaps the Service names to it, so the workload uses the names and ports unchanged. If a site's node sshd refuses forwarding,Reverse.Sshd: userspacestarts an sshd inside the job instead, andReverse.NodeExec: sruncarries the session through Slurm where ssh to the node is not possible, at lower throughput.interlink.eu/reverse-socks-portadds a SOCKS5 proxy for undeclared endpoints only where the virtual kubelet setsReverse.AllowSocks. It listens on127.0.0.1of the node, open to every user there, so a pod that asks for it gets a Warning event, the virtual kubelet and the shadow log a warning, and withoutAllowSocksthe pod is refused. The virtual kubelet's Events now reach the API server.Tested on three Slurm sites, with the node's sshd and, on one, with the user-space fallback: S3 and Redis by Service name, the Kubernetes API by FQDN with TLS verified, sockets
srw-------in adrwx------directory, SOCKS refused by default and working with its warnings when allowed. Unit tests cover the annotations, the rendered scripts in every mode (sh -n,bash -n) and the relay.Related issue : #548