Skip to content

feat: let offloaded pods reach cluster endpoints over the ssh shadow - #577

Open
matbun wants to merge 5 commits into
interlink-hq:mainfrom
matbun:feat/ssh-reverse-tunnel
Open

matbun wants to merge 5 commits into
interlink-hq:mainfrom
matbun:feat/ssh-reverse-tunnel

Conversation

@matbun

@matbun matbun commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Part of #548.

With ShadowMode: ssh an offloaded pod can open connections back into the cluster, without WireGuard and without a network namespace. The site enables it with Network.SSH.Reverse.Enabled: true; the pod lists what it needs:

annotations:
  interlink.eu/reverse-forwards: "minio.user.svc.cluster.local:9000 6443:kubernetes.default.svc.cluster.local:443"

The shadow's ssh-reverse container opens a session to the compute node's own sshd through the login node, the same hop as the forward of #580, and asks it for one ssh -R per endpoint, each ending in a unix socket in the pod's socket directory. A relay started by the job holds a loopback address derived from the pod UID for the whole job and passes on only connections from the job's own user, and the job's /etc/hosts maps the Service names to it, so the workload uses the names and ports unchanged. If a site's node sshd refuses forwarding, Reverse.Sshd: userspace starts an sshd inside the job instead, and Reverse.NodeExec: srun carries the session through Slurm where ssh to the node is not possible, at lower throughput.

interlink.eu/reverse-socks-port adds a SOCKS5 proxy for undeclared endpoints only where the virtual kubelet sets Reverse.AllowSocks. It listens on 127.0.0.1 of the node, open to every user there, so a pod that asks for it gets a Warning event, the virtual kubelet and the shadow log a warning, and without AllowSocks the pod is refused. The virtual kubelet's Events now reach the API server.

Tested on three Slurm sites, with the node's sshd and, on one, with the user-space fallback: S3 and Redis by Service name, the Kubernetes API by FQDN with TLS verified, sockets srw------- in a drwx------ directory, SOCKS refused by default and working with its warnings when allowed. Unit tests cover the annotations, the rendered scripts in every mode (sh -n, bash -n) and the relay.


Related issue : #548

@netlify

netlify Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for interlink-dev canceled.

Name Link
🔨 Latest commit 3e6b527
🔍 Latest deploy log https://app.netlify.com/projects/interlink-dev/deploys/6ae495972a82d195745f4e24

@matbun
matbun force-pushed the feat/ssh-reverse-tunnel branch 2 times, most recently from d1b5136 to 402dab5 Compare October 8, 2026 21:02
@matbun
matbun force-pushed the feat/ssh-reverse-tunnel branch 2 times, most recently from 0d998f2 to d82cc54 Compare October 23, 2026 13:57
matbun added a commit to matbun/interLink that referenced this pull request Oct 23, 2026
…terlink-hq#577

The branch carried early copies of interlink-hq#580, interlink-hq#583, interlink-hq#584 and interlink-hq#585 folded into
one commit. It now holds main as of interlink-hq#571 (interlink-hq#580, interlink-hq#581, interlink-hq#582, interlink-hq#583, interlink-hq#585 as
merged), interlink-hq#584 as reviewed and the rebased reverse tunnel, with the demo's
own commits re-applied: TenantNamespace, RefuseForeignPods, kept init
containers on confined nodes and their logs.

Fixes found while porting:
- interlink-hq#571 refreshed the pod IP through computeShadowResourceIdentity, which
  ignores TenantNamespace; on a confined node it looked for the shadow in
  <ns>-shadow and never moved the IP. It uses the provider's identity now.
- The completed kept-init wait saves the init containers' logs before it
  takes the wait over, which cancels its context.
- RefuseForeignPods handles the errors of SetWatchErrorHandler and
  AddEventHandler (errcheck).
- The node ConfigMap patch is a struct, so interlink-hq#584 and interlink-hq#577 together do not
  trip goconst on "metadata".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
@matbun
matbun force-pushed the feat/ssh-reverse-tunnel branch from d82cc54 to f4777a9 Compare October 23, 2026 16:23
matbun and others added 3 commits October 28, 2026 14:52
With ShadowMode ssh a pod can now reach cluster endpoints from the job,
without WireGuard and without a network namespace. The site enables it
with Network.SSH.Reverse.Enabled; the pod lists its endpoints in the
interlink.eu/reverse-forwards annotation.

The shadow gets an ssh-reverse container. Once the compute node is known
it starts a user-space sshd -i there through the login node (ssh <node>,
or srun --jobid --overlap with Reverse.NodeExec srun), as the job's user
and with the command's stdio as transport, and binds the forwards on it
with ssh -R on a loopback address derived from the pod UID. A snippet
prepended to slurm-job.vk.io/pre-exec maps the names in the container's
/etc/hosts and waits for the tunnel, so the workload uses the Service
name and port unchanged. Each forward ends in a unix socket only the
job's user can open, with a relay checking the connecting uid, unless
interlink.eu/reverse-owner-only is false. interlink.eu/reverse-socks-port
adds a SOCKS5 proxy for endpoints not declared, off unless asked for.

The annotations are parsed and validated in Go, and a pod asking for the
tunnel where it is not enabled is refused at creation rather than run
without it. shouldCreateShadow is also true for a pod with forwards and
no exposed port; the template then renders neither ssh-forward nor a
Service. Under Kerberos the login hop delegates the ticket for pods with
a reverse tunnel, because ssh or srun on the login node needs it.

Part of interlink-hq#548

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
The pod controller's event broadcaster had no sink, so every Event
recorded through it was dropped. It now records to the API server, and
the provider gets the same recorder, so the provider can also put
Events on the pods it runs. The helm chart's RBAC already lets the
virtual kubelet create Events.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
The first version of the reverse tunnel left gaps a site review would
not accept. The shadow reached the node with an unchecked
`ssh <node>` from the login node, so the cluster endpoints went to
whatever host answered to that name. The owner-only relay ran as the
tunnel's session command, so on every reconnect its address was free
for another user of the node to take. Owner-only and the carrier could
be switched per pod, and any pod could open a SOCKS proxy that every
user of the node can use.

The forwards are now made by the compute node's own sshd
(Network.SSH.Reverse.Sshd "node", the default), reached through the
same checked node hop as ForwardTarget "node", so NodeHostKeys applies
to the reverse tunnel too. Each one is `-R` to rev-<port>.sock in the
pod's socket directory. The session holds only while that directory
exists, and it writes to its channel so that it also ends on the node
when the connection drops. The user-space sshd stays as Sshd
"userspace" for sites whose node sshd refuses forwarding: it keeps its
files in the socket directory, drops GatewayPorts and ForceCommand,
and is started through the same node hop or srun.

The relay moves out of the shadow and into the job. The pre-exec
writes it to the socket directory and starts it with python3 -I before
the workload; it binds every port first and fails the job if one is
taken, signals readiness with rev-relay.ready, and exits when the
job's shell or the directory goes. The job therefore holds its address
for its whole life and a tunnel outage refuses connections instead of
handing them to someone else. Since it now lives as long as the job,
an accept or thread error no longer ends a listener, and each kind of
failure is logged once and then every 1000th time: the log is the
job's output, which any user of the node could otherwise grow by
connecting in a loop. The /etc/hosts copy is a mktemp file in
node-local temporary space, because only the batch node has the relay.

Owner-only is no longer optional, and the carrier is the site's
decision: interlink.eu/reverse-owner-only and
interlink.eu/reverse-node-exec are gone (never released; pods setting
them are not refused). A SOCKS proxy needs Reverse.AllowSocks; a pod
asking for one without it is refused at creation. With it the proxy
binds 127.0.0.1 on the node, in a session of its own so that a port
another user takes while the tunnel is down cannot take the listed
forwards down too; the job waits until the listener is its own
user's, and the pod gets a VK warning, a Warning Event
ReverseSocksProxy and a WARNING line in the shadow log.

Kerberos delegation on the login hop is on only with Reverse.NodeExec
srun, where srun on the login node needs the ticket. The reverse
tunnel needs a plugin that sets SocketDirRoot; without it the job
fails at start with a message saying so.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
@matbun
matbun force-pushed the feat/ssh-reverse-tunnel branch from f4777a9 to c219af0 Compare October 28, 2026 13:53
@matbun
matbun marked this pull request as ready for review October 28, 2026 13:55
@dciangot
dciangot requested a balanced review from Copilot October 29, 2026 07:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical defects can skip requested reverse-forward setup, discard existing bind mounts, and generate invalid Deployments for UDP-only pods.

6 open issues
What changed in this PR

Adds opt-in SSH reverse tunnels so offloaded pods can reach cluster endpoints without WireGuard or a network namespace, extending the existing SSH shadow networking.

Changes:

  • Adds reverse-forward annotations, configuration validation, and workload setup.
  • Introduces an owner-only relay, node/userspace SSH forwarding, and Slurm carrier support.
  • Adds gated SOCKS access with warnings, Kubernetes Event delivery, and reverse-tunnel tests.
File Description
pkg/​virtualkubelet/​virtualkubelet.go Integrates reverse tunnels into shadow creation and pod validation.
pkg/​virtualkubelet/​templates/​shadow-ssh-template.yaml Renders reverse SSH sessions and optional forwarding containers.
pkg/​virtualkubelet/​templates/​reverse-relay.py Relays same-user loopback connections to Unix sockets.
pkg/​virtualkubelet/​shadow_ssh.go Normalizes reverse configuration and host-key requirements.
pkg/​virtualkubelet/​shadow_ssh_reverse.go Parses annotations and generates workload tunnel setup.
pkg/​virtualkubelet/​shadow_ssh_reverse_test.go Tests validation, rendered scripts, warnings, and relay behavior.
pkg/​virtualkubelet/​config.go Defines SSH reverse-tunnel settings.
cmd/​virtual-kubelet/​main.go Connects event recording to the Kubernetes API.

Review effort: Balanced

matbun and others added 2 commits October 30, 2026 13:28
- A pod with reverse annotations and interlink.eu/pod-vpn is refused at
  creation: pod-vpn disables the shadow, so the job would start without
  its forwards.
- The hosts file bind no longer drops the job's own binds under
  Apptainer: APPTAINER_BIND starts from SINGULARITY_BIND when only the
  legacy variable is set, since Apptainer reads it only while
  APPTAINER_BIND is unset.
- An ssh shadow is not created for a pod that exposes only UDP ports
  and asks for no reverse tunnel. ssh carries TCP only, and since the
  forward container is skipped without a TCP port, the Deployment had
  no container left with public-key auth.
- The relay also looks up the client in /proc/net/tcp6: a client on an
  IPv6 socket, as a JVM opens by default, reaches the IPv4 listener
  through a v4-mapped address and was refused.
- The srun carrier steps run on the batch host (squeue %B, srun -w),
  where the relay and the socket directory are, rather than on any node
  of a multi-node allocation.
- The session holding the forwards also ends when one of its sockets is
  gone, as after a requeue on the same node recreated the socket
  directory under the same path; the SOCKS session still holds on the
  directory alone.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
errcheck on the echo server's io.Copy, goconst on the forward and the
pod-vpn annotation (now test constants), and gosec G204 on the two
commands the tests build themselves.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants