Skip to content

docs: the reverse tunnel of the ssh shadow - #578

Open
matbun wants to merge 5 commits into
interlink-hq:mainfrom
matbun:docs/ssh-reverse-tunnel
Open

matbun wants to merge 5 commits into
interlink-hq:mainfrom
matbun:docs/ssh-reverse-tunnel

Conversation

@matbun

@matbun matbun commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Part of #548. Documents #577.

A section in the SSH tunnel guide on the reverse tunnel: how it works with the node's own sshd and the relay the job starts, what the site has to allow, the annotations, the user-space and srun fallbacks and their cost, the SOCKS proxy behind a warning and Reverse.AllowSocks, who else on the node can use what, the limits, the lines a tunnel-only key's wrapper needs, and troubleshooting. The pod annotations guide gets the interlink.eu/reverse-* annotations.

The Docusaurus build passes.


Related issue : #548

@netlify

netlify Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for interlink-dev ready!

Name Link
🔨 Latest commit f3ab336
🔍 Latest deploy log https://app.netlify.com/projects/interlink-dev/deploys/6ae48d778d06e99524627807
😎 Deploy Preview https://deploy-preview-578--interlink-dev.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@matbun
matbun force-pushed the docs/ssh-reverse-tunnel branch 2 times, most recently from 4849262 to 73ae988 Compare October 16, 2026 18:34
@matbun
matbun force-pushed the docs/ssh-reverse-tunnel branch from 943a5ff to 37de0d5 Compare October 23, 2026 13:46
matbun and others added 4 commits October 28, 2026 14:53
A section in the SSH tunnel guide (how it works, what the site has to
allow, the annotations, SOCKS for undeclared endpoints, who else on the
node can use the forwards, limits, troubleshooting), the Reverse options
in the options table, and the interlink.eu/reverse-* annotations in the
pod annotations reference. The note saying the ssh shadow gives the pod
no way back into the cluster is updated.

Part of interlink-hq#548

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
The NodeExec row linked to a Performance section that did not exist. Add it:
srun --overlap carries the tunnel through Slurm's I/O forwarding, which has
lower throughput than ssh to the node, so srun is for nodes the login node
cannot reach over ssh. The owner-only relay adds no measurable cost.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
The reverse tunnel now ends on the compute node's own sshd, reached
through the login node with the same host key check as ForwardTarget:
node, and the owner-only relay is started by the job and holds the
pod's loopback address until the job ends. The guide described a
user-space sshd started for every tunnel and a relay that lived only
as long as the session, so it is rewritten to match: how it works,
what the site has to allow (forwarding on the node's sshd, a plugin
socket directory, python3 on the nodes), and the user-space sshd as
the fallback for nodes whose sshd refuses forwarding.

Owner-only is always on, so its annotation and opt-out go, and so does
reverse-node-exec: the carrier is the site's decision and is set only
in the virtual kubelet configuration, with srun documented as the last
resort. The SOCKS proxy gets a warning box in both guides: it listens
on 127.0.0.1 of the node, is open to every user there, and is refused
unless the virtual kubelet sets Reverse.AllowSocks.

The tunnel-only key section gains the wrapper lines the reverse tunnel
needs, and says that the user-space sshd cannot work with such a key,
since it runs a script the shadow sends. Troubleshooting covers the new
job log messages and the refused SOCKS pod.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
The hold command the shadow runs on the node ends with "echo;", so the
wrapper pattern given for a tunnel-only key refused it and the reverse
tunnel never came up with such a key. Match the command as rendered.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>
@matbun
matbun force-pushed the docs/ssh-reverse-tunnel branch from 37de0d5 to fd2368f Compare October 28, 2026 13:53
@matbun
matbun marked this pull request as ready for review October 28, 2026 13:55
@dciangot
dciangot requested a balanced review from Copilot October 29, 2026 07:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documented SSH allowlist permits deletion of unrelated account-owned sockets, while the guides inaccurately describe host private-key storage and reverse-bind address uniqueness.

3 open issues
What changed in this PR

Documents pod-to-cluster connectivity over the SSH shadow introduced in #577, supporting the SSH-only deployment work in #548.

Changes:

  • Explains reverse-tunnel setup, operation, site requirements, and fallback modes.
  • Documents security boundaries, SOCKS opt-in, tunnel-only key configuration, limitations, and troubleshooting.
  • Adds the three interlink.eu/reverse-* annotations to the pod annotations guide.
File Description
docs/​docs/​guides/​14-ssh-tunnel-configuration.mdx Adds reverse-tunnel guidance and updates related SSH configuration and troubleshooting sections.
docs/​docs/​guides/​09-pod-annotations.mdx Documents reverse-forward, SOCKS-port, and relay-bind annotations.

Review effort: Balanced

Comment thread docs/docs/guides/09-pod-annotations.mdx Resolved
The derived reverse-bind address makes a collision unlikely, not
impossible, and the annotation resolves one. The user-space sshd's host
key and the client's public key are written to the socket directory;
only the client's private key stays in the shadow. The tunnel-only key
wrapper's directory pattern is narrowed to SocketDirRoot plus the pod's
12-digit directory, since its rm -f line could otherwise delete any
rev-*.sock the account owns, and its hold, squeue and srun lines follow
the commands the shadow now sends. Also documented: pod-vpn refuses the
reverse annotations, UDP-only pods get no ssh shadow, and the srun
carrier runs on the batch host.

Part of interlink-hq#548

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Matteo Bunino <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants