Skip to main content
Version: Next

SSH Tunnel Configuration

Many HPC sites give compute nodes no outbound internet access, and many Kubernetes clusters have no publicly reachable ingress. The wstunnel shadow needs both: the workload dials out to an ingress and runs a wstunnel client there. On an air-gapped site it cannot.

The SSH shadow inverts the direction. Instead of the compute node dialing out, the cluster dials in to the site's SSH login node and forwards each exposed port from the compute node the job landed on:

browser → Ingress / Service → shadow pod (ssh -L) → HPC login node → compute node (Jupyter)

The only requirement is outbound SSH from the cluster to the login node. The offloaded pod runs nothing on its side: no wstunnel client, no WireGuard configuration, no pre-exec injection.

info

The SSH shadow replaces wstunnel, not full mesh. It exposes the offloaded pod's ports to the cluster. A workload that needs the other direction, in-cluster object storage or a message bus for example, gets it over the same login node with the reverse tunnel, without WireGuard and without a network namespace. Combining the SSH shadow with full mesh is rejected at startup.

Before you start: what the login node has to allow​

The shadow is an ordinary SSH client, so everything it needs is decided by the login node's sshd_config.

  1. AllowTcpForwarding yes for the account the shadow logs in as. This is the one that catches people out: plenty of HPC sites set AllowTcpForwarding no globally and re-enable it only for a subset of users, often only those who authenticate with MFA. Check it before anything else:

    # from your own machine, with the same key/principal the shadow will use
    ssh -N -L 19999:<login node>:22 <user>@<login node> &
    nc -z 127.0.0.1 19999 && head -c 40 < /dev/tcp/127.0.0.1/19999

    If the site refuses, the shadow still starts and still reports Ready — the failure only shows up when traffic arrives, as a connection reset at the client and this line in kubectl logs <shadow pod> -c ssh-forward:

    channel 1: open failed: administratively prohibited: open failed

    If the site will not allow it, switch to ForwardMode: exec, which relays through a command on the login node instead and needs no forwarding privilege.

  2. A route from the login node to the compute nodes, on the ports the pod exposes. ssh -L resolves and connects to the compute node from the login node, using whatever name the plugin reported. Verify with the name the plugin actually reports (hostname -f on the compute node for the Slurm plugin), not the short name:

    ssh <user>@<login node> "curl -sv http://<compute node fqdn>:<port>/"

  3. Outbound SSH from the cluster to the login node's port, from the namespaces shadows are created in. Network policies that restrict pod egress have to allow it.

ssh -L carries TCP only. UDP ports on the offloaded pod are skipped, with a warning in the virtual kubelet log.

Which shadow does what​

DirectionTriggerWorkload must run
wstunnel (default)cluster → podEnableTunnel + exposed portswstunnel client, dials out
sshcluster → podEnableTunnel + exposed portsnothing
ssh reversepod → clusterSSH.Reverse.Enabled + interlink.eu/reverse-forwardsnothing: the batch script waits for the tunnel
full meshbidirectionalFullMesh: true, every podmesh.sh (slirp4netns + WireGuard)

Configuration​

Virtual Kubelet​

# VirtualKubeletConfig.yaml
Network:
  EnableTunnel: true
  ShadowMode: ssh
  SSH:
    LoginHost: login.hpc.example.org
    User: alice
    KeySecret: hpc-ssh-key

EnableTunnel turns shadow pods on for offloaded pods with exposed ports; ShadowMode picks which shadow is rendered. Both are needed.

Helm​

virtualNode:
  network:
    enableTunnel: true
    shadowMode: ssh
    ssh:
      loginHost: login.hpc.example.org
      user: alice
      keySecret: hpc-ssh-key

See examples/ssh_tunnel.yaml in the chart repository for a complete deployment.

Options​

OptionDefaultDescription
LoginHost—SSH login node to forward through (required)
User—Login name on that node (required)
Port22Login node's SSH port
Imageghcr.io/interlink-hq/interlink/ssh-tunnel:<version>Image the shadow runs. Needs an ssh client, plus kinit for Kerberos
Authpublickeypublickey or kerberos
ForwardModeportforwardportforward (ssh -L) or exec (relay through a command on the login node)
ExecConnectCommandncCommand run on the login node in exec mode, invoked as <command> <node> <port>
KeySecret—Secret holding the private key (publickey)
KeySecretKeyid_ed25519Key inside KeySecret
KeytabSecret—Secret holding the keytab (kerberos)
KeytabSecretKeyuser.keytabKey inside KeytabSecret
Principal—Kerberos principal (kerberos)
Krb5ConfigMap—ConfigMap with a krb5.conf, mounted at /etc/krb5.conf
KnownHostsConfigMap—ConfigMap with a known_hosts file
ReplicateCredentialstrueCopy the credential into each shadow's namespace
NodeWaitTimeout2hHow long the shadow waits for the compute node
ExtraOptions[]Extra ssh options, each passed as -o <option>
Reverse.EnabledfalseLet pods ask for a tunnel back into the cluster
Reverse.NodeExecsshHow the shadow runs a command on the compute node from the login node: ssh (ssh <node>) or srun (srun --jobid=<job> --overlap, several times slower, see Performance)
Reverse.SshdPath/usr/sbin/sshdThe sshd binary on the compute node
Reverse.PythonPathpython3The python3 on the compute node, for owner-only forwards

Misconfiguration fails at startup, not on the first offloaded pod, so a missing LoginHost or a kerberos block without a Principal is caught immediately.

Forward modes​

The default, portforward, is one ssh -L per exposed port. It is the cheapest option and depends on nothing but the ssh client, but it requires AllowTcpForwarding yes on the login node.

exec is for sites that do not grant it. It gives each exposed port a local listener in the shadow and relays every accepted connection through a command run on the login node — by default nc <compute node> <port>. Nothing asks sshd to forward anything, so the only privileges involved are the ones the shadow already uses to log in.

Network:
  ShadowMode: "ssh"
  SSH:
    LoginHost: "login.hpc.example.org"
    User: "alice"
    KeySecret: "hpc-ssh-key"
    ForwardMode: "exec"
    ExecConnectCommand: "nc"   # ncat, or "nc -q 1" on a netcat that does not half-close

The two are interchangeable above the socket: the Service still fronts the same container port, and the offloaded workload still only binds 0.0.0.0. Nothing on either side can tell them apart.

portforwardexec
Login node must allowTCP forwardingrunning a command, and have a netcat
Connectionsone long-lived sshone ssh channel per TCP connection
Cost per connectionnoneone multiplexed channel open
Failure surfacesat connect timeat connect time

The relays share a single SSH connection, opened as a control master before the listeners start. Without it every connection would pay a full handshake — a notebook UI opens dozens — and the login node would see a session per connection. Because they share one TCP connection they also share its head-of-line blocking, which is fine for a UI and less so for bulk transfer.

Two things to check when picking exec:

  • Which netcat. nc varies. netcat-traditional does not propagate a half-close, so a client that closes its write side can leave the connection hanging; nc -q 1 or ncat avoids it. Set ExecConnectCommand accordingly.
  • Process limits. Even multiplexed, the login node runs a netcat per connection. Sites with per-user process caps will notice a busy shadow.

Authentication​

Public key​

ssh-keygen -t ed25519 -f ./hpc_key -N ''
ssh-copy-id -i ./hpc_key.pub alice@login.hpc.example.org

kubectl create secret generic hpc-ssh-key -n interlink \
  --from-file=id_ed25519=./hpc_key
SSH:
  Auth: publickey
  KeySecret: hpc-ssh-key

Kerberos​

For GSSAPI sites (GSSAPIAuthentication yes on the login node):

kubectl create secret generic hpc-keytab -n interlink \
  --from-file=user.keytab=/path/to/user.keytab
SSH:
  Auth: kerberos
  KeytabSecret: hpc-keytab
  Principal: alice@EXAMPLE.ORG
  Krb5ConfigMap: krb5-config   # if the image's /etc/krb5.conf lacks your realm

The shadow runs kinit in an init container and refreshes the ticket from a sidecar for the life of the tunnel, so long-running notebooks survive ticket expiry.

Host key verification​

Without KnownHostsConfigMap the shadow uses StrictHostKeyChecking=accept-new, which trusts whatever key the login node presents on first contact. That is a small MITM window on every new shadow pod. For production, pin the key:

ssh-keyscan login.hpc.example.org > known_hosts
kubectl create configmap hpc-known-hosts -n interlink --from-file=known_hosts
SSH:
  KnownHostsConfigMap: hpc-known-hosts

This switches the shadow to StrictHostKeyChecking=yes.

How the compute node is discovered​

The shadow is created when the pod is, which is long before the batch system has scheduled anything — so at creation time there is no node to forward to yet.

  1. interLink creates a ConfigMap, <shadow>-node, holding an empty compute-node key.
  2. The plugin reports the allocated node as PodStatus.NodeName once the job runs.
  3. interLink patches that value into the ConfigMap. It is mounted, not injected into the pod spec, so kubelet refreshes it in place — restarting the shadow would change its pod IP, which Kubernetes has already been told is the offloaded pod's IP.
  4. The shadow's wait-for-node init container polls the mounted file, then ssh-forward starts the tunnel.

While the job is queued the shadow sits in Init:0/1. That is the expected state, not a failure — it reads as "waiting for the allocation" rather than a crash-looping tunnel. After NodeWaitTimeout the init container fails with an explicit message.

If the job is later requeued onto a different host, the plugin reports the new one, interLink republishes it, and the shadow rebuilds its forwarders in place:

compute node changed from node07.hpc.example.org to node11.hpc.example.org, rebuilding the tunnel

The container is not restarted, so the shadow keeps its pod IP — the one Kubernetes has already been told is the offloaded pod's IP.

warning

This requires a plugin that reports PodStatus.NodeName. A plugin that does not will leave the shadow waiting until the timeout expires.

What a plugin may report​

The reported name ends up as an argument to ssh, so interLink accepts only an RFC 1123 hostname or an IP literal and refuses anything else rather than trying to escape it:

Refusing to publish compute node "node07 -oProxyCommand=..." for ns/pod:
not a hostname or IP address. The shadow passes this to ssh, so it is rejected
rather than escaped.

A name carrying whitespace would otherwise split into an extra ssh option, and one carrying a quote or $( would break out of the relay command in exec mode. Plugins should report what the node calls itself — hostname -f for the Slurm plugin — and nothing else.

Credential replication​

Shadow pods follow the offloaded pod. With interlink.eu/shadow-same-ns: "true" — the usual choice for multi-tenant setups such as Kubeflow notebooks — that means the shadow lands in a per-user namespace, where the SSH credential does not exist.

By default interLink copies the credential Secret, and any Krb5ConfigMap / KnownHostsConfigMap, from its own namespace into the shadow's namespace.

Copies are marked with interlink.eu/replicated-from, and only objects carrying that annotation are overwritten on a later pass. If an object of the same name is already there and holds something else, pod creation fails rather than replacing it:

refusing to overwrite secret alice/hpc-ssh-key: it already exists, holds different
content and was not created by interLink

That matters with shadow-same-ns, where the target is somebody's own namespace and a name collision would otherwise destroy their Secret. An unmarked copy whose content already matches is adopted instead of refused, so upgrading from a version that predates the marker does not break existing deployments.

warning

Replication makes the credential readable by anyone who can read Secrets in those namespaces. If that is not acceptable, set ReplicateCredentials: false and provision the Secret in each namespace yourself, through a secrets operator or a replication controller.

Pod configuration​

Nothing special is required — an offloaded pod that exposes a port gets a shadow:

apiVersion: v1
kind: Pod
metadata:
  name: notebook
  annotations:
    # Keep the shadow in this pod's namespace, so it can mount credentials
    # replicated here rather than into a separate <ns>-shadow namespace.
    interlink.eu/shadow-same-ns: "true"
spec:
  nodeSelector:
    kubernetes.io/hostname: hpc-vk
  tolerations:
  - key: virtual-node.interlink/no-schedule
    operator: Exists
    effect: NoSchedule
  containers:
  - name: jupyter
    image: docker://jupyter/scipy-notebook
    ports:
    - containerPort: 8888
      name: notebook

The shadow's Service carries the same ports, so anything that can reach the Service — an Ingress, an Istio VirtualService, kubectl port-forward — reaches the notebook.

interlink.eu/wstunnel-extra-ports works here too, for ports not declared in the container spec.

note

ssh -L forwards TCP only. UDP ports are skipped, with a warning in the virtual kubelet log.

Pod to cluster: the reverse tunnel​

Everything above carries connections from the cluster to the pod. A workload that needs the other direction, an object store, a message bus or the Kubernetes API that only exist inside the cluster, gets it over the same login node. The site switches it on with Network.SSH.Reverse.Enabled: true; the pod lists what it needs:

apiVersion: v1
kind: Pod
metadata:
  name: trainer
  annotations:
    interlink.eu/reverse-forwards: "minio.user.svc.cluster.local:9000 6443:kubernetes.default.svc.cluster.local:443"

The workload then uses the Service name and port as it would in the cluster, http://minio.user.svc.cluster.local:9000 here. The compute node needs no route to the cluster, no WireGuard and no network namespace, so its own interfaces, InfiniBand included, stay visible to the job.

How it works​

job container ──connect name:port──> 127.x.y.z:port on the compute node (sshd, -R)
                                           │ carried back over the shadow's ssh session
                                           ▼
                     shadow pod (ssh-reverse) ──dials──> Service inside the cluster
  1. The shadow runs a second container, ssh-reverse. Once the compute node is known it starts a user-space sshd -i there, as the job's user, through a command run on the login node (ssh <node> or srun --overlap, see NodeExec). The command's stdin and stdout are that sshd's transport: it listens on no port. Its host key and the one client key it accepts are generated for each start and never leave the shadow.
  2. The shadow connects to it with ssh -R. Each forward is bound on the compute node, on a loopback address derived from the pod UID (127.x.y.z), and every connection made there is dialled from the shadow pod, inside the cluster.
  3. The virtual kubelet prepends a snippet to the pod's slurm-job.vk.io/pre-exec. It maps the forwarded names to that address in the container's /etc/hosts (through SINGULARITY_BIND and APPTAINER_BIND) and holds the workload until the tunnel answers, for at most ten minutes. The workload itself is unchanged.

The tunnel is rebuilt if the job moves to another node, like the forwards.

What the site has to allow​

  • A command on the compute node, run from the login node as the job's user. Either ssh <node> into a node where that user has a running job, which sites with pam_slurm_adopt allow, or srun --jobid=<job> --overlap, which needs nothing beyond Slurm. NodeExec picks one; a pod can override it.
  • sshd on the compute node (Reverse.SshdPath) and, for owner-only forwards, python3. Both are stock on most distributions and neither needs root.
  • Nothing in the login node's sshd_config: the forwards are bound by the sshd the shadow starts, not by the site's.

Annotations​

AnnotationValue
interlink.eu/reverse-forwardsSpace-separated endpoints, each host:port or listen-port:host:port
interlink.eu/reverse-socks-portPort of a SOCKS5 proxy for endpoints not listed; off when unset
interlink.eu/reverse-bindThe 127.x.y.z address to use instead of the derived one
interlink.eu/reverse-node-execssh or srun, overriding Reverse.NodeExec
interlink.eu/reverse-owner-onlytrue (default) or false, see below

A Service port below 1024 cannot be bound by an unprivileged sshd on the compute node: give it a listen port, 6443:kubernetes.default.svc.cluster.local:443, and the job connects to port 6443. The name resolves to the right address either way. An annotation that does not parse is refused when the pod is created, with the reason in the virtual kubelet log and the pod's events.

Endpoints not known in advance​

interlink.eu/reverse-socks-port: "1080" adds a SOCKS5 proxy on the same address. A program told to use it, ALL_PROXY=socks5h://127.x.y.z:1080 for curl and Python requests, reaches any cluster address by name, resolved inside the cluster at connect time. It is off unless asked for, and it is a plain loopback listener that every user of that compute node can also use, so declare what you can and keep SOCKS for the rest.

Who else can use the forwards​

A port on a loopback address is reachable by every user of the compute node. By default each forward therefore ends in a unix socket that only the job's user can open, with a small relay in front, run by the sshd session on the node, that accepts a connection only when the connecting socket belongs to that user (read from /proc/net/tcp). interlink.eu/reverse-owner-only: "false" binds the ports directly instead: no relay, no python3 on the node, and no protection from other users.

Performance​

All of a pod's forwards share one ssh session, so the ceiling per pod is what one SSH channel stream does on that link, and the path has three legs: the job to the listener on the node, the carrier between the node and the login node, and ssh from the login node to the cluster.

  • The carrier is the biggest factor. With NodeExec: ssh the bytes go through the node's own sshd; with srun they go through Slurm's I/O forwarding, which was 4 to 5 times slower in our runs (CERN: about 200 MB/s against 40 to 80; SURF: 95 against 15 to 50). On a slow WAN both hit the link first (MareNostrum 5 from CERN: about 50 MB/s either way).
  • Owner-only mode costs nothing measurable: with and without the relay the same runs gave the same figures at the three sites. It is one small process per pod on the node.
  • Small requests pay the round trip of the whole path, not the throughput. The relay sets TCP_NODELAY; without it, request and reply patterns such as a filesystem over the tunnel lost 40 ms per operation to Nagle's algorithm.
  • Figures above are single runs of a 200 MiB object PUT and GET from the job, with several test jobs on the same node at once.

Limits​

  • TCP only.
  • The listeners exist on the node that runs the batch script. Ranks of a multi-node job on other nodes have no tunnel.
  • The tunnel comes up once the shadow has learned the compute node, a status poll or two after the job starts; the pre-exec waits for it.
  • The pre-exec assumes the Slurm plugin with Singularity or Apptainer, as the mesh pre-exec does.
  • /tmp/il-rev-<shadow> on the compute node holds the keys and sockets while the tunnel is up; the session removes it when the tunnel ends.

Troubleshooting​

Shadow stuck in Init:0/1. Normal while the job is queued. If it persists, check that the job is actually running and that the plugin reports the node:

kubectl get configmap <shadow>-node -o jsonpath='{.data.compute-node}'

Empty means no plugin has reported one yet.

ssh-forward crash-loops with a permission error. The credential Secret is missing from the shadow's namespace, or KeySecretKey does not match the key inside it. With ReplicateCredentials: false the Secret must already exist there.

Connection refused through the tunnel. The tunnel reached the compute node but nothing is listening on that port. Services often bind 127.0.0.1 by default; bind 0.0.0.0 instead — for a notebook, --ip=0.0.0.0.

Connection reset, and channel N: open failed: administratively prohibited in the ssh-forward log. The login node refuses port forwarding for this account. Either have the site allow it, or switch to ForwardMode: exec.

Host key verification failed. KnownHostsConfigMap does not contain the login node's current key. Re-run ssh-keyscan and update the ConfigMap.

Refusing to publish compute node ... in the virtual kubelet log, and the shadow stays in Init:0/1. The plugin reported something that is not a hostname or an IP address, so it was not passed to ssh. See what a plugin may report; the plugin needs fixing, and the value it produced is quoted in the message.

Pod creation fails with refusing to overwrite secret .... The shadow's namespace already contains a different object under the credential's name. Rename the credential, or set ReplicateCredentials: false and provision it yourself.

interLink: the reverse tunnel did not come up within 600s in the job log. The shadow's ssh-reverse container never got through; read its log. could not prepare ... on <node> means the command to the compute node was refused, so try NodeExec: srun; no running job named ... on <node> in srun mode means squeue on the login node did not list a running job under the pod's UID.

Pod creation fails with interlink.eu/reverse-forwards: .... The annotation did not parse; the message names the entry and what is wrong with it.

An nginx wstunnel Deployment appears instead of the SSH shadow. A custom template file is overriding the built-in one. The virtual kubelet prefers Network.WstunnelTemplatePath over its own templates; the Helm chart stops mounting it in ssh mode, but a hand-written config may still point at one.

References​