Run an Inference Server on Offloaded GPUs

This guide runs vLLM on an HPC GPU as a Slurm job, and makes it reachable from Kubernetes as a normal ClusterIP Service.

Tested with interLink 0.6.3-pre1, RKE2 1.36, vLLM 0.29.0, one NVIDIA H100 80GB.

Prerequisites

• A ready virtual node with the Slurm plugin. If interLink is not installed yet, start with Deploy your plugin.

• Mesh networking enabled (EnableTunnel and FullMesh). Without it the Pod has no address the cluster can reach. See Mesh Network Configuration.

• Apptainer on the compute nodes, and a vLLM .sif image on a shared filesystem.

• Permission to hold a GPU for the whole walltime.

• A privileged <namespace>-wstunnel namespace, if the cluster sets a baseline or restricted Pod Security default. The mesh pods are privileged and are rejected otherwise, which leaves the Pod Pending with no clear error. interLink creates the namespace itself, but unlabelled, so create it first:

  apiVersion: v1
  kind: Namespace
  metadata:
    name: default-wstunnel
    labels:
      pod-security.kubernetes.io/enforce: privileged
      pod-security.kubernetes.io/audit: privileged
      pod-security.kubernetes.io/warn: privileged

  RKE2 is unrestricted by default and needs nothing here. Clusters using the CIS profile, or a custom admission configuration, do.

Step 1: Download the Model Weights

Compute nodes run offline, so the serving container uses HF_HUB_OFFLINE=1 and every weight file has to be in the cache before it starts. Stage the weights with a separate offloaded Pod. No GPU is needed for a download, so this Pod asks for no --gres and no --nv:

apiVersion: v1
kind: Pod
metadata:
  name: hf-download
  namespace: default
  annotations:
    slurm-job.vk.io/flags: "--partition=gpu --time=02:00:00 --mem=8G --cpus-per-task=4"
    slurm-job.vk.io/pre-exec: "module load apptainer;"
    slurm-job.vk.io/singularity-options: "--bind /home/myusername/vllm-work:/vllm-work"
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: my-node
  tolerations:
    - key: virtual-node.interlink/no-schedule
      operator: Exists
  containers:
    - name: download
      image: /home/myusername/vllm-work/vllm.sif
      env:
        - name: HF_HOME
          value: "/vllm-work/hf-cache"
      command: ["bash", "-lc"]
      args:
        - hf download google/gemma-4-31B-it

kubectl apply -f hf-download.yaml
kubectl get pod hf-download -w           # wait for Completed
kubectl logs hf-download | tail -5

The compute nodes need outbound network for this, and the walltime has to be long enough to finish the download.

Step 2: Create the Deployment

A Deployment is used instead of a single Pod. The Slurm job ends at walltime, and the ReplicaSet then submits a new one, so the server comes back on its own.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-gemma
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
      annotations:
        slurm-job.vk.io/flags: "--partition=gpu --gres=gpu:1 --cpus-per-task=24 --mem=200G --time=04:00:00"
        slurm-job.vk.io/pre-exec: "module load apptainer;"
        slurm-job.vk.io/singularity-options: "--nv --bind /home/myusername/vllm-work:/vllm-work"
    spec:
      nodeSelector:
        kubernetes.io/hostname: my-node         # your virtual node
      tolerations:
        - key: virtual-node.interlink/no-schedule
          operator: Exists
      containers:
        - name: gemma
          image: /home/myusername/vllm-work/vllm.sif
          args:
            - "--model=google/gemma-4-31B-it"
            - "--served-model-name=gemma-4-31b"
            - "--host=0.0.0.0"
            - "--port=8000"
            - "--max-model-len=32768"
            - "--quantization=fp8"
            - "--kv-cache-memory-bytes=24000000000"
          env:
            - name: HF_HOME
              value: "/vllm-work/hf-cache"
            - name: HF_HUB_OFFLINE
              value: "1"
            - name: VLLM_CACHE_ROOT
              value: "/tmp/vllm-cache"
          ports:
            - containerPort: 8000
              name: http
          resources:
            requests: { cpu: "4", memory: 16Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: default
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000

All available annotations are listed in the Pod Annotations Reference. These are easy to get wrong:

• image is a file path, not a registry reference. The Slurm plugin passes it to Apptainer.
• pre-exec must end with ;. The plugin joins it with the job script path on one line. Without the ; Lmod reports an unknown module.
• --nv gives the container the GPU. Without it vLLM starts and finds no device.
• Cache paths must not be under $HOME. The container runs in a user namespace, so writes there fail with OSError: [Errno 38].
• --quantization=fp8 keeps the model inside the card. At bf16 the weights alone need about 59 GiB on an 80 GB GPU, which leaves no room for the KV cache. --kv-cache-memory-bytes then sets the cache size explicitly.

Step 3: Apply and Check

kubectl apply -f vllm-gemma.yaml
kubectl get pod -l app=vllm -w
kubectl logs -l app=vllm -f

Step 4: Test the Server

The Service selects the offloaded Pod like any other Pod, so it resolves under its own name in your namespace:

kubectl run curl --image=curlimages/curl --restart=Never --command -- \
  curl -s http://vllm.default.svc.cluster.local:8000/v1/models

kubectl logs curl
kubectl delete pod curl

To serve more than one model behind a single URL, see Serve Several Models Behind One Endpoint.

Limitations

interLink is not designed for long-running services:

• The GPU is held continuously. The Deployment resubmits at every walltime, also when no requests arrive. On a shared cluster this is a policy question.
• Every restart is a new Slurm job, so per-user job limits apply.
• Every restart costs the cold start, about ten minutes before the server answers again.
• Nothing scales to zero. Stopping the server means deleting the Deployment.