NVIDIA GPU Sharing on Kubernetes

TL;DR Recommended Path for Your Environment

# 1. Generate a worker ignition config from the cluster
oc extract -n openshift-machine-api \
  secret/worker-user-data --to=- > worker.ign

# 2. Serve worker.ign over HTTP from a reachable host, then boot the bare-metal
#    node with the RHCOS ISO. At the boot prompt:
#    coreos.inst.install_dev=/dev/sda
#    coreos.inst.ignition_url=http://<bastion>:8080/worker.ign

# 3. Approve the CSRs that appear once the node contacts the API server
oc get csr -o name | xargs oc adm certificate approve

# 4. Confirm node joined and taint it
oc get nodes
oc adm taint node <gpu-node-name> nvidia.com/gpu=present:NoSchedule

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gpu-operator
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.ngc.nvidia.com/nvidia
    chart: gpu-operator
    targetRevision: v24.9.2   # pin to a specific release
    helm:
      releaseName: gpu-operator
      values: |
        mig:
          strategy: mixed
        nfd:
          enabled: false   # disable if NFD already installed cluster-wide
        operator:
          defaultRuntime: crio  # OCP uses CRI-O
  destination:
    server: https://kubernetes.default.svc
    namespace: gpu-operator
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

oc adm policy add-scc-to-user nvidia-scc \
  -z default -n <your-workload-namespace>

# or via ClusterRoleBinding for all service accounts in a namespace:
oc adm policy add-scc-to-group nvidia-scc \
  system:serviceaccounts:<your-workload-namespace>

apiVersion: v1
kind: ConfigMap
metadata:
  name: default-mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      # profile name referenced by node label below
      audio-and-llm:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 2   # Whisper / live audio
            "2g.20gb": 1   # 7B-13B LLM
            "3g.40gb": 1   # 30B-40B LLM
---
# Label the GPU node to activate the profile:
# kubectl label node <gpu-node> nvidia.com/mig.config=audio-and-llm

# Pod targeting 1g.10gb slice (Whisper / audio)
resources:
  limits:
    nvidia.com/mig-1g.10gb: "1"
---
# Pod targeting 2g.20gb slice (7B-13B LLM)
resources:
  limits:
    nvidia.com/mig-2g.20gb: "1"
---
# Pod targeting 3g.40gb slice (30B-40B LLM)
resources:
  limits:
    nvidia.com/mig-3g.40gb: "1"
---
# Triton env vars to pin to its assigned MIG slice
env:
  - name: NVIDIA_VISIBLE_DEVICES
    valueFrom:
      fieldRef: { fieldPath: status.containerStatuses[0].allocatedResources['nvidia.com/mig-1g.10gb'] }
  - name: CUDA_VISIBLE_DEVICES
    value: "MIG-GPU-..."  # injected automatically by device plugin

# Pod 1: 1g.10gb slice — audio transcoding
model-repo-audio/
  whisper-large-v3/
    config.pbtxt          # backend: "openai", model_type: whisper
    1/
      model.py            # faster-whisper Python backend

# Pod 2: 2g.20gb slice — small LLM (7B-13B)
model-repo-llm-small/
  llama3-8b/
    config.pbtxt          # backend: "vllm"
    1/
      model.json          # {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "gpu_memory_utilization": 0.85}

# Pod 3: 3g.40gb slice — large LLM (30B-40B)
model-repo-llm-large/
  mistral-22b/
    config.pbtxt          # backend: "vllm"
    1/
      model.json          # {"model": "mistralai/Mistral-Small-3.1-22B-Instruct-2503", "gpu_memory_utilization": 0.90}

datadog:
  apiKey: <DD_API_KEY>
  clusterName: ocp-dc1
  tags:
    - "env:production"
    - "dc:dc1"
  nvidia:
    enabled: true       # enables NVML/DCGM GPU metrics
  kubelet:
    tlsVerify: false    # required for OCP kubelet cert
agents:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists  # ensure agent lands on GPU node

Stack architecture

00 Hardware Sharing Mechanisms

Fig Approach Diagrams

Four ways to share 8 GPUs. Each diagram shows where isolation lives, what controls scheduling, and what a workload actually sees.

01 Time-Slicing via NVIDIA Device Plugin

Simplest option. Device plugin advertises N virtual GPUs per physical GPU. Pods request nvidia.com/gpu: 1 normally.

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        replicas: 4   # 4 pods share 1 physical GPU

02 MIG Partitioning (Ampere+)

GPU Operator's MIG Manager reconfigures profiles via node label. Handles drain/reconfigure/rejoin automatically — no manual nvidia-smi mig calls.

# Apply to node to trigger reconfiguration
nvidia.com/mig.config: "all-3g.40gb"   # 2x MIG instances on A100-80GB

resources:
  limits:
    nvidia.com/mig-3g.40gb: 1

Common MIG Profiles — A100-80GB

03 MPS — Multi-Process Service

Multiple CUDA contexts run simultaneously (not time-sliced). Lower latency, no ctx-switch overhead. Better throughput for concurrent small kernels; same lack of memory isolation as time-slicing.

sharing:
  mps:
    replicas: 4

04 Scheduling Optimizations

Bin-Packing

Default kube-scheduler spreads workloads across nodes. Override with MostAllocated to pack GPUs full before using next node.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 10
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

GPU Feature Discovery Node Selectors

nodeSelector:
  nvidia.com/gpu.product: "A100-SXM4-80GB"
  nvidia.com/gpu.memory: "81920"
  nvidia.com/mig.capable: "true"

spec:
  devices:
    requests:
    - name: gpu
      selectors:
      - cel:
          expression: |
            device.attributes["gpu.nvidia.com"].productName.startsWith("A100") &&
            device.capacity["gpu.nvidia.com"].memory.isGreaterThan(quantity("40Gi"))

DRA vs. HAMi — when to use which

Core split: DRA answers which GPU slot does this pod get — hardware binding, no sub-GPU VRAM enforcement beyond MIG. HAMi answers how much of a GPU slot can this pod consume — soft quotas, multiple pods per physical device. They solve adjacent problems and can run together (HAMi has experimental DRA integration). Practical today: HAMi if K8s < 1.34, multi-vendor, or VRAM quotas needed. DRA if K8s ≥ 1.34 and clean K8s-native semantics matter more than sub-GPU splitting.

Priority Classes + Preemption

High-priority inference pods preempt low-priority training jobs when GPU capacity is needed.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-high
value: 1000
preemptionPolicy: PreemptLowerPriority
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-train-low
value: 100
preemptionPolicy: Never

05 Dynamic MIG Reconfiguration

Shift MIG profiles by time of day to match workload patterns. GPU Operator handles node drain/reconfigure/rejoin — no manual intervention.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mig-reconfigure-night
spec:
  schedule: "0 20 * * *"   # 8pm
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: reconfigure
            image: bitnami/kubectl
            command:
            - kubectl
            - label
            - node
            - <gpu-node>
            - nvidia.com/mig.config=all-1g.10gb

Hyb Hybrid Approaches

The mechanisms above compose. The most useful production hybrid is vLLM + MIG: partition each GPU into hardware-isolated slices, then run an independent vLLM instance per slice. You get K8s scheduling flexibility, MIG's hard resource guarantees, and vLLM's continuous-batching throughput — simultaneously, with no tradeoff between them.

vLLM + MIG — per-model isolation at scale

Pick a MIG profile that fits your model's VRAM footprint, schedule one vLLM pod per MIG slice. Each pod sees an isolated GPU device with dedicated SMs and VRAM — noisy-neighbor exhaustion is impossible.

Profile sizing guide (A100-80GB)

MIG + MPS within a slice

Once a MIG slice is assigned to a pod, that pod can itself start an MPS daemon to multiplex multiple CUDA processes within the slice — running concurrent kernels without serialization, without surrendering the slice's hard VRAM floor. This stacks two levels of isolation: hardware boundary between slices, MPS concurrency within a slice. Useful when a slice is large enough to run several small inference replicas simultaneously and context-switch latency matters.

ISO VRAM Isolation Landscape

Not all GPU sharing approaches provide VRAM isolation. This matters operationally: without it, a greedy or buggy workload can exhaust all GPU memory and take down every co-tenant on that device.

Non-MIG options in detail

NVIDIA MPS pinned memory limits (Volta+)

Set per-client VRAM caps inside the MPS daemon. Already wired into the k8s-device-plugin MPS sharing mode (splits evenly across replicas) and the NVIDIA DRA driver (MpsConfig.DefaultPinnedDeviceMemoryLimit). GKE supports it natively.

# per-client cap set in MPS daemon
echo "set_default_device_pinned_mem_limit 0 4096m" | nvidia-cuda-mps-control

# or via environment variable before launching client
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT="0=4096m" python inference.py

AMD MI300X — NPS4 memory partitioning

The MI300X and MI325X support Non-Uniform Memory Access Partitioning (NPS4), which physically splits the 192 GB HBM stack into 4 × 48 GB regions. Combined with CPX (Compute Partitioning), up to 64 hardware partitions per 8-GPU node are possible. The AMD GPU Operator's Device Config Manager applies profiles declaratively; supported on OpenShift. This is the AMD hardware equivalent of MIG, limited to MI300-class GPUs.

dmem cgroup controller (Linux ≥ 6.14)

A kernel-level VRAM cgroup with min/low/max semantics identical to the memory cgroup. Merged for Intel xe in Linux 6.14; AMD amdgpu support targeting 6.15/6.16; nouveau patch on mailing list. Does not work with the NVIDIA proprietary driver. No runc, containerd, or K8s wiring exists yet — cgroup files must be set manually via an NRI plugin or privileged DaemonSet. Known gaps: hipMallocAsync pools spill uncapped into system RAM; lowering dmem.max does not reclaim already-allocated memory. HAMi is evaluating dmem as its AMD backend.

Inf Inference Server Layer — Triton vs. vLLM

Triton Inference Server and vLLM sit at the application layer, above K8s GPU sharing. They don't replace MIG, device plugin, or DRA — they run inside whatever GPU resource(s) K8s assigns to their pod.

What Triton adds at the application layer

• instance_group — load N copies of a model spread across specific GPU indices the pod owns. K8s gave pod GPUs 0–3; Triton puts 2 instances on GPU 0, 2 on GPU 1.
• Concurrent model execution — multiple different models share one GPU simultaneously through Triton's own scheduler. No VRAM isolation between them (same noisy-neighbor exposure as MPS — application-level, not hardware).
• Dynamic batching — coalesces requests before GPU invocation. Reduces effective GPU calls. Complements K8s-level sharing.
• Rate limiter — caps simultaneous requests per model, prevents one model starving others on the same GPU.
• Multi-backend + ensembles — TensorRT, PyTorch, ONNX, TF, OpenVINO, Python, and a vLLM backend. One Triton instance can serve a CNN, reranker, and LLM together in a single ensemble pipeline — something three separate vLLM instances cannot do.

Triton vs. vLLM

Triton + MIG — same pattern as vLLM + MIG

One Triton pod per MIG slice, each slice running different models, isolated at hardware level. For mixed-model workloads this beats running separate vLLM instances:

Each slice is hardware-isolated — the embedding model cannot OOM the LLM slice. Triton's ensemble API then wires the pipeline together over gRPC without leaving the node. A single vLLM instance cannot express this topology.

— Agent Orchestration: Turnstone

Turnstone is a self-hosted, local-first agent orchestration layer (Apache 2.0, Python 3.13+). It sits above your inference stack — Triton and vLLM do the model serving; Turnstone drives the agent loop, routes tool calls, and manages multi-turn workstreams across your cluster.

Where it fits in your stack

Key fit points

• OpenAI-compatible target — Turnstone calls any /v1/chat/completions endpoint; your Triton vLLM backend exposes exactly that. Point Turnstone at the pod's ClusterIP service.
• Local-first / no telemetry — aligns with on-prem DC requirements. No phone-home, no SaaS dependency.
• Cluster-native — tasks fan out across nodes; your OCP cluster with multiple MIG slices maps directly to its multi-machine model.
• MCP tool servers — any MCP server (kubectl, custom APIs, S3, databases) plugs in as agent tools, same protocol Claude Code uses.
• LLM risk judge — evaluates each tool call before execution. Useful when agents have kubectl or shell access in a production cluster.

Routing agents to MIG slices

Configure separate Turnstone endpoints per slice type so heavy reasoning steps land on the larger slice and lighter tool calls use the smaller one:

models:
  - name: llm-large          # reasoning, planning, summarization
    api_base: http://triton-llm-large.gpu-operator.svc.cluster.local:8000/v1
    model: mistral-22b
  - name: llm-small          # tool-call generation, classification
    api_base: http://triton-llm-small.gpu-operator.svc.cluster.local:8000/v1
    model: llama3-8b
  - name: audio-asr          # audio → text (Whisper via Triton HTTP)
    api_base: http://triton-audio.gpu-operator.svc.cluster.local:8000/v2
    model: whisper-large-v3

— Alternative: llm-d + Dual A100

llm-d is a Kubernetes-native LLM inference routing layer, co-developed by Red Hat and Google, built on the Gateway API Inference Extension (KEP-4804, targeting GA in K8s 1.33+). It adds KV-cache-aware load balancing and InferencePool semantics on top of plain vLLM pods — no Triton, no Turnstone, no custom routing code.

As a major llm-d contributor, Red Hat ships it as a supported component on OCP. If you are already on OCP and want a Red Hat-supported LLM routing layer, this is the path to evaluate alongside the Triton + Turnstone approach.

Reference architecture — dual A100-80GB

Audio bypasses llm-d entirely — it routes gRPC directly to the Whisper OCP Service. LLM traffic hits the llm-d Gateway, which routes by model name to the appropriate InferencePool.

llm-d request path

Key components

• InferencePool — a K8s CRD grouping replicas that serve the same model. Replaces a plain Service for LLM traffic; exposes replica-level metadata (KV-cache fill, queue depth) to the endpoint picker.
• InferenceModel — maps a model name (e.g. mistral-22b) to an InferencePool, with criticality tiers (critical / sheddable). The Gateway reads this to route without client-side logic.
• Endpoint Picker (ext-proc) — an Envoy external processor that receives each request, queries live replica stats, and returns the best backend. Default picker: lowest KV-cache utilization. Pluggable via the picker API.
• Gateway — standard Envoy Proxy (OCP's Gateway API implementation), extended with the ext-proc filter. No custom proxy fork needed.

llm-d vs. Turnstone — what each layer solves

These are not mutually exclusive. llm-d handles replica selection beneath the LLM endpoint; Turnstone handles orchestration above it. You could run Turnstone pointing at the llm-d Gateway and get both KV-cache routing and agent orchestration.

llm-d quick-start on OCP

helm repo add llm-d https://llm-d.ai/helm-charts
helm repo update

helm install llm-d-operator llm-d/llm-d-operator \
  --namespace llm-d-system \
  --create-namespace

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llm-small
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama3-8b
  extensionRef:
    name: llm-d-epp   # endpoint picker deployment
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama3-8b
spec:
  modelName: llama3-8b
  criticality: Critical
  poolRef:
    name: llm-small

— Live Simulation: What the Diagram Shows

The interactive simulation animates the full request path through this architecture — 10 concurrent clients, day/night MIG reconfiguration, QoS preemption, and Datadog observability. This section explains what each animated element represents and where the simulation simplifies reality.

What each element represents

Caveats — what the simulation omits or simplifies

• Batch is shown as one packet, but is two model calls. A real batch pipeline routes Whisper (audio → text) and then the LLM (text → summary) as two sequential requests. The diagram shows one packet to keep the animation readable.
• Turnstone dwell time not shown. Packets transit Turnstone nearly instantaneously in the animation. In reality the orchestration layer adds latency for context assembly, tool call round-trips, and inter-model hops — commonly 200 ms–2 s depending on tool count and MCP server latency.
• Concurrency model is simplified. The simulation caps each slice at max=1 or max=2 for visual clarity. Real Triton/vLLM instances support dynamic batching; actual saturation depends on request size, sequence length, and KV-cache pressure — not just count.
• Preemption is instantaneous in the simulation. Real K8s preemption involves a graceful termination period (default 30 s), pod eviction, and rescheduling. Affected clients must handle 5xx errors and retry.
• MIG reconfiguration window is 2.5 s in the sim. Production reconfiguration takes 2–5 minutes: pod drain, slice destroy/create, device plugin restart, new pod scheduling. Plan maintenance windows accordingly.
• No network latency, no KV-cache state. The animation treats all hops as instant. Real token-streaming latency, KV-cache warm-up on first request, and network RTT between services are not represented.
• Turnstone is optional. If workloads are simple single-turn completions with no tool use, clients hit OCP Service directly. The Audio (P1) and LLM-sm (P2) clients in the simulation already show this bypass pattern.

06 Recommended Stack

K8s Value-Add vs Bare-Metal / Hypervisor vGPU

07 VMs: Boon or Bust?

Mostly bust. The VM layer adds overhead and cost that K8s + MIG already solves for most workloads.

Middle Ground: KubeVirt

Run VMs as K8s workloads with GPU passthrough. Gets K8s-native lifecycle + VM isolation + MIG hardware partitioning. Loses pod startup speed; adds KubeVirt operator complexity.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  template:
    spec:
      domain:
        devices:
          gpus:
          - deviceName: nvidia.com/mig-3g.40gb
            name: gpu0

Decision Matrix

08 GPU Operator Install Reference

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set mig.strategy=mixed \
  --set devicePlugin.config.name=time-slicing-config

09 Next Steps

Ranked by usefulness. Each builds directly on what's here.

Ref References

Primary source

• Sharing NVIDIA GPUs at the System Level: Time-Sliced and MIG-Backed vGPUs — Colfax Research. Basis of this guide.

NVIDIA official documentation

• NVIDIA GPU Operator — GPU Operator overview, installation, MIG manager, MPS daemon configuration.
• NVIDIA MIG User Guide — MIG architecture, profiles, partitioning, and instance management on A100/H100/A30.
• NVIDIA Multi-Process Service (MPS) — MPS architecture, client/server model, use cases.
• MPS environment variables — CUDA_MPS_PINNED_DEVICE_MEM_LIMIT and related per-client VRAM cap configuration.
• NVIDIA Triton Inference Server — model repository, instance groups, dynamic batching, ensemble pipelines, backend support.
• GPU Operator + KubeVirt — vGPU manager integration; VMs on K8s; DRA-mode mixed container/VM nodes.

Kubernetes GPU projects

• HAMi — Heterogeneous AI Computing Virtualization Middleware (CNCF Incubating) — CUDA shim, per-pod VRAM limits, GPU-type filtering, multi-vendor support.
• NVIDIA DRA Driver API reference — ResourceClaim structured attributes; MpsConfig.DefaultPinnedDeviceMemoryLimit.
• k8s-device-plugin issue #764 — MPS pinned memory limits: cudaMemGetInfo still reports full card; tracking issue.
• Intel GPU Plugin for Kubernetes — shared-dev-num, SR-IOV VF support, Intel dGPU scheduling.
• HAMi AMD core — dmem cgroup evaluation — tracking issue for using the Linux dmem cgroup as the AMD VRAM enforcement backend.

AMD

• AMD ROCm — Compute and Memory Partition Modes — SPX/CPX compute partitioning; NPS1/NPS4 memory partitioning on MI300X/MI325X.
• AMD GPU Operator — Applying Partition Profiles — Device Config Manager; NPS4 setup on OpenShift.

Kernel / hardware

• dmem cgroup controller — VRAM control in Linux — Phoronix coverage of the kernel-level GPU memory cgroup merged in Linux 6.14 for Intel xe.
• Linux 6.15 DRM fixes — amdgpu dmem cgroup support timeline.