Forem: 霓漠Nimbus

A Quick Take on K8s 1.34 GA DRA: 7 Questions You Probably Have

霓漠Nimbus — Thu, 11 Sep 2025 13:39:19 +0000

The 7 questions

What problem does DRA solve?
Does “dynamic” mean hot-plugging a GPU to a running Pod or in-place GPU memory resize?
What real-world use cases (and “fun” possibilities) does DRA enable?
How does DRA relate to the DevicePlugin? Can they coexist?
What’s the status of GPU virtualization under DRA? What about HAMi?
Which alpha/beta features around DRA are worth watching?
When will this be production-ready at scale?

Before we dive in, here’s a mental model that helps a lot:

Know HAMi + know PV/PVC ≈ know DRA.

More precisely: DRA borrows the dynamic provisioning idea from PV/PVC and adds a structured, standardized abstraction for device requests. The core insight is simple:

Previously, the DevicePlugin didn’t surface enough structured information for the scheduler to make good decisions. DRA fixes that by richly describing devices and requests in a way the scheduler (and autoscaler) can reason about.

In plain English: report more facts, and make the scheduler aware of them. That’s DRA’s “structured parameters” in a nutshell.

If you’re familiar with HAMi’s Node & Pod annotation–based mechanism for conveying device constraints to the scheduler, DRA elevates the same idea into first-class, structured API objects that the native scheduler and Cluster Autoscaler can reason about directly.

A bit of history (why structured parameters won)

The earliest DRA design wasn’t structured. Vendors proposed opaque, driver-owned CRDs. The scheduler couldn’t see global availability or interpret those fields, so it had to orchestrate a multi-round “dance” with the vendor controller:

Scheduler writes a candidate node list into a temp object
Driver controller removes unfit nodes
Scheduler picks a node
Driver tries to allocate
Allocation status is written back
Only then does the scheduler try to bind the Pod

Every step risked races, stale state, retries—hot spots on the API server, pressure on drivers, and long-tail scheduling latency. Cluster Autoscaler (CA) also had poor predictive power because the scheduler itself didn’t understand the resource constraints.

That approach was dropped in favor of structured parameters, so scheduler and CA can reason directly and participate in the decision upfront.

Now the Q&A

1) What problem does DRA actually solve?

It solves this: “DevicePlugin’s reported info isn’t enough, and if you report it elsewhere the scheduler can’t see it.”

DRA introduces structured, declarative descriptions of device needs and inventory so the native scheduler can decide intelligently.

2) Does “dynamic” mean hot-plugging GPUs into a running Pod, or in-place VRAM up/down?

Neither. Here, dynamic primarily means flexible, declarative device selection at scheduling time, plus the ability for drivers to prepare/cleanup around bind and unbind. Think of it as flexible resource allocation, not live GPU hot-plugging or in-place VRAM resizing.

3) What new toys does DRA bring? Where does it shine?

DRA adds four key concepts:

DeviceClass → think StorageClass
ResourceClaim → think PVC
ResourceClaimTemplate → think VolumeClaimTemplate (flavor or “SKU” you’d expose on a platform)
ResourceSlice → a richer, extensible inventory record, i.e., a supercharged version of what DevicePlugin used to advertise

This makes inventory and SKU management feel native. A lot of the real “fun” lands with features that are α/β today (see below), but even at GA the information model is the big unlock.

4) What’s the relationship with DevicePlugin? Can they coexist?

DRA is meant to replace the legacy DevicePlugin path over time. To make migration smoother, there’s KEP-5004 (DRA Extended Resource Mapping) which lets a DRA driver map devices to extended resources (e.g., nvidia.com/gpu) during a transition.

Practically:

You can run both in the same cluster during migration.
A single node cannot expose the same named extended resource from both.
You can migrate apps and nodes gradually.

5) What about GPU virtualization? And HAMi?

Template-style (MIG-like) partitioning: see KEP-4815 – DRA Partitionable Devices.
Flexible (capacity-style) sharing like HAMi: the community is building on KEP-5075 – DRA Consumable Capacity (think “share by capacity” such as VRAM or bandwidth).

HAMi’s DRA driver (demo branch) lives here:

https://github.com/Project-HAMi/k8s-dra-driver/tree/demo

6) What α/β features look exciting?

Already mentioned, but here’s the short list:

KEP-5004 – DRA Extended Resource Mapping: smoother migration from DevicePlugin
KEP-4815 – Partitionable Devices: MIG-like templated splits
KEP-5075 – Consumable Capacity: share by capacity (VRAM, bandwidth, etc.)

And more I’m watching:

KEP-4816 – Prioritized Alternatives in Device Requests

Let a request specify ordered fallbacks—prefer “A”, accept “B”, or even prioritize allocating “lower-end” first to keep “higher-end” free.
KEP-4680 – Resource Health in Pod Status

Device health surfaces directly in PodStatus for faster detection and response.
KEP-5055 – Device Taints/Tolerations

Taint devices (by driver or humans) e.g., “nearing decommission” or “needs maintenance”, and control placement with tolerations.

7) When will this be broadly production-ready?

For wide, low-friction production use, you typically want β maturity + ecosystem drivers to catch up. A rough expectation: ~ 8–16 months for most shops, depending on vendors and your risk posture.

Virtualizing Any GPU on AWS with HAMi: Free Memory Isolation

霓漠Nimbus — Thu, 11 Sep 2025 13:08:27 +0000

TL;DR: This guide spins up an AWS EKS cluster with two GPU node groups (T4 and A10G), installs HAMi automatically, and deploys three vLLM services that share a single physical GPU per node using free memory isolation. You’ll see GPU‑dimension binpack in action: multiple Pods co‑located on the same GPU when limits allow.

Why HAMi on AWS?

HAMi brings GPU‑model‑agnostic virtualization to Kubernetes—spanning consumer‑grade to data‑center GPUs. On AWS, that means you can take common NVIDIA instances (e.g., g4dn.12xlarge with T4s, g5.12xlarge with A10Gs), and then slice GPU memory to safely pack multiple Pods on a single card—no app changes required.

In this demo:

Two nodes: one T4 node, one A10G node (each with 4 GPUs).
HAMi is installed via Helm as part of the Terraform apply.
vLLM workloads request fractions of GPU memory so two Pods can run on one GPU.

One‑Click Setup

Repo: github.com/dynamia-ai/hami-ecosystem-demo

0) Prereqs

Terraform or OpenTofu
AWS CLI v2 (and aws sts get-caller-identity succeeds)
kubectl, jq

1) Provision AWS + Install HAMi

git clone https://github.com/dynamia-ai/hami-ecosystem-demo.git
cd infra/aws
terraform init
terraform apply -auto-approve

When finished, configure kubectl using the output:

terraform output -raw kubectl_config_command
# Example:
# aws eks update-kubeconfig --region us-west-2 --name hami-demo-aws

2) Verify Cluster & HAMi

Check that HAMi components are running:

kubectl get pods -n kube-system | grep -i hami

hami-device-plugin-mtkmg             2/2     Running   0          3h6m
hami-device-plugin-sg5wl             2/2     Running   0          3h6m
hami-scheduler-574cb577b9-p4xd9      2/2     Running   0          3h6m

List registered GPUs per node (HAMi annotates nodes with inventory):

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.hami\.io/node-nvidia-register}{"\n"}{end}'

You should see four entries per node (T4 x4, A10G x4), with UUIDs and memory:

ip-10-0-38-240.us-west-2.compute.internal   GPU-f8e75627-86ed-f202-cf2b-6363fb18d516,10,15360,100,NVIDIA-Tesla T4,0,true,0,hami-core:GPU-7f2003cf-a542-71cf-121f-0e489699bbcf,10,15360,100,NVIDIA-Tesla T4,0,true,1,hami-core:GPU-90e2e938-7ac3-3b5e-e9d2-94b0bd279cf2,10,15360,100,NVIDIA-Tesla T4,0,true,2,hami-core:GPU-2facdfa8-853c-e117-ed59-f0f55a4d536f,10,15360,100,NVIDIA-Tesla T4,0,true,3,hami-core:
ip-10-0-53-156.us-west-2.compute.internal   GPU-bd5e2639-a535-7cba-f018-d41309048f4e,10,23028,100,NVIDIA-NVIDIA A10G,0,true,0,hami-core:GPU-06f444bc-af98-189a-09b1-d283556db9ef,10,23028,100,NVIDIA-NVIDIA A10G,0,true,1,hami-core:GPU-6385a85d-0ce2-34ea-040d-23c94299db3c,10,23028,100,NVIDIA-NVIDIA A10G,0,true,2,hami-core:GPU-d4acf062-3ba9-8454-2660-aae402f7a679,10,23028,100,NVIDIA-NVIDIA A10G,0,true,3,hami-core:

Deploy the Demo Workloads

Apply the manifests (two A10G services, one T4 service):

kubectl apply -f demo/workloads/a10g.yaml
kubectl apply -f demo/workloads/t4.yaml
kubectl get pods -o wide

NAME                                       READY   STATUS    RESTARTS   AGE    IP            NODE                                        NOMINATED NODE   READINESS GATES
vllm-a10g-mistral7b-awq-5f78b4c6b4-q84k7   1/1     Running   0          172m   10.0.50.145   ip-10-0-53-156.us-west-2.compute.internal   <none>           <none>
vllm-a10g-qwen25-7b-awq-6d5b5d94b-nxrbj    1/1     Running   0          172m   10.0.49.180   ip-10-0-53-156.us-west-2.compute.internal   <none>           <none>
vllm-t4-qwen25-1-5b-55f98dbcf4-mgw8d       1/1     Running   0          117m   10.0.44.2     ip-10-0-38-240.us-west-2.compute.internal   <none>           <none>
vllm-t4-qwen25-1-5b-55f98dbcf4-rn5m4       1/1     Running   0          117m   10.0.37.202   ip-10-0-38-240.us-west-2.compute.internal   <none>           <none>

What the two key annotations do

In the Pod templates you’ll see:

metadata:
  annotations:
    nvidia.com/use-gputype: "A10G"   # or "T4" on the T4 demo
    hami.io/gpu-scheduler-policy: "binpack"

nvidia.com/use-gputype restricts scheduling to the named GPU model (e.g., A10G, T4).
hami.io/gpu-scheduler-policy: binpack tells HAMi to co‑locate Pods on the same physical GPU when memory/core limits permit (GPU‑dimension binpack).

How the free memory isolation is requested

Each container sets GPU memory limits via HAMi resource names so multiple Pods can safely share one card:

On T4: nvidia.com/gpumem: "7500" (MiB) with 2 replicas ⇒ both fit on a 16 GB T4.
On A10G: nvidia.com/gpumem-percentage: "45" for each Deployment ⇒ two Pods fit on a 24 GB A10G.

HAMi enforces these limits inside the container, so Pods can’t exceed their assigned GPU memory.

Expected Results: GPU Binpack

T4 deployment (vllm-t4-qwen25-1-5b with replicas: 2): both replicas are scheduled to the same T4 GPU on the T4 node.
A10G deployments (vllm-a10g-mistral7b-awq and vllm-a10g-qwen25-7b-awq): both land on the same A10G GPU on the A10G node (45% + 45% < 100%).

How to verify co‑location & memory caps

In‑pod verification (nvidia-smi)

# A10G pair
for p in $(kubectl get pods -l app=vllm-a10g-mistral7b-awq -o name; \
           kubectl get pods -l app=vllm-a10g-qwen25-7b-awq -o name); do
    echo "== $p =="
    # Show the GPU UUID (co‑location check)
    kubectl exec ${p#pod/} -- nvidia-smi --query-gpu=uuid --format=csv,noheader
    # Show memory cap (total) and current usage inside the container view
    kubectl exec ${p#pod/} -- nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv,noheader
    echo
done

Expected

The two A10G Pods print the same GPU UUID → confirms co‑location on the same physical A10G.
memory.total inside each container ≈ 45% of A10G VRAM (slightly less due to driver/overhead; e.g., ~10,3xx MiB), and memory.used stays below that cap.

Example output

== pod/vllm-a10g-mistral7b-awq-5f78b4c6b4-q84k7 ==
GPU-d4acf062-3ba9-8454-2660-aae402f7a679
NVIDIA A10G, 10362 MiB, 7241 MiB

== pod/vllm-a10g-qwen25-7b-awq-6d5b5d94b-nxrbj ==
GPU-d4acf062-3ba9-8454-2660-aae402f7a679
NVIDIA A10G, 10362 MiB, 7355 MiB

# T4 pair (2 replicas of the same Deployment)
for p in $(kubectl get pods -l app=vllm-t4-qwen25-1-5b -o name); do
    echo "== $p =="
        kubectl exec ${p#pod/} -- nvidia-smi --query-gpu=uuid --format=csv,noheader
        kubectl exec ${p#pod/} -- nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv,noheader
    echo
done

Expected

Both replicas print the same T4 GPU UUID → confirms co‑location on the same T4.
memory.total = 7500 MiB (from nvidia.com/gpumem: "7500") and memory.used stays under it.

Example output

== pod/vllm-t4-qwen25-1-5b-55f98dbcf4-mgw8d ==
GPU-f8e75627-86ed-f202-cf2b-6363fb18d516
Tesla T4, 7500 MiB, 5111 MiB

== pod/vllm-t4-qwen25-1-5b-55f98dbcf4-rn5m4 ==
GPU-f8e75627-86ed-f202-cf2b-6363fb18d516
Tesla T4, 7500 MiB, 5045 MiB

Quick Inference Checks

Port‑forward each service locally and send a tiny request.

T4 / Qwen2.5‑1.5B

kubectl port-forward svc/vllm-t4-qwen25-1-5b 8001:8000

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON' | jq -r '.choices[0].message.content'
{
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "temperature": 0.2,
  "messages": [
    {
      "role": "user",
      "content": "Summarize this email in 2 bullets and draft a one-sentence reply:\\n\\nSubject: Renewal quote & SSO\\n\\nHi team, we want a renewal quote, prefer monthly billing, and we need SSO by the end of the month. Can you confirm timeline?\\n\\n— Alex"
    }
  ]
}
JSON

Example output

Summary:
- Request for renewal quote with preference for monthly billing.
- Need Single Sign-On (SSO) by the end of the month.

Reply:
Thank you, Alex. I will ensure that both the renewal quote and SSO request are addressed promptly. We aim to have everything ready before the end of the month.

A10G / Mistral‑7B‑AWQ

kubectl port-forward svc/vllm-a10g-mistral7b-awq 8002:8000

curl -s http://127.0.0.1:8002/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON' | jq -r '.choices[0].message.content'
{
  "model": "solidrust/Mistral-7B-Instruct-v0.3-AWQ",
  "temperature": 0.3,
  "messages": [
    {
      "role": "user",
      "content": "Write a 3-sentence weekly update about improving GPU sharing on EKS with memory capping. Audience: non-technical executives."
    }
  ]
}
JSON

Example output

In our ongoing efforts to optimize cloud resources, we're pleased to announce significant progress in enhancing GPU sharing on Amazon Elastic Kubernetes Service (EKS). By implementing memory capping, we're ensuring that each GPU-enabled pod on EKS is allocated a defined amount of memory, preventing overuse and improving overall system efficiency. This update will lead to reduced costs and improved performance for our GPU-intensive applications, ultimately boosting our competitive edge in the market.

A10G / Qwen2.5‑7B‑AWQ

kubectl port-forward svc/vllm-a10g-qwen25-7b-awq 8003:8000

curl -s http://127.0.0.1:8003/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON' | jq -r '.choices[0].message.content'
{
  "model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
  "temperature": 0.2,
  "messages": [
    {
      "role": "user",
      "content": "You are a customer support assistant for an e-commerce store.\n\nTask:\n1) Read the ticket.\n2) Return ONLY valid JSON with fields: intent, sentiment, order_id, item, eligibility, next_steps, customer_reply.\n3) Keep the reply friendly, concise, and action-oriented.\n\nTicket:\n\"Order #A1234 — Hi, I bought running shoes 26 days ago. They’re too small. Can I exchange for size 10? I need them before next weekend. Happy to pay the price difference if needed. — Jamie\""
    }
  ]
}
JSON

Example output

{
  "intent": "Request for exchange",
  "sentiment": "Neutral",
  "order_id": "A1234",
  "item": "Running shoes",
  "eligibility": "Eligible for exchange within 30 days",
  "next_steps": "We can exchange your shoes for size 10. Please ship back the current pair and we'll send the new ones.",
  "customer_reply": "Thank you! Can you please confirm the shipping details?"
}

Clean Up

cd infra/aws
terraform destroy -auto-approve

Coming next (mini-series)

Advanced scheduling: GPU & Node binpack/spread, anti‑affinity, NUMA‑aware and NVLink‑aware placement, UUID pinning.
Container‑level monitoring: simple, reproducible checks for allocation & usage; shareable dashboards.
Under the hood: HAMi scheduling flow & HAMi‑core memory/compute capping (concise deep dive).
DRA: community feature under active development; we’ll cover support progress & plan.
Ecosystem demos: Kubeflow, vLLM Production Stack, Volcano, Xinference, JupyterHub. (**vLLM Production Stack, Volcano, and Xinference already have native integrations.)