Forem: Lalit Somavarapha

Optimizing GPU Workload Placement in Kubernetes with NVLink-Aware Scheduling

Lalit Somavarapha — Tue, 27 Jan 2026 22:33:50 +0000

The hidden performance tax

You bought GPUs with NVLink interconnects. You're probably not using them effectively.

NVLink provides high-bandwidth, low-latency communication between GPUs—up to 900 GB/s on modern hardware compared to ~64 GB/s over PCIe. For distributed training workloads, this difference is massive. Gradient synchronization, tensor parallelism, and model sharding all depend on fast GPU-to-GPU communication.

Here's the problem: Kubernetes doesn't know NVLink exists.

The default scheduler sees GPUs as interchangeable resources. Request 4 GPUs, get any 4 GPUs. But on a node with 8 GPUs arranged in two NVLink domains of 4 GPUs each, placement matters enormously. Four GPUs within the same NVLink domain can communicate at full NVLink speed. Four GPUs split across domains fall back to slower PCIe interconnects.

We measured up to 40% degradation in multi-GPU communication performance from suboptimal placement. That's a 40% tax on every distributed training job—paid invisibly, every time.

NVLink topology primer

Modern GPU nodes organize GPUs into NVLink domains—groups of GPUs with direct high-speed interconnects. A typical configuration:

Developer nodes (G1): 2 GPUs, 1 NVLink domain
Production nodes (G2): 4 GPUs, 2 NVLink domains (2 GPUs each)
High-performance nodes (G2-Expansion): 8 GPUs, 2 NVLink domains (4 GPUs each)

Within a domain, GPUs communicate via NVLink. Across domains, they fall back to PCIe or NVSwitch fabric—significantly slower.

The scheduling goal is simple: keep workloads within NVLink domain boundaries whenever possible.

The solution: NVLink-aware scoring

We built a Kubernetes scheduler plugin that scores nodes based on NVLink topology awareness. It operates in the Score phase of the scheduling cycle, evaluating candidate nodes after filtering and before final selection.

The plugin integrates with DCGM (Data Center GPU Manager) via Prometheus to track real-time GPU allocations, then applies a scoring algorithm that considers:

Domain integrity — Can the workload fit within a single NVLink domain?
Node-workload matching — Is this the right-sized node for this workload?
Domain completion — Does this placement fill a partially-used domain?
Resource efficiency — What's the node utilization after placement?

Scoring algorithm

Each node starts with a base score of 50 points, then receives adjustments.

Domain Integrity (primary signal)

This is the core optimization. Keeping GPUs within the same NVLink domain is the whole point.

Workload fits single domain: +50 points
Exact domain fit (e.g., 2 GPUs on 2-GPU domain): +30 bonus
Cross-domain placement: -20 per additional domain

Node-Workload Matching

Right-size workloads to nodes. Don't waste an 8-GPU node on a 1-GPU job.

Small workloads (1–2 GPUs) on small NVLink nodes: +60 to +80 points
Small workloads on oversized nodes: -40 points
Large workloads (5+ GPUs) on large nodes: +80 to +100 points
Large workloads on undersized nodes: -80 points

Domain Completion

Bin-pack within domains before spilling to new ones.

Placement completes a partially-filled domain: +40 points

Resource Efficiency

Favor high utilization to reduce fragmentation.

High utilization (>70%) after placement: +20 to +30 points
Low utilization (<30%): -20 points

Real-time allocation tracking

Static scoring isn't enough. The plugin needs to know which GPUs are currently allocated to make intelligent placement decisions.

We integrate with DCGM metrics exposed via Prometheus. This tells us which GPUs have active workloads and which pods own them. The plugin reconstructs domain state in real-time:

Domain 0: total=2, allocated=1, available=1
Domain 1: total=2, allocated=0, available=2

When a new 1-GPU workload arrives, the plugin recognizes that placing it in Domain 0 completes that domain (+40 bonus), while Domain 1 would leave fragmentation.

Example: Domain completion in action

Consider a 2-GPU node (single NVLink domain) with one GPU already allocated.

Incoming workload: 1 GPU

Scoring breakdown:

Base:              50
Domain integrity:  +50 (single domain)
Node matching:     +80 (perfect fit)
Domain completion: +40 (completes the domain)
Efficiency:        +30 (100% utilization)
───────────────────────
Final:             250 → normalized: 92/100

Compare to placing the same workload on an empty 8-GPU node:

Base:              50
Domain integrity:  +50 (single domain)
Node matching:     -40 (oversized node)
Domain completion:  0  (not completing anything)
Efficiency:        -20 (12.5% utilization)
───────────────────────
Final:             40 → normalized: 31/100

The 2-GPU node wins decisively, preserving the 8-GPU node for workloads that actually need it.

What we learned

Topology awareness compounds. The 40% improvement isn't just about individual job performance—it's about cluster-wide efficiency. Better placement means less fragmentation, which means more workloads scheduled successfully, which means higher overall utilization.

DCGM integration is essential. Without real-time allocation data, the plugin would make decisions based on stale information. The Prometheus integration adds minimal overhead but provides critical visibility.

Scoring weights need tuning. Different clusters have different workload mixes. A cluster dominated by small jobs might want stronger penalties for oversized placement. We exposed the key parameters for operator customization.

Related links:

Reclaiming Idle GPUs in Kubernetes: Why We Built a Custom Scheduler Plugin

Lalit Somavarapha — Fri, 09 Jan 2026 08:38:51 +0000

The Problem Nobody Talks About

GPUs are expensive — and yours are probably sitting idle right now. A single NVIDIA A100 can cost $10,000+, and in a Kubernetes cluster running AI workloads, you might have dozens of them. Here's the uncomfortable truth: most of the time, they're sitting idle. If you're struggling with GPU scheduling in Kubernetes or looking for ways to reclaim idle GPUs, you're not alone.

A data scientist spins up a training job, requests 4 GPUs, runs for two hours, then leaves for lunch. The GPUs sit allocated but unused. Meanwhile, another team's job is queued, waiting for resources that technically exist but aren't available.

Standard Kubernetes scheduling doesn't help here. It sees allocated resources as unavailable — period. It doesn't care whether those GPUs are actually being used.

What Kubernetes Gets Wrong About GPUs

Kubernetes was built for CPUs. Its scheduling model assumes resources are either allocated or free, with nothing in between. For CPUs, this mostly works — a pod using 10% of its requested CPU isn't blocking others in the same way.

GPUs are different. They're discrete, expensive, and often requested in large quantities. A pod requesting 4 GPUs gets exactly 4 GPUs, even if it's only actively using them 20% of the time. This is the core challenge of GPU resource management in Kubernetes — the scheduler has no concept of actual utilization.

The default Kubernetes preemption mechanism (DefaultPreemption) can evict lower-priority pods to make room for higher-priority ones. But it only considers priority — not actual utilization. A pod sitting completely idle has the same protection as one running a critical training job, as long as their priorities match.

We looked for existing solutions. NVIDIA's device plugin handles allocation but not reclamation. Cluster autoscaler can add nodes but won't reclaim idle resources on existing ones. Various GPU sharing approaches exist, but they don't address the fundamental scheduling problem.

The Core Idea: Utilization-Aware Preemption

We needed utilization-aware preemption that considers what GPUs are actually doing, not just what they've been allocated. The solution: a custom Kubernetes scheduler plugin for idle GPU reclaim that replaces the default preemption logic with something smarter.

The plugin, which we called ReclaimIdleResource, operates in the PostFilter phase of the scheduling cycle. This is where Kubernetes looks for preemption candidates when a pod can't be scheduled normally.

Here's the key insight: instead of just comparing priorities, we also query Prometheus for actual GPU utilization metrics from DCGM (NVIDIA's Data Center GPU Manager). A pod is only eligible for preemption if:

Its priority is below the preemptor's threshold
It's been running long enough to establish a usage pattern
Its actual GPU utilization is below a configured threshold

This means an idle pod with priority 1000 can be preempted by a pod with priority 500, if the idle pod isn't actually using its GPUs.

How It Works

The plugin hooks into the scheduler as a PostFilter extension:

profiles:
- schedulerName: default-scheduler
  plugins:
    postFilter:
      enabled:
      - name: ReclaimIdleResource
      disabled:
      - name: DefaultPreemption

When a GPU-requesting pod can't be scheduled, the plugin:

Checks cooldown — Has this pod recently triggered preemption? If so, wait. This prevents thrashing.
Scans potential victims — Finds all lower-priority pods on candidate nodes that have GPUs.
Evaluates each victim — Parses its PriorityClass for reclaim policy annotations, checks if it's still in its "toleration period" (grace period after scheduling), queries Prometheus for average GPU utilization over the monitoring window, and compares utilization against the idle threshold.
Selects minimal victims — Sorts eligible victims by GPU count (descending) and priority (ascending), then selects the minimum set needed to free enough GPUs.
Validates the decision — Runs filter plugins to confirm the preemptor will actually fit after preemption.

The policy is defined per-PriorityClass through annotations:

kind: PriorityClass
metadata:
  name: batch-workload
  annotations:
    # Pods can be preempted if idle (<10% GPU) for 1 hour, by priority ≥10000
    reclaim-idle-resource.scheduling.x-k8s.io/minimum-preemptable-priority: "10000"
    reclaim-idle-resource.scheduling.x-k8s.io/toleration-seconds: "3600"
    reclaim-idle-resource.scheduling.x-k8s.io/resource-idle-seconds: "3600"
    reclaim-idle-resource.scheduling.x-k8s.io/resource-idle-usage-threshold: "10.0"
value: 8000

This says: pods in this priority class can tolerate preemption for one hour after scheduling, and can be preempted if their GPU usage stays below 10% for an hour — but only by pods with priority 10000 or higher.

Key Design Decisions

Why PriorityClass annotations?

We considered a custom CRD, but PriorityClass already exists in the scheduling mental model. Teams already think about priority when designing workloads. Adding reclaim policy as annotations keeps the configuration close to where people expect it.

Why a monitoring window instead of instant utilization?

GPU workloads are bursty. A training job might spike to 100% utilization during forward/backward passes, then drop to near-zero during data loading. Instant measurements would give false positives. We use a configurable window (typically 30–60 minutes) to capture the true usage pattern.

Why query Prometheus instead of using in-memory metrics?

The scheduler runs as a single replica. We needed utilization data that survives scheduler restarts and can be queried historically. DCGM exports to Prometheus naturally, and most GPU clusters already have this pipeline.

Why a cooldown period?

Without it, a preemptor pod could trigger preemption, fail to schedule for unrelated reasons, and immediately trigger another preemption attempt. The 30-second cooldown prevents rapid-fire preemption storms.

What We Learned

Tuning matters more than we expected. The idle threshold and monitoring window need to match your workload patterns. Too aggressive and you'll preempt jobs mid-training. Too conservative and you won't reclaim much.

Observability is essential. We added extensive logging and Kubernetes events so operators can understand why preemption decisions were made. When someone's job gets preempted, they want to know why.

MIG complicates everything. NVIDIA's Multi-Instance GPU feature means a single physical GPU can be partitioned. We had to add partition-size compatibility checks to avoid preempting pods on nodes where the preemptor couldn't run anyway.

What's Next

This article covers the "what" and "why." In the next post, I'll walk through building a Kubernetes scheduler plugin from scratch in Go — the project structure, the interfaces you need to implement, and the gotchas we hit along the way.

If you're running GPU workloads on Kubernetes and wrestling with utilization challenges, I'd love to hear how you're approaching it — drop a comment below!