Forem: Pratheesh Satheesh Kumar

Why Two-Thirds of AI Teams Are Betting on Kubernetes (And What That Means for You)

Pratheesh Satheesh Kumar — Mon, 04 May 2026 02:08:30 +0000

Kubernetes and AI have become unlikely bedfellows—and the numbers prove it. New data from CNCF and SlashData reveals that two-thirds of organizations running generative AI models have standardized on Kubernetes for orchestration. But here's the thing: it's not because Kubernetes magically solves AI problems. It's because the engineering fundamentals that make Kubernetes valuable—standardization, repeatability, resource isolation—are exactly what AI workloads demand when they move beyond the laptop and into production.

If you're building or scaling AI systems, this isn't just trivia. It's a signal about where the industry is converging, and whether Kubernetes is right for you depends less on hype and more on what you're actually trying to accomplish.

The Real Story Behind the Numbers

Let's be clear: Kubernetes didn't become the platform of choice for AI because it was purpose-built for LLMs or model inference. It became the default because:

Standardization across teams: When you have data scientists, ML engineers, and infrastructure teams all shipping models, Kubernetes provides a common deployment target. No more "it works on my machine" fragmentation.
Resource orchestration: AI workloads are hungry. GPUs, accelerators, memory—Kubernetes abstracts these away and lets you define what each model needs without manual provisioning.
Multi-tenancy at scale: If you're running multiple models for different teams or products, isolation and fair resource allocation become non-negotiable.

But here's what the data really highlights: success with AI still comes down to boring, foundational work. The teams winning aren't the ones who found the perfect Kubernetes YAML template. They're the ones with solid internal developer platforms (IDPs), clear observability, and a relentless focus on developer experience.

The IDP Question Every AI Team Needs to Answer

The most important implication from this research is the emphasis on internal developer platforms. Here's why:

AI teams move fast but often lack the operational maturity of traditional backend teams. They want to experiment, iterate, and ship—quickly. But you can't scale that without abstraction.

An effective IDP for AI sits between your data scientists (who want to ship models) and Kubernetes (which handles the orchestration). It provides:

Self-service model deployment: Data scientists submit a model; the platform handles GPU allocation, versioning, and rollback.
Standardized observability: Metrics, logs, and traces for inference endpoints—not just for ops, but for the ML team to catch drift and degradation early.
Cost visibility: AI is expensive. Your IDP should show teams exactly what their models cost to run.

# Example: A simplified model deployment abstraction
apiVersion: ml.company.io/v1
kind: ModelEndpoint
metadata:
  name: gpt-classifier-prod
spec:
  model: gcr.io/company/gpt-classifier:v2.1.3
  resources:
    accelerators: "nvidia.com/gpu: 2"
    memory: "32Gi"
  autoscaling:
    minReplicas: 2
    maxReplicas: 10
    targetUtilization: 70

This layer matters more than Kubernetes itself. Kubernetes is just the underlying engine.

Practical Takeaway: Do You Actually Need Kubernetes for AI?

Honest answer: probably, eventually. But not on day one.

If you're:

Running a single model for inference with predictable load → managed services (Vertex AI, SageMaker, Modal) might be faster to market.
Experimenting with models in notebooks → local containers and lightweight orchestration are enough.
Running multiple models, multiple teams, with variable workloads and cost constraints → Kubernetes becomes the logical choice.

The trap is assuming Kubernetes is the goal. It's not. The goal is reliable, scalable, observable AI systems that developers actually enjoy maintaining. Kubernetes is often the best tool for that—but it requires:

Strong foundations first: GitOps, infrastructure-as-code, observability.
An IDP on top: Don't expose Kubernetes complexity to data scientists.
Clear resource governance: AI compute is expensive; track it ruthlessly.

What's Missing from the Narrative

One thing the data doesn't capture: the operational overhead. Two-thirds of teams use Kubernetes for AI, but we don't know how many are struggling with it. How many are maintaining custom YAML hell? How many have visibility into whether their GPU allocation actually makes sense?

The fact that two-thirds converge on Kubernetes is less about it being perfect and more about it being the least bad option at scale. That's important context.

The Bottom Line

Kubernetes isn't demanded by AI. It's enabled by AI teams that have mature engineering practices and the discipline to build abstractions on top of it.

If you're starting an AI project, ask yourself: Do we have the fundamentals in place? Do we have an IDP or the plan to build one? If the answer is "not yet," Kubernetes can wait. If you're already managing multiple models across teams, you're probably not far from needing it.

What's your experience? Are you running AI workloads on Kubernetes? What would have made the journey smoother—and what would you do differently next time?

How Cloudflare Built Resilience: Lessons from Their Infrastructure Overhaul

Pratheesh Satheesh Kumar — Sun, 03 May 2026 04:06:18 +0000

How Cloudflare Built Resilience: Lessons from Their Infrastructure Overhaul

When a single misconfiguration can cascade across a global CDN and take down customer traffic, every deployment becomes a high-stakes decision. Cloudflare recently completed a massive push to make their infrastructure fundamentally more resilient—and their approach offers critical lessons for anyone operating at scale.

The Problem: Risk Concentrates in Configuration

Most infrastructure incidents don't happen because of hardware failures or clever attacks. They happen because someone pushed a configuration change, the change propagated faster than expected, and there was no circuit breaker in between.

Cloudflare's situation was familiar to anyone running global-scale systems: their engineering teams were shipping improvements constantly, but each deployment carried latent risk. A small mistake in a configuration file could reach millions of users before detection. The traditional guardrails—code review, staging tests, gradual rollouts—weren't enough to catch every edge case.

This is why they launched "Fail Small," an engineering initiative focused on preventing large-scale incidents by making small failures impossible to propagate.

The Two-Tool Foundation: Snapstone and Engineering Codex

The solution wasn't a single tool. Instead, Cloudflare invested in two complementary systems:

Snapstone: Safer Configuration Changes

Snapstone is a configuration validation and deployment framework that treats configuration changes with the same rigor as code deployments. Here's what makes it different:

Pre-flight validation: Changes are tested against historical traffic patterns and failure scenarios before rollout
Staged rollout control: Configuration doesn't flip globally—it rolls out in waves with automated rollback if anomalies appear
Change hygiene: Every configuration change is tagged with context: who changed it, why, what it affects, and what the rollback plan is

Think of it as infrastructure-as-code discipline applied to runtime configuration. The payoff is measurable: configuration-related incidents drop significantly because bad changes simply don't reach production simultaneously across all regions.

Engineering Codex: Embedding Best Practices

Tools alone don't prevent incidents—culture does. The Engineering Codex is Cloudflare's answer: a formalized knowledge base of "how we safely operate infrastructure" that's embedded into workflows.

When engineers write configuration or deploy services, they're nudged toward patterns that have been proven safe:

Deployment templates that encode retry logic and timeout handling
Configuration examples that highlight common failure modes
Runbooks that appear automatically when certain alerts fire

It's not gatekeeping. It's scaffolding. New engineers learn the "right way" by default, and experienced engineers can deviate with confidence because they understand the underlying principles.

Why This Matters Beyond Cloudflare

You might think: "Sure, this makes sense for a global CDN. But we're running a smaller operation." That's exactly backward.

Cloudflare's insight applies especially to smaller teams:

Your blast radius is fixed regardless of team size. A misconfigured load balancer breaks things just as hard at a 50-person startup as at Cloudflare.
You have fewer engineers to catch mistakes. Automation and frameworks matter more when you don't have five people reviewing every change.
Incidents are more expensive relative to revenue. A 2-hour outage costs a larger company less (relatively) than a small startup.

The Fail Small philosophy: Make the safe path the default path.

Actionable Takeaway: Start With Configuration as Code

If you take one thing from Cloudflare's approach, it's this:

Treat configuration changes with the same discipline as code deployments.

Today:

Audit your current configuration management. Is it in version control? Are changes tested before rollout? Is there a rollback procedure?
Identify your highest-risk configuration files (anything that affects traffic routing, authentication, or resource limits).
Implement one simple control: all changes to critical configuration must be reviewed and tested in staging before production rollout.

You don't need to build Snapstone from scratch. Tools like Terraform, ArgoCD, or even careful GitOps practices get you 80% of the way there.

The Bigger Picture: Resilience is Systematic

Cloudflare's Fail Small initiative reminds us that infrastructure resilience isn't about heroic incident response. It's about making bad outcomes progressively harder to achieve.

Each control they added—validation, staged rollouts, embedded best practices—removes one more degree of freedom from the "I broke production" state space.

What's one configuration change that could take down your service right now? How many approval gates stand between someone and deploying it? That's where to start.

What's your team's biggest source of configuration-related incidents? Have you invested in preventing them, or mostly in recovering from them? Drop your thoughts below.