Forem: DevOps Daily

Claude Code Hidden Features You Probably Missed

DevOps Daily — Wed, 01 Apr 2026 17:21:58 +0000

Most people use Claude Code to write code, fix bugs, and maybe generate a commit message. That's fine, but you're leaving a lot on the table.

Boris Cherny, the creator of Claude Code, recently shared a thread on X about features that even daily users tend to overlook. Some of these genuinely changed how I work. Here's a rundown of the ones worth knowing about.

TLDR

Claude Code has mobile sessions, automated scheduling, voice input, parallel agents, git worktrees, hooks, and a browser extension. Most people use about 20% of what it can do.

Move Your Session Anywhere with /teleport

You can start a session on your laptop and pick it up on your phone. Or move it to the web. The /teleport command transfers your full session context between devices.

The reverse also works. If you're reviewing something on your phone during a commute, you can /teleport it back to your terminal when you sit down.

There's also /remote-control which lets you connect to a running session from another device without transferring it.

# On your laptop
/teleport

# On your phone or web - enter the code to pick up the session

This is useful when you kick off a long-running task on your workstation and want to check progress from your phone.

Automate Repetitive Tasks with /loop and /schedule

This one is a genuine workflow changer. You can tell Claude Code to run a task on a recurring schedule for up to a week.

# Review PRs every 30 minutes
/loop 30m review open PRs and post comments

# Run a health check every hour
/schedule every 1h check if the staging environment is healthy

Think about what you do repeatedly: reviewing PRs, checking CI status, monitoring deployments, updating dependencies. You can automate all of it without writing a single script.

Some practical examples:

Review all open PRs every morning at 9 AM
Monitor a Slack channel for feedback and create GitHub issues
Run your test suite after every push and report failures
Check for dependency updates weekly

Hooks for Deterministic Automation

Hooks let you run code at specific points in Claude Code's lifecycle. Unlike the AI-driven /loop command, hooks are deterministic - they always run the same way.

You configure them in your settings and they fire on events like:

Session start - set up your environment, load context
Before bash commands - validate or log commands before execution
On permission requests - auto-approve specific patterns
Continuous operation - keep Claude running without manual intervention

This is powerful for teams. You can enforce standards (like running linters before every commit) without relying on each engineer to remember.

Git Worktrees for Parallel Sessions

If you've ever wanted Claude to work on two different branches at the same time, worktrees make this possible. Each session gets its own isolated copy of the repo.

# Start a session in a worktree
claude --worktree

Why this matters: you can have Claude refactoring module A while simultaneously building feature B. Neither session interferes with the other.

This pairs well with /batch, which fans out work across dozens of parallel agents. Need to update 50 files? /batch can process them concurrently instead of one at a time.

Voice Input with /voice

You can dictate to Claude instead of typing. This sounds gimmicky until you try it for longer explanations.

/voice

It's particularly useful for:

Explaining complex requirements ("I need a migration that handles both the old and new schema formats, with a rollback path if...")
Code reviews ("Look at the authentication flow in this PR and tell me if...")
Brainstorming ("What's the best way to structure this API given these constraints...")

Typing detailed prompts takes time. Talking is faster for anything longer than a few sentences.

The Chrome Extension for Frontend Work

Claude Code has a Chrome extension that lets the AI see what your app looks like in the browser. Instead of describing UI bugs, Claude can verify its own output visually.

This closes the feedback loop for frontend work. Claude makes a change, checks the browser, adjusts if something looks off. You stop being the human screenshot tool.

/branch and --fork-session for Experiments

Want to try two different approaches to the same problem? /branch creates a copy of your current session so you can explore a different path without losing your progress.

# Fork the current session
/branch

# Or fork when starting
claude --fork-session <session-id>

This is like git branches but for your AI conversation. Try approach A in one branch, approach B in another, then pick the winner.

/btw for Side Questions

When Claude is working on a long task, you might have an unrelated question. Instead of interrupting the main task, /btw lets you ask a side question.

/btw what's the difference between SIGTERM and SIGKILL?

Claude answers your side question and goes right back to what it was doing. No context switching, no lost progress.

--bare for SDK Speed

If you're using Claude Code in scripts or CI pipelines, the --bare flag skips loading plugins and extra features, making startup up to 10x faster.

claude --bare -p "generate a migration for adding user roles"

This matters when you're calling Claude from automation scripts where every second counts.

--add-dir for Multi-Repo Work

Working across multiple repositories? You can give Claude access to all of them in a single session.

claude --add-dir ~/projects/api --add-dir ~/projects/frontend

Now Claude can see your API schema and your frontend code at the same time. No more copying types between repos or explaining your API structure manually.

Custom Agents with --agent

You can create custom agent configurations with their own system prompts and tool permissions.

claude --agent reviewer    # Uses your custom reviewer agent config
claude --agent deployer    # Uses your custom deployer agent config

Define these in your .claude/agents/ directory. Each agent can have different instructions, different tool access, and different behaviors. A code reviewer agent doesn't need write access. A deployment agent doesn't need to browse the web.

What This Means for DevOps

These features shift Claude Code from "AI code assistant" to "AI DevOps team member." The combination of scheduling, hooks, parallel sessions, and multi-repo access means you can automate workflows that previously required custom tooling.

Here's a realistic DevOps setup:

/schedule reviews all PRs every morning
Hooks enforce linting and security scanning on every session
Worktrees let you debug production while shipping features
--add-dir gives Claude access to your infra and app repos simultaneously
/loop monitors your staging environment and alerts you on issues

The key insight from Boris's thread: "There is no one right way to use Claude Code." The tool is intentionally flexible. Experiment with these features and build the workflow that fits your team.

Try It Out

If you haven't updated Claude Code recently, run:

claude update

Many of these features are recent additions. The mobile app, scheduling, and hooks in particular have been added in the last few months.

For more DevOps tools and guides, check out our exercises and quizzes to sharpen your skills.

This post was inspired by Boris Cherny's thread on X. Boris is the creator of Claude Code at Anthropic.

🎄 Advent of DevOps: 25 Days to Level Up Your DevOps Game!

DevOps Daily — Sun, 30 Nov 2025 22:00:00 +0000

Hey DevOps enthusiasts! 👋

Remember how exciting advent calendars were as a kid? Each day bringing a new surprise behind those little doors? Well, we're bringing that same excitement to the DevOps world, but instead of chocolate (sorry! 🍫), you're getting something even better: real-world DevOps skills that will make you a better engineer.

🎁 What is Advent of DevOps?

Think "Advent of Code" meets real-world DevOps challenges. Starting December 1st, we're releasing 25 daily hands-on challenges that cover everything you need to know to thrive in modern DevOps environments.

Each day unlocks a new practical challenge focusing on tools and techniques you'll actually use in production. No theory-heavy lectures, no boring slides—just pure, hands-on learning that you can apply immediately.

🚀 What's Inside?

Here's a taste of what you'll tackle over 25 days:

🐳 Containerization & Orchestration
⚙️ CI/CD & Automation
🏗️ Infrastructure as Code
🔒 Security & Observability
☁️ Cloud & Scaling

💡 Why Join?

🎯 Real-World Skills: Every challenge is based on actual scenarios you'll face in production

📈 Progressive Learning: Start easy, level up gradually. Whether you're a beginner or seasoned pro, there's something for you

🎮 Fun & Engaging: Gamified progress tracking makes learning addictive (in a good way!)

🌟 Community-Driven: Share solutions, learn from others, and grow together

⏰ Learn at Your Pace: Can't keep up daily? No problem! All challenges remain available year-round

🎄 How It Works

Pick Your Challenge: Start with Day 1 or jump to what interests you most
Get Hands-On: Each challenge includes clear tasks, starter code, and success criteria
Build & Learn: Complete the challenge at your own pace
Share & Celebrate: Post your wins and solutions with the community
Level Up: Review reference solutions and explanations to deepen your understanding

Each challenge includes:

✅ Clear task description
🎯 Success criteria
🔧 Starter code (when applicable)
💡 Solution & explanation
🔗 Additional resources

🌟 Join the Community

This isn't just about solo learning—it's about growing together!

Share your progress:

Follow us on X/Twitter: @thedevopsdaily
Use hashtag: #AdventOfDevOps
Share on LinkedIn, dev.to, wherever you hang out!

Contribute:

Found a cool solution? Share it!
Have ideas for challenges? We're open-source!
Check out our GitHub repo and contribute

🎯 Ready to Start?

Don't wait for December 1st to check it out—head over to the page now and get familiar with what's coming:

👉 devops-daily.com/advent-of-devops

Mark your calendar 📅, set your reminders ⏰, and get ready to transform your DevOps skills one day at a time!

🤔 Who Should Join?

DevOps Engineers looking to sharpen their skills
Developers wanting to understand the ops side better
System Administrators transitioning to DevOps
Students & Career Changers building practical experience
Anyone curious about modern infrastructure practices

No gatekeeping here, if you're interested in DevOps, you're welcome! 🙌

🎊 Let's Make This December Special

Learning doesn't have to be boring. It doesn't have to be stressful. And it definitely doesn't have to be lonely.

This December, join hundreds (thousands?) of DevOps practitioners around the world in leveling up together. One challenge at a time, one skill at a time, one day at a time.

See you on December 1st! 🎄✨

P.S. - Can't wait? Start exploring the challenges now at devops-daily.com/advent-of-devops. They're already live and ready for early birds! 🐦

P.P.S. - This is completely free, open-source, and community-driven. No paywalls, no upsells, just pure learning. If you find value, give us a star on GitHub and spread the word! ⭐

Follow DevOps Daily:

🐦 X/Twitter: @thedevopsdaily
💻 GitHub: The-DevOps-Daily/devops-daily
🌐 Website: devops-daily.com

Happy DevOps-ing! 🚀

Building a DDoS Attack Simulator to Understand Defense Strategies

DevOps Daily — Fri, 21 Nov 2025 09:53:22 +0000

I created an educational content piece for DevOps Daily and realized something: most explanations of DDoS attacks are either too abstract or too technical. We talk about "request floods" and "mitigation strategies," but it's hard to visualize what's actually happening.

So I built an interactive simulator to help bridge that gap.

The Problem with Learning About DDoS 📚

When you're reading about DDoS protection, you see phrases like "distributes load across multiple servers" or "rate limiting prevents abuse." But what does that actually mean when thousands of requests are hitting your infrastructure?

I wanted something that would help people - especially those newer to infrastructure work - actually see these concepts in action.

What the Simulator Does 🎮

You can try it here: devops-daily.com/games/ddos-simulator

It lets you simulate three common attack types:

HTTP Flood 🌊 - overwhelming with legitimate-looking requests
SYN Flood 🔄 - exploiting TCP handshake mechanics
UDP Flood 📦 - connectionless packet storms

The interesting part is watching how different defense mechanisms respond. You can toggle:

Firewall 🛡️ - blocks about 30% based on signatures
Load Balancer ⚖️ - reduces impact by 50%
Auto Rate Limit 🚦 - blocks high-frequency traffic

What I Learned Building It 💡

A few things became clear while working on this:

Attack intensity matters less than you'd think. The attack type and your defense configuration matter way more. A moderate SYN flood with no defenses is worse than an intense HTTP flood with proper rate limiting.

Single defenses aren't enough. This is obvious in theory, but seeing it play out makes it concrete. A firewall alone, or a load balancer alone, only gets you so far.

Visualization helps understanding. Watching the server health bar drop while packets animate across the screen creates an intuition that documentation doesn't.

Who Might Find This Useful ⚙️

If you're:

Learning about infrastructure security
Trying to explain DDoS concepts to your team
Deciding what protections to implement
Just curious how attacks and defenses interact

It might be helpful to play around with it for a bit.

What's Next 🚀

I'm planning to add more waves with additional attack vectors and defense mechanisms. Things like:

Application-layer attacks
CDN protection
Anycast routing
More realistic traffic patterns

If you have thoughts on what would be useful to include, I'd be interested to hear them.

The goal here is education, not creating chaos. Understanding how attacks work helps you build better defenses. 🛡️

If you try it out, let me know what you think or if anything is unclear.

Right-Sizing Kubernetes Resources with VPA and Karpenter

DevOps Daily — Fri, 22 Aug 2025 17:02:04 +0000

TLDR

Setting CPU and memory requests too high in Kubernetes wastes money and reduces cluster efficiency. This guide shows you how to identify overprovisioned workloads, use Vertical Pod Autoscaler (VPA) to right-size your pods, and implement Karpenter for smarter node scaling. You'll also learn to monitor costs and validate your improvements with real metrics.

When you set resource requests too conservatively in Kubernetes, your cluster reserves more capacity than workloads actually need. This leads to underutilized nodes and higher cloud bills. The problem gets worse at scale - imagine 200 pods each requesting 2 CPU cores but only using 200m. That's 400 reserved cores when actual demand is closer to 40 cores.

The solution involves right-sizing both your pods and nodes. You'll use monitoring data to understand actual usage, apply VPA to adjust pod requests automatically, and leverage Karpenter to provision nodes that match your workload requirements.

Prerequisites

Before you start, make sure you have:

A Kubernetes cluster (version 1.20 or higher) with metrics-server installed
kubectl configured with admin access to your cluster
Prometheus and Grafana deployed for monitoring (or similar observability stack)
Basic understanding of Kubernetes resource requests and limits

You'll also need the ability to install cluster-wide components like VPA and Karpenter.

Identifying Overprovisioned Workloads

The first step is understanding how your current workloads use resources compared to what they request. You can start with kubectl to get a quick snapshot of resource usage across your cluster.

# Check current resource usage for all nodes
kubectl top nodes

# View pod resource usage across all namespaces
kubectl top pods --all-namespaces --sort-by=cpu

# Get detailed resource requests vs usage for a specific namespace
kubectl describe nodes | grep -A 15 "Allocated resources"

These commands show you the gap between requested and actual resource usage. If you see pods consistently using 50Mi of memory while requesting 1Gi, or using 100m CPU while requesting 1000m, those are prime candidates for right-sizing.

For deeper analysis, you'll want historical data from Prometheus. Here are some key queries to run in your Grafana dashboard:

# CPU utilization percentage (actual usage vs requests)
(rate(container_cpu_usage_seconds_total{container!=""}[5m]) * 100) /
(container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""})

# Memory utilization percentage
(container_memory_working_set_bytes{container!=""} * 100) /
container_spec_memory_limit_bytes{container!=""}

# Top 10 pods with the highest request-to-usage ratio (biggest waste)
topk(10,
  (container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""}) /
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Run these queries over a 2-week period to account for traffic variations and identify consistent patterns. Workloads running at 10-20% utilization with stable traffic are good candidates for optimization.

Installing and Configuring VPA

Vertical Pod Autoscaler analyzes your workloads and recommends optimal CPU and memory values. Start by installing VPA in your cluster.

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Deploy VPA components
./hack/vpa-up.sh

This script installs three main components: the VPA recommender (analyzes usage), the updater (applies changes), and the admission controller (validates recommendations).

Next, create a VPA configuration for a workload you want to optimize. Start with recommendation mode to see suggested values before making changes.

# vpa-web-service.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: 'apps/v1'
    kind: Deployment
    name: web-service
  updatePolicy:
    updateMode: 'Off' # Only provide recommendations, don't auto-update
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        # Set boundaries to prevent extreme recommendations
        maxAllowed:
          cpu: '2'
          memory: '4Gi'
        minAllowed:
          cpu: '100m'
          memory: '128Mi'
        controlledResources: ['cpu', 'memory']

Apply the VPA configuration and wait for recommendations to generate:

kubectl apply -f vpa-web-service.yaml

# Wait a few minutes, then check recommendations
kubectl describe vpa web-service-vpa -n production

The output shows recommended values for CPU and memory under the Status section. VPA typically suggests values based on the 90th percentile of usage over the past 8 days, which provides a safety buffer while eliminating waste.

Applying VPA Recommendations Safely

Once you have solid recommendations, you can apply them gradually. Start with non-critical workloads and monitor for any issues.

# Update your deployment with VPA recommendations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-service
  template:
    metadata:
      labels:
        app: web-service
    spec:
      containers:
        - name: web-app
          image: nginx:1.21
          resources:
            requests:
              cpu: '250m' # Reduced from 1000m based on VPA recommendation
              memory: '512Mi' # Reduced from 2Gi based on VPA recommendation
            limits:
              cpu: '500m' # Set limits 2x requests for burst capacity
              memory: '1Gi'

After updating requests, monitor your workloads for at least a week. Watch for:

Increased pod restarts or OOMKilled events
Higher response times or error rates
Pods getting evicted under memory pressure

If everything runs smoothly, you can switch VPA to automatic mode:

# Update VPA to automatically apply changes
kubectl patch vpa web-service-vpa -n production --type='merge' -p='{"spec":{"updatePolicy":{"updateMode":"Auto"}}}'

In Auto mode, VPA will restart pods when it detects they need different resource allocations. Make sure you have proper PodDisruptionBudgets in place to maintain availability during updates.

Setting Up Karpenter for Node Optimization

While VPA optimizes individual pods, Karpenter optimizes your entire node infrastructure. Instead of fixed node groups, Karpenter provisions nodes dynamically based on your workload requirements.

First, install Karpenter in your cluster. The exact steps depend on your cloud provider, but here's the process for AWS EKS:

# Install Karpenter using Helm
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "0.32.0" \
  --namespace "karpenter" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueueName=${CLUSTER_NAME}" \
  --wait

Next, create a NodePool that defines what types of nodes Karpenter can provision:

# karpenter-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  # Template for nodes Karpenter will create
  template:
    metadata:
      labels:
        node-type: general-purpose
    spec:
      # Instance requirements - Karpenter will pick the best fit
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand'] # Allow both for cost optimization
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ['m6i.large', 'm6i.xlarge', 'm6i.2xlarge', 'r6i.large', 'r6i.xlarge']

      # Node configuration
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: general-purpose

      # Taints to control which pods can schedule here
      taints:
        - key: karpenter.sh/unschedulable
          value: 'true'
          effect: NoSchedule

  # Scaling and disruption policies
  limits:
    cpu: 1000 # Maximum CPU across all nodes in this pool
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Create the corresponding EC2NodeClass for AWS-specific configuration:

# karpenter-nodeclass.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: general-purpose
spec:
  # AMI and instance configuration
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'

  # Instance store configuration
  userData: |
    #!/bin/bash
    /etc/eks/bootstrap.sh ${CLUSTER_NAME}

  # Tags for cost tracking
  tags:
    Team: platform
    Environment: production

Apply both configurations:

kubectl apply -f karpenter-nodepool.yaml
kubectl apply -f karpenter-nodeclass.yaml

Karpenter will now monitor unschedulable pods and provision appropriately-sized nodes. When you deploy workloads with right-sized resource requests (thanks to VPA), Karpenter will select smaller, more cost-effective instances.

Monitoring Cost Impact

To validate your optimizations, you need visibility into resource costs. Kubecost provides detailed insights into how much each workload costs and how much capacity you're wasting.

Install Kubecost in your cluster:

# Add the Kubecost Helm repository
helm repo add kubecost https://kubecost.github.io/cost-analyzer/

# Install Kubecost with Prometheus integration
helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --create-namespace \
  --set kubecostToken="your-token-here" \
  --set prometheus.server.global.external_labels.cluster_id="${CLUSTER_NAME}"

Access the Kubecost UI by port-forwarding:

kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9090:9090

In the Kubecost dashboard, focus on these key metrics:

Efficiency scores: Shows the percentage of requested resources actually being used
Idle costs: Money spent on provisioned but unused resources
Right-sizing recommendations: Suggestions for adjusting requests and limits
Namespace costs: Helps identify which teams or applications drive costs

Track these metrics before and after implementing VPA and Karpenter to quantify your savings.

Real-World Optimization Example

Let's walk through optimizing a typical microservice deployment. You start with a Node.js API that was conservatively configured:

# Before optimization
resources:
  requests:
    cpu: '1000m'
    memory: '2Gi'
  limits:
    cpu: '2000m'
    memory: '4Gi'

After running this workload for two weeks, your monitoring shows:

Average CPU usage: 150m (15% of requests)
Average memory usage: 400Mi (20% of requests)
Peak CPU usage: 300m
Peak memory usage: 800Mi

Based on this data, VPA recommends:

# VPA recommendations (with safety buffer)
resources:
  requests:
    cpu: '200m' # Covers 99th percentile usage
    memory: '512Mi' # Accounts for memory spikes
  limits:
    cpu: '400m' # 2x requests for burst capacity
    memory: '1Gi' # Prevents OOM while allowing growth

The cost impact for 20 replicas of this service:

Before: 20 CPU cores, 40Gi memory requested
After: 4 CPU cores, 10Gi memory requested
Savings: 80% reduction in resource allocation

With Karpenter managing nodes, this workload now runs on smaller instances, further reducing costs by eliminating the need for oversized nodes.

Setting Resource Quotas and Guardrails

As you roll out right-sizing across your organization, implement quotas to prevent teams from reverting to oversized requests:

# namespace-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: backend-team-quota
  namespace: backend
spec:
  hard:
    requests.cpu: '50' # Total CPU requests across all pods
    requests.memory: '100Gi' # Total memory requests
    limits.cpu: '100' # Total CPU limits
    limits.memory: '200Gi' # Total memory limits
    pods: '100' # Maximum number of pods

You can also create LimitRanges to enforce reasonable defaults:

# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
  namespace: backend
spec:
  limits:
    - type: Container
      default: # Default limits if not specified
        cpu: '500m'
        memory: '1Gi'
      defaultRequest: # Default requests if not specified
        cpu: '100m'
        memory: '256Mi'
      max: # Maximum allowed values
        cpu: '4'
        memory: '8Gi'
      min: # Minimum required values
        cpu: '50m'
        memory: '64Mi'

These guardrails help maintain optimization gains while giving teams flexibility within reasonable bounds.

Troubleshooting Common Issues

When implementing VPA and Karpenter, you might encounter some challenges. Here are solutions to the most common problems:

VPA recommendations seem too aggressive: VPA sometimes suggests very low values during low-traffic periods. Check that your monitoring data covers representative traffic patterns. You can also adjust the VPA algorithm:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        controlledValues: RequestsOnly # Only adjust requests, leave limits alone
        mode: Auto

Karpenter nodes aren't scaling down: This usually happens when pods can't be evicted. Check for:

# Look for pods without PodDisruptionBudgets
kubectl get pods --all-namespaces -o wide | grep -v Terminating

# Check for pods using local storage or host networking
kubectl get pods --all-namespaces -o yaml | grep -A 5 hostNetwork

# Verify PodDisruptionBudgets allow eviction
kubectl get pdb --all-namespaces

Pods getting OOMKilled after VPA optimization: This indicates VPA recommendations were too low. Temporarily increase memory requests and check for memory leaks in your application:

# Check recent OOM events
kubectl get events --sort-by=.metadata.creationTimestamp | grep OOMKilled

# Monitor memory usage patterns
kubectl top pods --sort-by=memory --all-namespaces

You can make VPA more conservative by setting higher safety margins:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        maxAllowed:
          memory: '2Gi' # Set a reasonable upper bound

Next Steps

Now that you have VPA and Karpenter working together, consider these additional optimizations:

Horizontal Pod Autoscaling: Combine with VPA to handle both vertical and horizontal scaling
Cluster Autoscaler tuning: If using multiple node provisioners, configure them to work together
Cost alerts: Set up notifications when resource costs exceed thresholds
Regular reviews: Schedule monthly reviews of VPA recommendations and cost reports

You can also explore more advanced Karpenter features like multiple NodePools for different workload types (CPU-intensive, memory-intensive, GPU workloads) and spot instance strategies for non-critical workloads.

The key is to treat right-sizing as an ongoing process. As your applications evolve and traffic patterns change, continue monitoring and adjusting to maintain optimal resource utilization.

The 5-Minute Kubernetes Cluster Health Check

DevOps Daily — Fri, 15 Aug 2025 10:30:08 +0000

TLDR

You can check your Kubernetes cluster's health in under 5 minutes using five key commands: checking node status, monitoring resource usage, reviewing pod health across namespaces, investigating problem pods, and examining cluster events. This quick routine helps catch issues before they escalate into critical problems.

Kubernetes is great until it's not. One bad node, a pod stuck in CrashLoopBackOff, or a resource spike can ruin your day. The good news? You don't need to spend an hour digging through dashboards to spot trouble early. With a few quick commands, you can get a solid read on your cluster's health in under 5 minutes.

Here's how to do it effectively.

Make Sure Your Nodes Are Happy

Start by checking the overall status of your cluster nodes. This gives you the foundation-level health of your infrastructure.

kubectl get nodes -o wide

This command displays all nodes in your cluster along with their detailed information. You'll see each node's status, roles, age, version, internal and external IPs, OS image, kernel version, and container runtime.

What you want to see:

STATUS should be Ready for all nodes
No mystery nodes suddenly showing up in your cluster
Roles, IPs, and ages that make sense for your environment

If you spot NotReady, that's your cue to dig deeper. A node in this state might be experiencing network issues, resource exhaustion, or kubelet problems.

Check Resource Usage at a Glance

Next, get a quick overview of resource consumption across your nodes to identify potential bottlenecks.

kubectl top nodes

This command shows CPU and memory usage for each node in your cluster. It provides both absolute values and percentages, making it easy to spot resource pressure.

Keep an eye out for:

CPU or memory regularly above 80% on any node
One node doing all the heavy lifting while others are barely working
Sudden spikes that don't match your expected workload patterns

No metrics-server running? Install it with this command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

The metrics-server is essential for resource monitoring and is required for horizontal pod autoscaling to work properly.

Look at All Pods Across All Namespaces

Get a bird's-eye view of all pods running in your cluster to quickly identify any that are misbehaving.

kubectl get pods --all-namespaces

This command lists every pod across all namespaces, showing their current status, restart count, and age. It's like taking the pulse of your entire application ecosystem.

Healthy pods should be Running or Completed. If you see states like CrashLoopBackOff, ImagePullBackOff, Pending, or Error, note the namespace and pod name for further investigation.

Also watch the RESTARTS column closely. If a pod has restarted a dozen times in the last hour, something's definitely off. Frequent restarts often indicate:

Application crashes due to bugs or configuration issues
Failing health checks (readiness or liveness probes)
Resource limits being exceeded
Dependencies being unavailable

Zoom In on Problem Pods

When you spot problematic pods, dig deeper to understand what's causing the issues.

kubectl describe pod <pod-name> -n <namespace>

Replace <pod-name> and <namespace> with the actual values from your problem pods. This command provides detailed information about the pod's configuration, current state, and recent events.

Check for these common issues:

Events at the bottom (often the smoking gun that reveals the root cause)
Failing readiness or liveness probes that prevent the pod from receiving traffic
Image pull errors indicating registry access problems or incorrect image names
Resource limit issues where the pod exceeds its memory or CPU constraints

The events section is particularly valuable because it shows a chronological history of what happened to the pod, including scheduling decisions, volume mounts, and error conditions.

Check the Cluster's Event Log

Get insight into what's been happening across your entire cluster by examining recent events.

kubectl get events --sort-by=.metadata.creationTimestamp

This command shows cluster-wide events sorted by when they occurred, giving you a timeline of recent activity. Events provide context about system-level operations and can reveal patterns or issues that affect multiple components.

Events will tell you what's been happening behind the scenes:

Failed volume mounts that prevent pods from starting
DNS resolution errors affecting service communication
Scheduling issues when pods can't be placed on nodes
Node pressure warnings indicating resource constraints

Try k9s for a Better View

If you want something more interactive than command-line tools, give k9s a try. It's a terminal-based UI for Kubernetes that provides real-time cluster information in an intuitive interface.

k9s lets you browse resources, view logs, and drill into problems without typing long commands. You can navigate between different resource types using simple keystrokes, filter resources, and even perform actions like scaling deployments or deleting pods.

Once you try k9s, it's hard to go back to plain kubectl for exploratory tasks. It's particularly useful when you need to quickly jump between different namespaces or resource types during troubleshooting.

Five minutes a day is all it takes to stay ahead of most cluster problems. Make this health check part of your daily routine and you'll catch issues before they blow up and before your pager goes off at 3 a.m. Regular monitoring helps you understand your cluster's normal behavior, making it easier to spot anomalies when they occur.

What’s the Most Underrated DevOps Skill You’ve Learned (and How Did You Learn It)?

DevOps Daily — Tue, 05 Aug 2025 07:43:56 +0000

When we think about DevOps skills, we usually picture Kubernetes, Terraform, CI/CD pipelines, or cloud automation.

But some of the most valuable skills are the ones that never make it into a certification or a tech stack diagram.

It could be:

Staying calm during a production incident and knowing how to prioritize actions
Communicating effectively with teams under pressure
Spotting patterns in logs and metrics that others might miss
Finding ways to optimize cloud costs without slowing down delivery
Automating the boring stuff so you can focus on the real problems

For me, one of the most underrated skills I've learned is knowing when not to automate something. Sometimes the "manual but reliable" approach saves you from a lot of complexity and maintenance overhead later.

What about you?
What's the most underrated DevOps skill you've picked up along the way, and how did you learn it?

P.S. You might find some useful DevOps resources at devops-daily.com

What's the One DevOps "Best Practice" You Secretly Ignore (and Why)?

DevOps Daily — Wed, 30 Jul 2025 14:04:41 +0000

We've all read the books, followed the gurus, and tried to tick every box in the DevOps checklist.. but let’s be honest:

There's always that one best practice that just doesn’t work for your team, your stack, or your sanity.

Maybe you don't write as many tests as you should.
Maybe you still SSH into production (👀).
Maybe you use latest tags on your Docker images and pray.

No judgment here, just real talk from the trenches.

What's your "ignored" DevOps best practice, and why do you skip it?

Bonus points if you share how it's actually worked out for you.

🛠️ Posted by the team behind DevOps Daily

The Complete DevOps Roadmap for 2025 🚀

DevOps Daily — Sat, 26 Jul 2025 13:13:38 +0000

The DevOps landscape continues to evolve rapidly, and 2025 presents incredible opportunities for aspiring engineers. Organizations are increasingly adopting DevOps practices to deliver software faster, more reliably, and at scale. The demand for skilled DevOps professionals has never been higher.

Whether you're a developer looking to expand into operations, a system administrator aiming to modernize your skills, or a complete beginner drawn to this exciting field, this comprehensive roadmap will guide your journey to DevOps mastery.

Why DevOps in 2025? 🌟

DevOps represents a fundamental shift in how software is built, deployed, and maintained. It's not just about tools, it's about culture, collaboration, and continuous improvement. Here's why it matters:

🔄 Faster Delivery: Teams deploy multiple times per day instead of monthly releases
🛡️ Better Reliability: Automated testing and monitoring catch issues early
⚡ Improved Collaboration: Breaks down silos between development and operations
🔧 Enhanced Automation: Reduces manual work and human error
📈 Career Growth: High demand for skilled professionals across all industries

But beyond the benefits, DevOps offers intellectually rewarding work where you solve complex problems and see immediate impact on product delivery.

DevOps in the Age of AI: Why Infrastructure Matters More Than Ever 🤖

With AI transforming every industry, you might wonder: "Is DevOps still a smart career choice?" The answer is a resounding yes, and here's why:

🏗️ AI Runs on Infrastructure

Every AI application, from ChatGPT to autonomous vehicles, depends on robust, scalable infrastructure:

🚀 Model Training: Requires massive computational resources and distributed systems
⚡ Real-time Inference: Needs low-latency, highly available services
📊 Data Pipelines: AI models need continuous data flow and processing
🔄 Model Deployment: Rolling out AI models safely requires sophisticated CI/CD

🤝 AI Enhances DevOps (Doesn't Replace It)

Rather than replacing DevOps engineers, AI is becoming a powerful tool in our toolkit:

🔍 Intelligent Monitoring: AI helps predict system failures before they happen
🛠️ Automated Remediation: Smart systems can fix common issues automatically
📈 Resource Optimization: AI optimizes cloud costs and performance
🔐 Security Enhancement: AI-powered threat detection and response

🎯 The Human Element Remains Critical

While AI can automate many tasks, DevOps engineers provide irreplaceable value:

🧠 Strategic Thinking: Designing architecture and making technology choices
🔧 Complex Problem Solving: Debugging unique issues and system design
👥 Cross-team Collaboration: Bridging technical and business requirements
📋 Compliance & Governance: Ensuring systems meet regulatory requirements

🌐 Growing Complexity Requires Expertise

As AI adoption accelerates, infrastructure becomes more complex:

🔀 Multi-cloud Strategies: Managing resources across different providers
⚓ Container Orchestration: Running AI workloads at scale
🔒 Security Challenges: Protecting sensitive AI models and data
📊 Observability Needs: Understanding performance of AI-driven systems

The 9-Stage DevOps Learning Journey 🗺️

Stage 1: Master the Fundamentals 💻

Foundation Skills Every DevOps Engineer Needs

Before diving into advanced tools, you need rock-solid fundamentals:

🐧 Linux/Unix Systems: Command line proficiency is essential
📜 Shell Scripting (Bash): Automate repetitive tasks efficiently
🔀 Version Control (Git): Collaborate effectively with development teams
🐍 Basic Programming: Python or Go for automation scripts
🌐 Networking Fundamentals: Understand how services communicate

💡 Pro Tip: Don't rush this stage. These skills form the foundation for everything else.

What You'll Build: Personal development environment, system monitoring scripts, automated backup solutions.

Stage 2: Infrastructure as Code 🏗️

Manage Infrastructure Through Code

Infrastructure as Code (IaC) transforms how we manage infrastructure:

⚙️ Terraform: Industry standard for multi-cloud infrastructure
🔧 Ansible: Configuration management and application deployment
☁️ CloudFormation: AWS-native infrastructure provisioning
✅ Infrastructure Testing: Validate changes before deployment

Real Impact: Companies achieve consistent, reproducible deployments while reducing manual configuration errors.

What You'll Build: Multi-environment infrastructure, automated web application stacks, infrastructure testing pipelines.

Stage 3: Containerization & Orchestration 📦

Package and Orchestrate Applications

Container technology has revolutionized application deployment:

🐳 Docker Fundamentals: Package applications consistently
🔗 Container Networking: Understand service communication
⚓ Kubernetes: Orchestrate containers at scale
📋 Helm Charts: Simplify Kubernetes application deployment
🔒 Container Security: Protect your containerized workloads

Why It Matters: Containers solve the "it works on my machine" problem and enable consistent deployments across environments.

What You'll Build: Microservices e-commerce platform, container CI/CD pipeline, production-ready Kubernetes cluster.

Stage 4: CI/CD Pipelines ⚡

Automate Your Deployment Process

Continuous Integration and Deployment revolutionize software delivery:

🚀 GitHub Actions: Automate workflows directly in your repository
🔄 Jenkins: Build complex, enterprise-grade pipelines
🦊 GitLab CI: Integrated DevOps platform
🎯 ArgoCD: GitOps-style deployments
🧪 Testing Automation: Integrate quality gates

Game Changer: Teams can deploy changes safely and frequently, with automatic rollback capabilities.

What You'll Build: Multi-stage CI/CD pipeline, GitOps deployment system, blue-green deployment strategy.

Stage 5: Cloud Platforms ☁️

Master Modern Cloud Infrastructure

Cloud expertise is essential in today's landscape:

🌐 AWS Fundamentals: Learn the most widely adopted cloud platform
🔷 Azure Services: Microsoft's comprehensive cloud ecosystem
🔵 Google Cloud Platform: Strong in data and AI services
🌍 Multi-Cloud Strategy: Many organizations use multiple providers
💰 Cost Optimization: Control and reduce cloud spending

Industry Reality: Most organizations have moved to cloud-first strategies, making these skills essential.

What You'll Build: Multi-cloud architecture, serverless application suite, cost optimization dashboard.

Stage 6: Monitoring & Observability 📊

Ensure System Reliability and Performance

Observability provides visibility into system behavior:

📈 Prometheus & Grafana: Industry-standard metrics and visualization
📋 ELK Stack: Centralized logging and analysis
🔍 Distributed Tracing: Track requests across microservices
⚡ APM Tools: Application performance monitoring
🎯 SLO/SLI Design: Define and measure service reliability

Critical Importance: You can't improve what you can't measure. Monitoring prevents small issues from becoming major outages.

What You'll Build: Complete observability stack, SLO monitoring dashboard, performance analysis tools.

Stage 7: Security & Compliance 🛡️

Integrate Security Throughout the Pipeline

Security must be built-in, not bolted-on:

🔐 DevSecOps Practices: Shift security left in the development process
🛡️ Container Security: Secure runtime and images
🔑 Secrets Management: Handle credentials safely
📋 Compliance Automation: Automate SOC2, GDPR requirements
🔍 Security Scanning: Integrate vulnerability detection

Modern Approach: Security teams collaborate with development from day one, rather than reviewing at the end.

What You'll Build: Secure CI/CD pipeline, zero-trust network, compliance automation systems.

Stage 8: Database Management 🗄️

Handle Data Persistence and Reliability

Data management remains critical across all applications:

🗃️ SQL & NoSQL: Master both relational and document databases
🤖 Database Automation: Automate deployments and migrations
💾 Backup Strategies: Ensure data recovery capabilities
⚡ Performance Tuning: Optimize database performance
☁️ Cloud Databases: Leverage managed database services

Universal Need: Every application needs data persistence, making these skills valuable across all projects.

What You'll Build: Database migration pipeline, multi-database architecture, database monitoring system.

Stage 9: Continuous Learning 🎓

Embrace Lifelong Growth

Technology evolves rapidly, making continuous learning essential:

🌍 Open Source Contribution: Build your reputation in the community
✍️ Technical Writing: Share knowledge and build authority
👥 Mentoring: Guide others and develop leadership skills
🎤 Conference Participation: Stay current with industry trends
🛠️ Side Projects: Experiment with new technologies

Long-term Success: The most successful DevOps engineers are those who adapt and grow with the technology landscape.

What You'll Build: Open source contributions, technical blog series, mentorship programs.

Practical Learning Approach 🎯

1. Hands-On Projects Beat Theory 🔨

Don't just read about tools, build with them:

Set up a complete development environment
Create infrastructure across multiple cloud providers
Build and deploy a real application end-to-end
Implement comprehensive monitoring and alerting

2. Learn in Public 📢

Document your journey and help others:

Write blog posts about your learnings and challenges
Share code and configurations on GitHub
Participate in DevOps communities and forums
Help others troubleshoot problems you've solved

3. Focus on Problem-Solving 🧩

DevOps is about solving business problems:

Understand why tools exist, not just how to use them
Practice troubleshooting and debugging systematically
Learn to communicate with both technical and business stakeholders
Think about reliability, scalability, and maintainability

4. Embrace AI as a Tool 🤖

Learn to work alongside AI rather than compete with it:

Use AI-powered tools to enhance your productivity
Understand how to deploy and manage AI workloads
Learn about MLOps practices and AI model lifecycle management
Focus on the strategic and creative aspects that AI can't replace

Industry Trends to Watch in 2025 🔮

🏗️ Platform Engineering Rise

Organizations are investing in internal developer platforms to improve developer experience and reduce cognitive load.

🔄 GitOps Adoption

Git-based deployment workflows are becoming the standard for managing infrastructure and applications.

🤖 AI/ML Integration & Infrastructure Demands

AI is transforming DevOps in multiple ways:

🧠 AI-Powered Tools: Intelligent monitoring, predictive scaling, and automated incident response
🚀 MLOps Emergence: New discipline combining ML and DevOps practices
⚡ GPU Infrastructure: Managing specialized hardware for AI workloads
📊 AI Model Pipelines: Deploying and updating AI models safely at scale

🌱 Sustainability Focus

Green DevOps practices are becoming important as organizations focus on reducing their environmental impact, especially with energy-intensive AI workloads.

🔒 Security-First Mindset

Security considerations are moving earlier in the development lifecycle, making DevSecOps skills increasingly valuable, particularly for protecting AI models and data.

Your Next Steps 🚶‍♂️

Starting your DevOps journey can feel overwhelming, but remember: every expert was once a beginner. Here's how to begin:

📚 Start with the Fundamentals: Master Linux, Git, and basic programming
⏰ Practice Consistently: Dedicate time each day to hands-on learning
👥 Join Communities: Connect with other learners and experienced practitioners
🔨 Build Projects: Apply your knowledge to real-world scenarios
🔍 Stay Curious: Technology evolves rapidly, embrace continuous learning
📝 Document Everything: Keep notes and share your learning journey

Interactive Learning Resources 🎮

While this roadmap provides the structure, hands-on practice is essential. For an interactive experience with curated resources, practice labs, and detailed guidance for each skill, check out the complete DevOps roadmap.

The interactive version includes:

📚 Curated learning resources for each skill
💻 Hands-on project ideas with difficulty levels
🎯 Skills assessment and progress tracking
🔗 Direct links to tutorials, documentation, and practice platforms
🏆 Achievement badges and learning milestones
💡 Real-world examples and use cases

Conclusion 🎉

The DevOps field offers tremendous opportunities for those willing to invest in learning and skill development. Even in an AI-driven world, infrastructure expertise becomes more valuable, not less. As AI applications proliferate, they all depend on the robust, scalable systems that DevOps engineers build and maintain.

With the right roadmap and consistent effort, you can build a rewarding career that combines technical challenges with meaningful business impact. The rise of AI doesn't diminish the importance of DevOps, it amplifies it.

Remember, the goal isn't to master everything at once. Focus on building a strong foundation, then gradually expand your expertise. The industry rewards competence, problem-solving ability, and continuous learning, all qualities that define successful DevOps engineers.

The journey may seem long, but every step builds upon the previous one. Start where you are, use what you have, and do what you can. Your future self will thank you for starting today.

Start your journey now. The DevOps community is welcoming and always ready to help newcomers succeed! 🌟

What's your current position on this roadmap? Share your DevOps learning journey in the comments below! 💬

The 10 Most Common DevOps Mistakes (And How to Avoid Them in 2025)

DevOps Daily — Mon, 21 Jul 2025 11:00:00 +0000

DevOps isn't just about shipping code faster, it's about doing it smarter, safer, and saner. But let's be real: even the best teams make mistakes. Some are harmless. Others take down production on a Friday afternoon (yes, that Friday deploy).

Here are 10 common DevOps mistakes in 2025, how to avoid them, and a few moments that might hit a little too close to home.

1. Treating Infrastructure as Code Like a One-Off Script

You wrote Terraform once, it worked, and now it lives untouched in a dusty repo folder. That's not IaC, that's tech debt.

Avoid it:

Version control your IaC.
Apply formatting and linting.
Test it with tools like terraform plan or terratest.

2. Not Enforcing Version Control on CI/CD Configs

Your pipeline files are changing, but without versioning, there's no easy way to debug regressions.

Avoid it:

Store all CI/CD config files (like GitHub Actions, GitLab CI, etc.) in version control.
Treat pipeline logic like any other critical code.

3. Poor Secrets Management

Hardcoding secrets in code or using .env files without encryption is a fast way to land on HN for the wrong reasons.

Avoid it:

Use Vault, Doppler, AWS Secrets Manager, or SOPS.
Rotate secrets regularly.

4. No Rollback Strategy

You deploy. Something breaks. And there's no plan B.

Avoid it:

Use blue-green or canary deployments.
Automate rollbacks on failure.
Always have a rollback.sh or previous image ready.

5. Ignoring Observability Until It's Too Late

Monitoring isn't just about uptime. You can't fix what you can't see.

Avoid it:

Add metrics, logs, and traces from day one.
Use tools like Prometheus, Grafana, and OpenTelemetry.

6. Too Many Tools, Not Enough Integration

Your stack has 25 tools. None of them talk to each other. And your alert fatigue is real.

Avoid it:

Consolidate tools where possible.
Favor tools that integrate well with your existing stack.

7. Manual Approval for Every Tiny Change

A typo fix shouldn't need a 3-person review and a Slack war.

Avoid it:

Set up clear policies: auto-approve safe changes, gate critical ones.
Use GitHub environments, OPA, or custom bots to help.

8. No Documentation = Single Point of Failure

"Ask Alex, they built it." Alex is on vacation.

Avoid it:

Write docs as you go.
Use tools like Backstage, Docusaurus, or just plain Markdown.
Encourage a culture of async knowledge sharing.

9. Skipping Tests for Infrastructure Changes

You test app code, but deploy infra changes directly to prod? Bold.

Avoid it:

Use staging or preview environments.
Test IaC with checkov, terratest, or kitchen.

10. Forgetting Security in Your Pipelines

If your pipeline can deploy to prod, attackers might be able to as well.

Avoid it:

Use least privilege for pipeline credentials.
Run security checks like trivy, semgrep, and snyk.

Final Thoughts

DevOps is a journey. These mistakes are all lessons learned the hard way by teams around the world, and probably you, if you've been around long enough.

Want to avoid these mistakes before they cost you time, sleep, or your weekend? We're building checklists, guides, and battle-tested content at DevOps Daily. Come hang out.

PS: Got a DevOps horror story or lesson to share? Drop it in the comments or tag us on Twitter.

What's Your Go-To Stack for Personal Projects in 2025?

DevOps Daily — Fri, 18 Jul 2025 17:07:49 +0000

When you're building a side project in 2025, what's your default stack these days?

Are you still loving the reliability of Laravel or Ruby on Rails, or have you fully embraced Next.js, Bun, or something even more bleeding edge? Maybe you're mixing in tools like Supabase, Neon, or HTMX?

Curious to hear:

What's your go-to stack for quick MVPs or weekend builds?
Do you keep it simple or try to mirror production setups?
What are you hosting it on?

Been thinking about this a lot while working on something for DevOps Daily and it made me wonder what others are using this year.

Drop your stack below, someone might discover their next favorite combo from your setup!

A Day in the Life of a DevOps Engineer

DevOps Daily — Fri, 11 Jul 2025 13:00:00 +0000

TLDR

This post follows a DevOps engineer through a typical workday. You'll see how they handle morning deployments, infrastructure scaling, security alerts, and emergency hotfixes. The story covers real scenarios with tools like Kubernetes, Docker, Jenkins, and monitoring systems while showing how DevOps work directly impacts business operations. If you're curious about what DevOps engineers actually do day-to-day, this realistic walkthrough will give you insights into the challenges, responsibilities, and satisfying moments of the role.

The Day at a Glance

05:47 AM ⚠️  PagerDuty Alert - API Response Time Critical
07:30 AM 🔧  Emergency Hotfix Deployment
11:30 AM 🔒  Security Incident Response
02:00 PM 📊  Performance Review & Feature Flag Deployment
06:00 PM 🔄  Kubernetes Cluster Maintenance
10:30 PM 🚨  Database Performance Emergency
12:00 AM 💤  Crisis Resolved, Systems Stable

The phone buzzes at 5:47 AM. Not the alarm - that's set for 6:00 AM. It's PagerDuty. The production API response time has crossed the 2-second threshold, and customers are starting to complain on social media.

Sound familiar? Welcome to Monday morning in the life of a DevOps engineer.

Rolling out of bed, laptop in hand, connecting to the VPN before the coffee even starts brewing. The monitoring dashboard shows a clear pattern: response times started climbing around 5:30 AM, right when the European market opened. The weekend's supposedly "minor" feature deployment is now causing 40% of API calls to timeout.

Incident Severity Assessment:
┌─────────────────────────────────────────────────────────────┐
│ 🔴 CRITICAL: 40% API timeout rate                           │
│ 📱 Social media complaints increasing                       │
│ 🌍 European market affected (peak hours)                    │
│ ⏰ US market opens in 3 hours                               │
│ 💰 Revenue impact: ~$2,000/minute                           │
└─────────────────────────────────────────────────────────────┘

This is why DevOps engineers sleep with their phones next to the bed.

Morning Fire Fighting

🔥 Crisis Mode Activated

The first instinct is to check the application logs. The ELK stack reveals the story immediately. The new payment processing feature is making synchronous calls to a third-party service, and those calls are taking 8-12 seconds to complete. When European users woke up and started making purchases, the connection pool got exhausted.

Payment Flow Issue:
User Request → API Gateway → Payment Service → Third-Party Provider
     ↓              ↓              ↓              ↓
   Fast         Fast         SLOW (8-12s)     TIMEOUT

Connection Pool: [████████████████████] 200/200 (FULL!)

A quick check shows 200 active connections - they've hit the maximum pool size. This needs an immediate fix while working on the root cause. The temporary solution is to scale up the payment service pods from 3 to 6, buying time to implement a proper fix.

Watching the metrics after applying the scaling change, response times start dropping within two minutes. The immediate crisis is over, but this is just a band-aid. The real fix needs to happen in the application code, requiring coordination with the development team.

Key Insight: Sometimes the best solution is the fastest solution. Scaling infrastructure horizontally bought time to implement a proper fix without losing customers.

💡 Pro Tip: Always have a rollback plan ready. In this case, the scaling approach was reversible if it didn't work, keeping options open during the crisis.

Deployment Coordination

📞 Emergency War Room

By 7:30 AM, the first video call of the day begins with the lead developer and product manager. They're discussing the hotfix strategy while pulling up the deployment pipeline in Jenkins.

"The payment timeout issue affects roughly 30% of our European customers," the product manager explains, checking analytics. "We need this fixed before the US market opens, or we're looking at significant revenue loss."

The developer has already pushed a fix to the staging branch - making the third-party payment calls asynchronous with proper error handling. The DevOps engineer's job is to get this through the pipeline safely.

Hotfix Deployment Pipeline:
┌─────────────────────────────────────────────────────────────┐
│ 1. Code Review    ✅ (expedited, focused review)            │
│ 2. Build & Test   ✅ (automated, 5 minutes)                 │
│ 3. Staging Deploy ✅ (integration tests passing)            │
│ 4. Smoke Tests    ✅ (payments working correctly)           │
│ 5. Production     🟡 (waiting for approval)                 │
└─────────────────────────────────────────────────────────────┘

The staging deployment goes smoothly. Integration tests pass, and end-to-end tests confirm that payments are now processing correctly with the new asynchronous flow. The green light for production deployment comes at 8:45 AM.

Key Insight: Production hotfixes require extra caution. Even with time pressure, proper testing in staging prevented a second incident.

🎯 Reality Check: In emergency situations, communication becomes even more critical. Clear status updates kept all stakeholders informed and aligned.

Infrastructure Scaling Challenges

With the payment crisis resolved, attention turns to a brewing infrastructure problem. The marketing team is launching a major campaign next week, expecting a 3x increase in traffic. The current Kubernetes cluster can barely handle normal peak loads.

Opening Terraform to review the current infrastructure setup reveals t3.medium instances that are cost-effective for normal operations but won't handle the expected load surge. A scaling strategy is needed that can handle the traffic spike without breaking the budget.

Current Infrastructure:
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster                                          │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐             │
│ │ t3.medium   │ │ t3.medium   │ │ t3.medium   │             │
│ │ Node 1      │ │ Node 2      │ │ Node 3      │             │
│ └─────────────┘ └─────────────┘ └─────────────┘             │
│                                                             │
│ Campaign Week (3x traffic) = 💥 OVERLOAD                    │
└─────────────────────────────────────────────────────────────┘

Solution: Pre-provisioned c5.xlarge nodes (scaled to 0 until needed)

The plan involves creating a new node group with c5.xlarge instances, pre-created but kept at zero capacity until the campaign starts. This way, they can scale up quickly when needed and scale down immediately after to control costs.

Key Insight: Planning for predictable traffic spikes is cheaper than dealing with unexpected outages. Pre-provisioning resources that can be quickly activated saves both money and stress.

Security Alert Response

At 11:30 AM, the security monitoring tool flags something suspicious. The intrusion detection system shows unusual network traffic patterns from one of the application servers. Security incidents can escalate quickly, so immediate attention is required.

Initial investigation shows someone is trying to access the MySQL database directly from an external IP. A quick check of security groups and firewall rules shows they look correct - database access should only be allowed from application servers within the VPC. But the logs show connection attempts from a completely different IP range.

Digging deeper into the application logs reveals the issue. A developer accidentally committed database credentials to a public GitHub repository three days ago. The credentials were scraped by automated tools and are now being used for unauthorized access attempts.

Security Incident Timeline:
Day 1: Dev commits credentials → GitHub (public repo)
Day 2: Automated scrapers find credentials
Day 3: Credentials posted on dark web forums
Day 4: Unauthorized access attempts begin ← WE ARE HERE

Threat Actor → Internet → Firewall → Database (attempting access)
                             ↓
                       ⚠️  BLOCKED (but trying)

The immediate response is clear: rotate database credentials immediately and update the Kubernetes secret. The security incident is contained, but this requires a longer-term solution - implementing automated secret scanning in the CI/CD pipeline and scheduling security training for the development team.

Key Insight: Security incidents are rarely just technical problems. They're usually process problems that require both immediate fixes and long-term prevention strategies.

Monitoring and Alerting Improvements

After lunch, focus shifts to improving the monitoring setup. The morning's payment issue could have been caught earlier with better alerting. Opening Prometheus to review the current metrics collection shows it only monitors basic metrics like CPU and memory usage.

Working with the developer to add business-specific metrics that would have caught the payment timeout issue earlier becomes the priority. Custom metrics for payment processing duration, active connections, and success/failure rates are implemented.

With these metrics in place, new alerting rules are created that would have triggered within minutes of the morning's incident, giving time to respond before customers were affected.

Afternoon Deployment Pipeline

🚀 Major Feature Release

The afternoon brings a scheduled deployment of the new user dashboard feature. This is a major feature that's been in development for six weeks, and the product team is eager to get it in front of users.

The staging environment looks good, but something concerning appears in the performance tests. The new dashboard is making 47 database queries per page load. With the expected traffic increase from the marketing campaign, this could cause serious performance problems.

Database Query Analysis:
┌─────────────────────────────────────────────────────────────┐
│ Current Dashboard: 3 queries per page                       │
│ New Dashboard: 47 queries per page                          │
│                                                             │
│ Expected Traffic: 10,000 concurrent users                   │
│ Query Load: 470,000 queries/second                          │
│ Database Capacity: 50,000 queries/second                    │
│                                                             │
│ Result: 💥 DATABASE MELTDOWN                                │
└─────────────────────────────────────────────────────────────┘

An emergency meeting with the development team follows. The conversation is tense - the marketing campaign is already scheduled, and delaying the dashboard feature would mean missing the promotional opportunity.

The Dilemma:

✅ Ship on time → Happy marketing team, potential system failure
❌ Delay feature → Disappointed stakeholders, stable system
🤔 Find middle ground → ???

"We can't deploy this as-is," becomes the message, showing the performance metrics. "Each page load is hitting the database 47 times. With 10,000 concurrent users, that's 470,000 database queries per second. Our database will fall over."

The lead developer looks at the query analysis. "Most of these are N+1 queries. We can fix the worst ones with some eager loading, but it'll take at least two days to properly optimize."

A compromise is proposed: deploy the feature with a feature flag, initially enabled for only 10% of users. This gives real-world performance data while limiting the impact on the infrastructure.

Feature Flag Strategy:
┌─────────────────────────────────────────────────────────────┐
│ Incoming Users: 10,000/second                               │
│                                                             │
│ 90% → Old Dashboard (stable, fast)                          │
│ 10% → New Dashboard (testing, monitored)                    │
│                                                             │
│ Database Load: Manageable vs. Catastrophic                  │
└─────────────────────────────────────────────────────────────┘

The deployment goes ahead with the feature flag in place. Database performance is monitored closely as the feature rolls out to the limited user group. The impact is manageable at 10% traffic, but the metrics confirm concerns about a full rollout.

Key Insight: Feature flags aren't just for A/B testing. They're a powerful risk management tool that lets you test production performance without betting the entire infrastructure.

🔄 DevOps Wisdom: The best compromise is often a gradual rollout. It satisfies business needs while protecting system stability.

Evening Infrastructure Maintenance

As the day winds down, planned maintenance tasks need attention. The Kubernetes cluster needs a version upgrade, and several security patches need to be applied to the worker nodes.

The upgrade process requires careful coordination to avoid downtime. Nodes are drained one by one, system updates are applied, kubelet is restarted with the new version, and then the node is uncordoned back into service.

The upgrade process takes about 90 minutes, but it goes smoothly. Application metrics are monitored throughout the process - response times stay normal, and no alerts fire.

Late Night Emergency

🌙 10:30 PM - Not Again...

Just when getting ready for bed at 10:30 PM, the phone buzzes again. This time it's a critical alert: the main application database is reporting high CPU usage and slow query performance. The European overnight batch processing jobs are running much longer than usual.

Every DevOps engineer knows this feeling - the dreaded "just one more alert" before bed.

Connecting to the database server immediately reveals the problem. One of the batch jobs is running a query that's been executing for 3 hours. The query is scanning a table with 50 million rows without using an index.

Database Performance Crisis:
┌─────────────────────────────────────────────────────────────┐
│ Query: SELECT * FROM user_activities WHERE...               │
│ Status: Running for 3 hours ⏱️                              │
│ Rows Scanned: 50,000,000 (NO INDEX!)                        │
│ CPU Usage: ████████████████████████████████████ 95%         │
│ Other Queries: ⏳ WAITING... WAITING... WAITING...          │
└─────────────────────────────────────────────────────────────┘

The batch job developer probably tested with a small dataset and didn't realize the performance implications. A tough decision emerges: kill the long-running query to restore database performance, meaning the batch job will need to restart from the beginning, or let it finish but risk affecting the morning's application performance.

The Midnight Decision Matrix:

Option 1: Kill Query + Create Index
├─ Pros: Immediate relief, proper fix
├─ Cons: Batch job restarts (3 hours lost)
└─ Risk: Low

Option 2: Let Query Finish
├─ Pros: Batch job completes
├─ Cons: Database stays slow
└─ Risk: High (morning traffic impact)

The choice is made to kill the query and create the missing database index. The index creation takes 45 minutes on the large table, but once it's complete, the batch job can restart and finish in just 20 minutes instead of hours.

Key Insight: Sometimes you have to make tough decisions with incomplete information. The ability to quickly assess risk and choose the least harmful option is crucial in DevOps.

⚡ Late Night Wisdom: The best decisions aren't always the easiest ones. Protecting tomorrow's users was worth the short-term pain of restarting the batch job.

Reflection and Planning

🌅 Midnight - Systems Stable, Lessons Learned

By midnight, the laptop finally closes. The day started with a production crisis, included a security incident, featured a challenging deployment decision, and ended with a database performance emergency. Each situation required different skills: quick problem-solving, technical analysis, team coordination, and risk assessment.

Daily Impact Summary:
┌─────────────────────────────────────────────────────────────┐
│ 🔧 Issues Resolved: 4 critical, 2 medium priority           │
│ 👥 Customers Affected: Minimal (thanks to quick response)   │
│ 💰 Revenue Protected: ~$50,000 (prevented outages)          │
│ 🛠️ Systems Improved: 3 (monitoring, security, indexing)     │
│ 📈 Infrastructure: Scaled and optimized                     │
└─────────────────────────────────────────────────────────────┘

Tomorrow will bring new challenges. The marketing campaign is getting closer, and the infrastructure scaling plan needs finalization. The dashboard feature needs performance optimization before it can be fully rolled out. The development team needs security training to prevent credential leaks. The monitoring system needs those new business metrics.

But tonight, millions of users were able to make purchases, view their dashboards, and access the application without interruption. The infrastructure held up under pressure, the team collaborated effectively during crises, and the systems are more robust than they were this morning.

Tomorrow's Action Items:

✅ Finalize campaign infrastructure scaling
✅ Optimize dashboard database queries
✅ Implement automated secret scanning
✅ Deploy enhanced monitoring metrics
✅ Schedule security training session

This is the reality of DevOps work - part firefighting, part planning, part collaboration, and part continuous improvement. It's demanding and sometimes stressful, but it's also rewarding to know that your work directly enables the business to serve its customers.

The phone is on silent for the next six hours, but somewhere, monitoring systems are keeping watch, automated processes are handling routine tasks, and the infrastructure is quietly supporting thousands of users around the world. That's the real success of DevOps - building systems that work reliably, even when you're not watching.

The Human Side of DevOps

Being a DevOps engineer means being part detective, part architect, part diplomat, and part firefighter. Every day brings new challenges, but also new opportunities to make systems better, faster, and more reliable. The work never ends, but neither does the satisfaction of building technology that makes a real difference in people's lives.

The morning payment issue wasn't just about fixing code - it was about understanding the business impact of technical decisions. When European customers couldn't complete their purchases, it affected real people trying to buy gifts, pay bills, or run their businesses. The quick response prevented thousands of failed transactions and potential customer churn.

The security incident required more than just technical fixes. It highlighted the need for better developer education and process improvements. The conversation with the development team wasn't about blame - it was about learning and preventing similar issues in the future.

The deployment decision for the dashboard feature showcased the constant balance between business needs and technical constraints. The marketing campaign couldn't be delayed, but releasing a feature that would crash the database wasn't an option. The feature flag solution satisfied both requirements while providing valuable data for future improvements.

The Broader Impact

DevOps work extends far beyond keeping servers running. It's about enabling the entire organization to move faster and more reliably. The monitoring improvements implemented today will prevent future incidents. The infrastructure scaling plan will support business growth. The security training will protect customer data.

Each technical decision has ripple effects throughout the organization. The choice to scale up the payment service immediately instead of waiting for a code fix meant that the customer service team didn't get flooded with complaint calls. The decision to implement feature flags for the dashboard deployment gave the product team valuable usage data while protecting system stability.

The database performance fix at midnight wasn't just about query optimization - it was about ensuring that the morning's business reports would be ready on time, that the analytics team could access their data, and that the automated systems could process customer orders without delay.

Skills Beyond Technology

While technical skills are essential, DevOps engineering requires much more. Communication skills are crucial for coordinating with development teams, explaining technical issues to business stakeholders, and writing clear documentation for on-call procedures.

Problem-solving skills go beyond debugging code. They involve understanding complex systems, identifying root causes of issues, and designing solutions that prevent future problems. The ability to work under pressure while maintaining clear thinking is essential when production systems are down and customers are affected.

Risk assessment becomes second nature - every change, every deployment, every infrastructure modification needs to be evaluated for potential impact. The ability to make quick decisions with incomplete information is valuable when incidents are unfolding and time is critical.

The Satisfaction of Reliability

The most rewarding aspect of DevOps work isn't the dramatic incident responses or the complex technical solutions. It's the quiet satisfaction of building systems that work consistently, day after day, serving users around the world without interruption.

When a deployment goes smoothly, when monitoring catches an issue before it affects users, when an infrastructure upgrade happens without downtime - these moments of seamless operation represent the true success of DevOps practices.

The tools and technologies will continue to evolve, but the core mission remains the same: bridge the gap between development and operations, automate repetitive tasks, monitor everything that matters, and respond quickly when things go wrong. It's challenging work, but for those who enjoy solving complex problems and working with cutting-edge technology, there's nothing quite like it.

The best DevOps engineers are those who can see the bigger picture - understanding how their technical decisions impact users, businesses, and teams. They're the ones who can remain calm during crises, think strategically about infrastructure improvements, and communicate effectively with both technical and non-technical stakeholders.

This is what a day in the life of a DevOps engineer really looks like - not just managing servers and writing scripts, but being a crucial part of the technology ecosystem that powers modern business operations.

Key Takeaways for Aspiring DevOps Engineers

🎯 Essential Skills Demonstrated Today:

Crisis Management: Quick thinking under pressure while maintaining system stability
Risk Assessment: Evaluating trade-offs between speed and reliability
Cross-team Communication: Coordinating with developers, product managers, and business stakeholders
Technical Versatility: From Kubernetes to databases to security incidents
Business Impact Awareness: Understanding how technical decisions affect revenue and customers

🛠️ Core Tools in Action:

Monitoring: ELK Stack, Prometheus, PagerDuty
Infrastructure: Kubernetes, Docker, Terraform, AWS
CI/CD: Jenkins, automated testing pipelines
Security: Intrusion detection, credential management
Databases: MySQL, query optimization, indexing

📚 Want to Learn More?

If this day-in-the-life resonates with you, here are some next steps:

🚀 Getting Started: Practice with containerization (Docker), learn Kubernetes basics, and get comfortable with Linux command line

🔍 Dive Deeper: Set up monitoring in a personal project, practice incident response scenarios, learn infrastructure as code

💼 Career Path: Consider starting as a systems administrator, junior DevOps engineer, or SRE to build foundational skills

The Reality Check: DevOps isn't just about tools and automation. It's about building reliable systems that let businesses focus on serving their customers. Every alert, every deployment, every optimization contributes to that mission.

The most rewarding part? Knowing that somewhere in the world, users are seamlessly making purchases, accessing services, and getting value from applications - all because the infrastructure you built and maintain is working exactly as it should.

Go over the following DevOps Roadmap to see how you can build your skills and career in this exciting field.

That's the real satisfaction of DevOps work - building the invisible foundation that makes everything else possible.

Thinking of Launching a SaaS in 2025? Here's My #1 Piece of Advice

DevOps Daily — Thu, 10 Jul 2025 09:22:57 +0000

If I had to start over today, I'd keep it painfully simple:

Build stuff but also focus on marketing

Most developers (myself included) fall into the trap of tweaking features, polishing deploy scripts, and 'optimizing infra' way too early.

But in the early days, the real work is figuring out what people actually care about, and how fast you can get paid to solve it.

Here's what I wish someone told me earlier:

Launch before you're ready.
Start charging as soon as it works.
One Droplet, managed Postgres, maybe Redis, don't over-engineer.
Use boring tech you can support solo.
Spend more time on writing and talking than coding.

Anyway, if you're building something in 2025, what are you doing differently?

Let's trade notes.

(P.S. I recently redesigned DevOps Daily so I have to start doing marketing 🤪)