Forem: DevOps Start

Essential kubectl Commands Cheat Sheet

DevOps Start — Tue, 14 Apr 2026 15:36:24 +0000

Stop memorizing every flag! I've put together a handy kubectl cheat sheet for managing pods, deployments, and debugging. Originally published on devopsstart.com.

Pod Management

Command	Description
`kubectl get pods`	List all pods in current namespace
`kubectl get pods -A`	List pods across all namespaces
`kubectl describe pod <name>`	Show detailed pod information
`kubectl delete pod <name>`	Delete a specific pod
`kubectl logs <pod>`	View pod logs
`kubectl logs <pod> -f`	Stream pod logs
`kubectl exec -it <pod> -- sh`	Open shell in pod

Deployments

Command	Description
`kubectl get deployments`	List all deployments
`kubectl scale deploy <name> --replicas=3`	Scale a deployment
`kubectl rollout status deploy/<name>`	Check rollout status
`kubectl rollout undo deploy/<name>`	Rollback a deployment
`kubectl set image deploy/<name> <container>=<image>`	Update container image

Services and Networking

Command	Description
`kubectl get svc`	List all services
`kubectl get ingress`	List all ingress resources
`kubectl port-forward svc/<name> 8080:80`	Forward local port to service
`kubectl get endpoints`	List service endpoints

Debugging

Command	Description
`kubectl get events --sort-by=.metadata.creationTimestamp`	View cluster events
`kubectl top pods`	Show pod resource usage
`kubectl top nodes`	Show node resource usage
`kubectl logs <pod> --previous`	View logs from crashed container
`kubectl describe node <name>`	Check node conditions

Context and Config

Command	Description
`kubectl config get-contexts`	List all contexts
`kubectl config use-context <name>`	Switch context
`kubectl config current-context`	Show current context
`kubectl get ns`	List namespaces
`kubectl config set-context --current --namespace=<ns>`	Set default namespace

Debug Kubernetes CrashLoopBackOff in 30 Seconds

DevOps Start — Tue, 14 Apr 2026 15:31:20 +0000

Struggling with a pod stuck in CrashLoopBackOff? This quick guide, originally published on devopsstart.com, shows you the exact commands to diagnose the root cause in seconds.

The Problem

Your pod is stuck in CrashLoopBackOff and you need to find out why — fast.

The Solution

kubectl logs <pod-name> --previous

The --previous flag shows logs from the last crashed container instance. This is the single most useful flag for debugging crash loops.

Combine with describe for the full picture

kubectl describe pod <pod-name> | grep -A 5 "Last State"

This shows the exit code and reason for the last termination:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

Common Exit Codes

Code	Meaning	Fix
1	Application error	Check app logs
137	OOMKilled	Increase memory limits
139	Segfault	Check binary compatibility
143	SIGTERM	Graceful shutdown issue

Why It Works

Kubernetes keeps logs from the previous container instance even after it crashes. Without --previous, you'd only see logs from the current (possibly empty) instance that hasn't had time to produce output before crashing again.

Rapid Rollback: `kubectl set image` for Urgent Fixes

DevOps Start — Tue, 14 Apr 2026 15:26:17 +0000

Originally published on devopsstart.com. When production breaks, every second counts—here is how to use kubectl set image for a precise and rapid rollback.

Introduction

You've just deployed a new container image to production, and almost immediately, monitoring alerts start screaming. Latency is spiking, error rates are through the roof, and your customers are experiencing service degradation. In these high-pressure moments, a fast, reliable rollback mechanism is critical. While Kubernetes offers robust rollout and rollback capabilities via kubectl rollout undo, there are specific scenarios where kubectl set image can provide a quicker, more direct path to recovery, especially when you know exactly which image version you need to revert to.

This tip focuses on leveraging kubectl set image for urgent rollbacks. You'll learn when this command is most effective, how to accurately identify the correct previous image tag, and how to execute the command to quickly stabilize your application.

Understanding `kubectl set image`

The kubectl set image command is primarily designed to atomically update the image of one or more specific containers within a Kubernetes resource. It typically targets Deployments, StatefulSets, DaemonSets, or ReplicationControllers. When executed, it modifies the resource's Pod template to point to the new image tag, which then triggers a new rolling update.

While kubectl set image is frequently used for forward deployments (e.g., updating v1.1.9 to v1.2.0), its direct nature makes it exceptionally well-suited for rapid rollbacks. When you specify a previous, stable image, Kubernetes initiates a new rollout toward that desired state. This behavior differentiates it from kubectl rollout undo, which inherently steps back through the deployment's recorded history, revision by revision.

Here’s a common example of how kubectl set image is used to update an image:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.2.0 -n production

Expected output:

deployment.apps/my-app image updated

This command updates my-container within the my-app deployment in the production namespace to use the v1.2.0 image from my-registry.

The Urgent Rollback Scenario

Consider this scenario: your my-app:v1.2.0 release introduced a critical bug that bypassed your staging environment checks. You pushed it to production an hour ago, and now, critical alerts are firing, indicating significant application failures. You need to revert to the last known good image, let's say my-app:v1.1.9, immediately.

Why might kubectl set image be preferred over kubectl rollout undo in such a situation?

Directness and Precision: If you know the exact, stable image tag to which you need to revert, kubectl set image offers an explicit and precise command. This avoids ambiguity and ensures you land on the intended stable state directly.
Bypassing Unhealthy Revisions: If multiple faulty deployments occurred after your last stable one (e.g., you tried v1.2.0, then v1.2.1-hotfix, both failed), kubectl rollout undo would sequentially step back through these potentially problematic revisions. kubectl set image allows you to jump directly to the known good v1.1.9 without traversing the unstable intermediate states.
Forced Redeploy (Edge Cases): In rare cases, even if an image tag is theoretically the same, you might want to force Kubernetes to re-pull container images and redeploy pods due to local caching issues or other inconsistencies. Re-setting the image explicitly with kubectl set image can achieve this, ensuring fresh pods are created. For more on debugging common Kubernetes issues, refer to our article on Troubleshooting CrashLoopBackOff in Kubernetes.

Identifying the Previous Image Tag

The critical first step for a kubectl set image rollback is accurately identifying the last known good image tag. You can achieve this by inspecting your deployment's revision history:

Check Rollout History:
This command provides a concise summary of your deployment's revision history, showing the changes made at each step.
```
kubectl rollout history deployment/my-app -n production
```


shell

    **Expected output:**


```bash
    deployment.apps/my-app 
    REVISION  CHANGE-CAUSE
    1         <none>
    2         my-container: my-registry/my-app:v1.1.8
    3         my-container: my-registry/my-app:v1.1.9
    4         my-container: my-registry/my-app:v1.2.0

From this output, if `v1.2.0` (revision 4) is currently causing issues, then `v1.1.9` (revision 3) is your immediate target for rollback. Note that `CHANGE-CAUSE` may also contain details if `--record` was used during deployment.

Describe a Specific Revision (Optional Verification):
To be absolutely certain about the container images used in a particular revision, you can describe it in detail. This is a good verification step before initiating a rollback.
```
kubectl rollout history deployment/my-app --revision=3 -n production
```


shell

    **Expected (truncated) output:**


```bash
    deployment.apps/my-app with revision 3
    Pod Template:
      Labels:       app=my-app
                    pod-template-hash=54c9c76...
      Containers:
        my-container:
          Image:        my-registry/my-app:v1.1.9
          Port:         8080/TCP
          Environment:  <none>
    ...

This confirms that `my-registry/my-app:v1.1.9` was indeed the image used for revision 3, making it a reliable rollback target.

Executing the `kubectl set image` Rollback

Once you have identified the precise desired image tag (e.g., my-registry/my-app:v1.1.9 in our example), executing the rollback is straightforward and immediate:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.1.9 -n production

Expected output:

deployment.apps/my-app image updated

Upon execution, Kubernetes will immediately initiate a new rolling update. It will begin replacing the currently failing v1.2.0 pods with new ones running the specified stable v1.1.9 image.

You can monitor the progress of this new rollout using the following command:

kubectl rollout status deployment/my-app -n production

Expected output during rollout:

Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out

Once the rollout is complete, your application should be consistently running the stable v1.1.9 image, and your monitoring alerts should ideally begin to subside as service is restored.

Important Considerations

Rollback Strategy Impact: This kubectl set image method performs a rolling update. It's crucial that your application is designed to handle a brief period where both the old (problematic) and new (stable) versions of pods are running concurrently. This typically means ensuring backward and forward compatibility for APIs and data schemas.
Image Immutability: Always strive to use immutable image tags (e.g., v1.1.9, v1.2.0, sha256:abcdef...) rather than mutable tags like latest. Immutable tags guarantee that a specific tag always refers to the exact same image content, which is fundamental for reliable and reproducible rollbacks.
Auditing and History: Using kubectl set image creates a new revision in the deployment's history. This automatically ensures that your rollback action is recorded, providing a clear audit trail of changes made to your deployment.
Stateful Workloads: For StatefulSets, exercising caution when changing image versions is paramount. If a new image version introduces changes that affect persistent storage or state, a simple image rollback might not fully resolve database schema migrations or data portability issues. Always understand the data implications.

Conclusion

When a problematic image release throws production into disarray, reaction time is paramount. While kubectl rollout undo is a valuable tool, kubectl set image provides a direct, efficient, and precise alternative for reverting to a specific, known-good image. This capability can significantly reduce Mean Time To Recovery (MTTR) by allowing you to bypass potentially multiple failing revisions and jump straight to stability. By understanding your deployment history and precisely targeting the last stable

How to Set Up Argo CD GitOps for Kubernetes Automation

DevOps Start — Tue, 14 Apr 2026 15:21:13 +0000

Stop relying on manual kubectl applies and start treating your cluster as code. This comprehensive guide, originally published on devopsstart.com, walks you through setting up Argo CD for true GitOps automation.

Introduction

If you are still running kubectl apply -f manifests/ from your local machine or a Jenkins pipeline, you are operating in a "Push" model. In this model, your CI tool needs high-privileged credentials to your cluster, and you have no guarantee that what is actually running in production matches what is in your Git repository. One rogue developer running a manual kubectl edit can create a "configuration drift" that haunts you for months.

This is where GitOps comes in. GitOps is a paradigm where Git is the single source of truth for your infrastructure and application state. Instead of pushing changes to the cluster, a controller inside the cluster constantly monitors your Git repo and "pulls" the state to match.

In this tutorial (Part 1 of our series), we will move from imperative deployments to declarative continuous delivery using Argo CD v2.11.0. You'll learn how to install Argo CD, connect your repositories, handle configuration drift, and scale your deployments using ApplicationSets. By the end, you'll have a production-ready GitOps engine that ensures your cluster is always in the desired state. For a deeper dive into how this compares to other tools, check out our guide on /blog/argo-cd-vs-flux-a-guide-for-multi-cluster-gitops.

Prerequisites

Before we start, you need a working Kubernetes cluster. This can be a managed service like EKS, GKE, or AKS, or a local setup like Kind or Minikube. For this tutorial, we assume you are using Kubernetes v1.30 or newer.

You will need the following tools installed on your local workstation:

kubectl v1.30+: The standard Kubernetes CLI. Ensure your context is set to the correct cluster.
Git v2.40+: Required for managing the manifests that Argo CD will track.
A GitHub or GitLab account: You need a repository to store your Kubernetes YAML manifests.
Basic YAML knowledge: You should know how to write a basic Deployment and Service manifest. If you are totally new to this, refer to our /blog/kubernetes-for-beginners-deploy-your-first-application.

You don't need to install the Argo CD CLI for the basic setup, as we will use the Web UI and kubectl for most operations, but having it installed is helpful for advanced automation.

Overview

The goal of this tutorial is to build a fully automated deployment pipeline where a Git commit is the only trigger needed to update your application. We aren't just deploying a "Hello World" app; we are building a scalable architecture.

Here is the architecture we will implement:

The Manifest Repo: A dedicated Git repository containing the desired state of your cluster (YAML files).
Argo CD Controller: Installed in the argocd namespace, acting as the GitOps operator.
The Application CRD: A custom resource that tells Argo CD: "Watch this folder in Git and make sure it exists in this namespace in the cluster."
Automated Sync: A policy that automatically corrects any manual changes (drift) made to the cluster.
ApplicationSets: A template-based approach to deploy the same application across multiple namespaces (e.g., dev, staging, prod) without duplicating YAML files.

By moving to this "Pull" model, you eliminate the need to store Kubeconfigs in your CI tool (like GitHub Actions or GitLab CI), which reduces the attack surface of your infrastructure by removing high-privileged secrets from external runners.

Step 1: Installing Argo CD

Argo CD is installed as a set of deployments and services within your cluster. We will use the official manifests provided by the Argo project.

First, create a dedicated namespace for Argo CD to keep the installation isolated.

kubectl create namespace argocd

Now, apply the installation manifest. We will use the stable release manifest for v2.11.0.

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml

Wait for all pods to reach the Running state. You can monitor the progress with this command:

kubectl get pods -n argocd

The output should look like this:

NAME                                                 READY   STATUS    RESTARTS   AGE
argocd-server-7d5f8f8f5-abc12                       1/1     Running   0          2m
argocd-repo-server-5f4d7e9-def34                     1/1     Running   0          2m
argocd-application-controller-f7e8d9-ghi56           1/1     Running   0          2m
argocd-redis-7c8b9a0-jkl78                           1/1     Running   0          2m
argocd-notifications-controller-h9i0j1-mno90         1/1     Running   0          2m

If any pods stay in Pending or CrashLoopBackOff, you can diagnose the issue using our guide on /troubleshooting/crashloopbackoff-kubernetes.

Step 2: Initial Access and Authentication

By default, the Argo CD API server is not exposed to the public internet. For this tutorial, we will use port-forwarding to access the UI.

Start the port-forward in a separate terminal window:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Now, open your browser and go to https://localhost:8080. You will see a login screen. The default username is admin. The password, however, is automatically generated and stored in a Kubernetes secret.

Run the following command to retrieve the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

The output will be a plain-text string, for example: xYz123AbC456DefG. Copy this password and use it to log in to the UI.

Once you log in, change the admin password under the "User Management" settings. For production environments, avoid using the initial admin account and instead integrate with an OIDC provider like Okta or GitHub. You can find more details in the official Argo CD documentation.

Step 3: Connecting Your Git Repository

Argo CD needs permission to read your manifests. If your repository is public, you can just provide the URL. If it is private, you need to provide SSH keys or HTTPS credentials.

Let's assume you have a private GitHub repository located at git@github.com:your-org/gitops-manifests.git.

Log in to the Argo CD UI.
Go to Settings $\rightarrow$ Repositories.
Click Connect Repo.
Select via SSH.
Enter the Repository URL: git@github.com:your-org/gitops-manifests.git.
Paste your private SSH key (the one that has read access to the repo).
Click Connect.

If the connection is successful, the status will change to Successful. If you see Failed, ensure your SSH key is correct and that the Argo CD pod has outbound network access to GitHub.

Step 4: Creating Your First GitOps Application

In Argo CD, an "Application" is a Custom Resource (CRD) that defines the link between a source (Git) and a destination (Cluster).

We will create a simple application that deploys a guestbook app. First, ensure your Git repo has a folder named guestbook containing a deployment.yaml and a service.yaml.

Using YAML is the "GitOps way" because you can store the Application definition itself in Git (the App-of-Apps pattern). Save the following as guestbook-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook-gitops
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'git@github.com:your-org/gitops-manifests.git'
    targetRevision: HEAD
    path: guestbook
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: guestbook-demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Component Breakdown

source: This tells Argo CD where to look. targetRevision: HEAD tracks the latest commit on the default branch.
destination: https://kubernetes.default.svc refers to the cluster where Argo CD is currently installed.
syncPolicy:
- automated: Argo CD will automatically apply changes from Git to the cluster.
- prune: true: If you delete a file from Git, Argo CD will delete the corresponding resource from Kubernetes.
- selfHeal: true: If someone manually edits a resource in the cluster, Argo CD will instantly overwrite it with the Git version.
- CreateNamespace=true: Ensures the guestbook-demo namespace is created if it doesn't exist.

Apply this manifest:

kubectl apply -f guestbook-app.yaml

Now, go to the Argo CD UI. You will see the guestbook-gitops application. Initially, it will be "OutOfSync" while it calculates the difference, then it will transition to "Synced" and "Healthy" as it creates the pods and services.

Step 5: Handling Configuration Drift

Configuration drift occurs when the actual state of the cluster deviates from the desired state defined in Git. This usually happens when a developer uses kubectl edit to fix a production bug quickly but forgets to update the Git repo.

Let's simulate this. We will manually scale our deployment to 5 replicas using the CLI.

kubectl scale deployment guestbook --replicas=5 -n guestbook-demo

If you check the Argo CD UI now, the application status has changed from Synced to OutOfSync. The UI will highlight the exact difference in yellow: "Desired: 3 replicas, Actual: 5 replicas."

Because we enabled selfHeal: true, you won't have to do anything. Within a few seconds, Argo CD will detect the drift and automatically scale the deployment back down to 3 replicas to match Git.

If selfHeal were disabled, the application would stay OutOfSync. You would then have two choices:

Sync: Click the "Sync" button in the UI to force the cluster to match Git.
Update Git: Change the replica count in your Git repo to 5, commit, and push. Argo CD would then see the new desired state and update the cluster.

This mechanism ensures that your Git history is an audit log of every change ever made to your environment.

Step 6: Scaling with ApplicationSets

Creating one Application manifest for one app is easy. But if you have 50 microservices across dev, staging, and prod clusters, creating 150 Application manifests is a maintenance nightmare.

ApplicationSets allow you to use a template to generate multiple Applications automatically. They use "generators" to discover targets.

Let's use a List Generator to deploy the same guestbook app into three different namespaces. Save this as guestbook-appset.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-environments
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            namespace: guestbook-dev
          - cluster: engineering-staging
            namespace: guestbook-staging
          - cluster: engineering-prod
            namespace: guestbook-prod
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: 'git@github.com:your-org/gitops-manifests.git'
        targetRevision: HEAD
        path: guestbook
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Apply the ApplicationSet:

kubectl apply -f guestbook-appset.yaml

Argo CD iterates through the list and creates three separate Application resources: engineering-dev-guestbook, engineering-staging-guestbook, and engineering-prod-guestbook.

If you need to add a new environment (e.g., qa), you simply add one line to the elements list in the ApplicationSet and commit. For teams managing huge fleets of clusters, the Cluster Generator is even more powerful; it can automatically detect every cluster registered in Argo CD and deploy a "base" set of tools to all of them without any manual listing.

Step 7: Integrating the Full CI/CD Pipeline

Now that the "CD" (Continuous Delivery) part is handled by Argo CD, how does the "CI" (Continuous Integration) part fit in?

A common mistake is letting the CI tool (GitHub Actions, Jenkins, CircleCI) call kubectl apply. This breaks the GitOps model. Instead, the CI tool should only be responsible for updating the manifest repository.

Here is the professional workflow:

Developer Pushes Code: A developer pushes a change to the application source code.
CI Pipeline Runs: GitHub Actions triggers a build, runs tests, and builds a Docker image.
Image Push: The CI tool pushes the image to a registry (e.g., Amazon ECR) with a unique tag (the Git SHA).
Manifest Update: The CI tool clones the gitops-manifests repo and updates the image tag in the deployment.yaml file.
Git Commit: The CI tool commits and pushes the change back to the manifest repo.
Argo CD Pull: Argo CD detects the commit in the manifest repo and pulls the change into the cluster.

Example GitHub Action snippet for manifest update

- name: Update Kubernetes image tag
  run: |
    git clone https://x-token-auth:${{ secrets.GITOPS_TOKEN }}@github.com/your-org/gitops-manifests.git
    cd gitops-manifests
    sed -i "s|image: my-app:.*|image: my-app:${{ github.sha }}|g" guestbook/deployment.yaml
    git config user.name "GitHub Action"
    git config user.email "action@github.com"
    git add .
    git commit -m "Update guestbook image to ${{ github.sha }}"
    git push

This separation of concerns is critical. The CI tool has no access to the cluster; it only has access to a Git repository. If your CI tool is compromised, the attacker cannot delete your production pods; they can only propose changes to Git, which can be blocked by a Pull Request review.

Troubleshooting

1. Application stuck in "Progressing" status

If your application is "Synced" but stays in "Progressing", the pods are likely failing to start. Argo CD is waiting for the Kubernetes health check to return Healthy.

Check the pod events:

kubectl get pods -n guestbook-demo
kubectl describe pod <pod-name> -n guestbook-demo

Look for ImagePullBackOff or CrashLoopBackOff. If you see the latter, use the tips in /tips/debug-crashloopbackoff to find the root cause.

2. Repository Connection Failed

If Argo CD cannot connect to your Git repo, check the argocd-repo-server logs:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server

Common causes include:

Incorrect SSH private key.
Firewall rules blocking port 22 (SSH) or 443 (HTTPS) from the cluster to GitHub.
Using a GitHub Deploy Key that doesn't have read access to the repository.

3. "OutOfSync" loop

Sometimes an application flips between Synced and OutOfSync rapidly. This is often caused by a conflict between Argo CD and another controller (like a Horizontal Pod Autoscaler).

If HPA is changing the replica count and Argo CD is trying to force it back to the Git value, they will fight forever. To fix this, ignore the replicas field in the Application spec:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

FAQ

Q: Can Argo CD manage Helm charts?
A: Yes. Argo CD natively supports Helm. You can point the source to a Helm repository or a folder containing a Chart.yaml. You can also provide a values.yaml file in Git to override default settings.

Q: What is the "App-of-Apps" pattern?
A: This is a pattern where you create one "Root" Argo CD Application that points to a folder containing other Application manifests. This allows you to manage your entire cluster state (including other apps) using a single GitOps entry point.

Q: Does Argo CD support multi-cluster management?
A: Yes. You can add external clusters to Argo CD via the CLI or UI. Once added, you can set the destination.server in your Application manifest to the API server URL of the remote cluster.

Conclusion

You have now transitioned from manual kubectl deployments to a professional GitOps workflow. We have installed Argo CD v2.11.0, connected a private repository, and deployed an application using the declarative model. By implementing selfHeal, you've ensured that your cluster is resilient to manual configuration drift. Furthermore, by using ApplicationSets, you've built a foundation that can scale from one application to hundreds across multiple environments.

The key takeaway is the shift in trust. You no longer trust the state of the cluster; you trust the state of Git. This makes your deployments repeatable, auditable, and significantly more secure.

Actionable next steps:

Migrate one existing production service to Argo CD.
Set up a "Management" repo that uses the App-of-Apps pattern to manage all your other Application manifests.
Implement a PR-based workflow where no one is allowed to push directly to the main branch of your manifest repo.

In Part 2 of this series, we will cover advanced Argo CD features, including Blue/Green and Canary deployments using Argo Rollouts, and how to integrate Prometheus metrics to trigger automatic rollbacks if a new release increases your error rate.

How to Configure Advanced Argo CD Sync Policies for GitOps

DevOps Start — Tue, 14 Apr 2026 15:16:09 +0000

Want to move beyond basic GitOps? I've put together a deep dive on mastering Argo CD sync policies, originally published on devopsstart.com.

Prerequisites

Before diving into advanced sync policies, you need a functioning Kubernetes cluster and a baseline Argo CD installation. This tutorial assumes you've already moved past the "Hello World" phase of GitOps. If you haven't set up your initial environment yet, follow the guide on /tutorials/how-to-set-up-argo-cd-gitops-for-kubernetes-automation to get the controller running.

To follow the examples in this guide, ensure the following tools are installed on your local machine:

Kubernetes Cluster: v1.28 or newer.
kubectl: v1.28 or newer, configured to communicate with your cluster.
Argo CD CLI: v2.10.0 or newer. This is essential for performing manual rollbacks and interacting with the API without the GUI.
Git: A repository (GitHub, GitLab or Bitbucket) containing your Kubernetes manifests.
A Sample Application: A deployment consisting of at least one Deployment, one Service and one ConfigMap.

You should have a basic understanding of the Application CRD (Custom Resource Definition) and how Argo CD tracks the state between your Git repository (the desired state) and your cluster (the live state). If you are unsure how to structure your Git folders, refer to /tutorials/how-to-set-up-argo-cd-gitops-for-kubernetes-automation.

Overview

Most teams start with Argo CD using "Manual Sync." You push code to Git, see the "Out of Sync" yellow badge in the UI and click the "Sync" button. While this feels safe, it's not production-grade. In large-scale environments, manual syncing creates a bottleneck and leads to configuration drift, where the cluster state diverges from Git for hours because a manual trigger was missed.

Simply turning on "Automatic Sync" can be dangerous. By default, Argo CD ensures that what is in Git is present in the cluster, but it won't necessarily remove what is not in Git. This leads to orphaned resources (leftover services or secrets) that can cause naming conflicts or security holes.

In this tutorial, we will build a production-ready synchronization strategy. You will learn how to implement automated pruning to keep your cluster clean, configure self-healing to prevent manual "hot-fixes" from persisting and manage complex deployment orders using sync waves. We will also tackle the "Day 2" problem of rollbacks: deciding when to revert a Git commit versus using the Argo CD rollback feature.

By the end of this guide, you'll have a robust GitOps pipeline that handles infrastructure lifecycle management automatically, reduces human error during deployments and provides a clear path for disaster recovery. You can find more details on the core architecture in the official Argo CD Documentation.

Step 1: Implementing Automated Pruning and Self-Healing

The first step toward production-grade GitOps is eliminating manual intervention. Many operators avoid prune: true fearing the accidental deletion of production resources. However, without pruning, your cluster becomes a graveyard of old ConfigMaps and abandoned Services.

Understanding Pruning and Self-Healing

Pruning is the process where Argo CD identifies resources that exist in the cluster (and are managed by the app) but are no longer present in the Git repository. If pruning is disabled, deleting a file in Git does nothing to the cluster.

Self-healing goes a step further. If a developer uses kubectl edit to change a replica count or an environment variable directly in the cluster, Argo CD detects the drift and immediately overwrites those changes with the state defined in Git.

Configuration

To enable these, modify the syncPolicy section of your Application manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/argoproj/argocd-example-apps.git'
    targetRevision: HEAD
    path: guestbook
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: guestbook
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Apply this configuration using kubectl:

kubectl apply -f application.yaml

Testing the Policy

To verify pruning, delete a resource from your Git repository (for example, a Service manifest) and push the change.

git rm manifests/service.yaml
git commit -m "Remove legacy service"
git push origin main

Wait for the next sync cycle (usually 3 minutes by default, or instantly if you have a webhook configured), then check your cluster:

kubectl get svc -n guestbook
# Expected output: Error from server (NotFound)

Now, test self-healing. Try to manually scale your deployment:

kubectl scale deployment guestbook-ui --replicas=10 -n guestbook

Run kubectl get pods -n guestbook. You'll notice the pods scale up for a moment, but within 60 to 120 seconds, Argo CD will detect the drift and scale them back down to the number specified in Git.

Step 2: Mastering Advanced Sync Options

Standard syncing works for 90% of resources, but Kubernetes has immutable fields. For example, if you try to change the selector of a Service or certain fields in a Job, the Kubernetes API rejects the update with a 422 Unprocessable Entity error. Argo CD will remain in a "Sync Failed" state indefinitely.

Using Replace=true

The Replace=true option tells Argo CD to use kubectl replace or kubectl create instead of kubectl apply. This effectively deletes and recreates the resource if an update fails due to immutable fields.

Add this to your syncOptions:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true
      - SkipDryRunOnMissingResource=true

SkipDryRunOnMissingResource=true is particularly useful when dealing with complex CRDs. Sometimes the dry-run validation fails because a dependent resource doesn't exist yet, even though the actual application would succeed.

ApplicationSet Level Policies

If you manage 50 clusters using an ApplicationSet, you don't want to define these policies 50 times. Define the syncPolicy within the template section of the ApplicationSet.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-config
spec:
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            url: https://kubernetes.default.svc
          - cluster: engineering-prod
            url: https://prod-cluster.example.com
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/argoproj/argocd-example-apps.git'
        targetRevision: HEAD
        path: guestbook
      destination:
        server: '{{url}}'
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Step 3: Implementing Sync Waves and Hooks

In production, you cannot deploy everything simultaneously. You might need a database schema migration to finish before the API server starts, or a smoke test to pass before the LoadBalancer switches traffic.

Sync Waves

Sync waves allow you to assign an order to resources. Argo CD applies resources in increasing order of their wave number. Resources with the same wave are applied concurrently.

Add the annotation argocd.argoproj.io/sync-wave to your manifests.

Database Migration (Wave 1):

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: migration-tool:v1.2.0
      restartPolicy: OnFailure

Application Deployment (Wave 2):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  # Deployment spec here

Cache Warmup (Wave 3):

apiVersion: batch/v1
kind: Job
metadata:
  name: cache-warmup
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  # Job spec here

Argo CD waits for the Wave 1 Job to reach a "Healthy" state before attempting to create the Wave 2 Deployment.

Sync Hooks

Hooks are used for transient tasks rather than permanent resources.

PreSync: Runs before the sync starts. Ideal for backups.
Sync: Runs during the sync.
PostSync: Runs after the sync completes. Ideal for notifications or integration tests.

Example of a PreSync backup hook:

apiVersion: batch/v1
kind: Job
metadata:
  name: pre-sync-backup
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: backup
          image: backup-util:latest
      restartPolicy: OnFailure

The HookSucceeded policy ensures the Job object is deleted from the cluster once it completes successfully, preventing the buildup of thousands of finished Job objects.

Step 4: Designing Robust Rollback Strategies

When a production deployment fails, the pressure to recover "right now" often leads to a conflict between "Pure GitOps" and "Fast Recovery."

Strategy A: The Git-based Rollback (Pure GitOps)

In this approach, you never use the Argo CD UI for rollbacks. You use git revert.

Pros:

Perfect audit trail.
Zero drift between Git and Cluster.
Works across multiple clusters simultaneously.

Cons:

Slower recovery time. You must commit, push and wait for the sync cycle.

Execution:

git log --oneline
# a1b2c3d (HEAD) Update image to v2.1.0 (BROKEN)
# e5f6g7h Update image to v2.0.0 (STABLE)

git revert a1b2c3d
git push origin main

Strategy B: The Argo CD UI/CLI Rollback (Emergency Fast-Track)

Argo CD allows you to rollback to a previous successful revision of the application. This is an immediate operation that bypasses Git.

Execution using CLI:

argocd app rollback guestbook-production 12

The Danger Zone: If automated: selfHeal: true is enabled, a manual rollback will be immediately overwritten. Argo CD will see the cluster is running v2.0.0 (due to the rollback) while Git still says v2.1.0. Because self-healing is on, it will "fix" the cluster by re-deploying the broken v2.1.0.

The Decision Matrix

Follow these rules for professional rollback management:

For Non-Critical Bugs: Use git revert. It is the only way to ensure the environment remains reproducible.
For Critical Outages (P0):

Step 1: Disable Auto-Sync in the UI or CLI.
Step 2: Perform the Argo CD Rollback to a known good revision.
Step 3: Fix the code in Git.
Step 4: Update Git to the fixed version.
Step 5: Re-enable Auto-Sync.

If you encounter constant Pod failures during these transitions, you might be facing a /troubleshooting/crashloopbackoff-kubernetes scenario, which requires log analysis before deciding on a rollback strategy.

Step 5: Implementing Custom Health Checks

Argo CD knows how to check the health of standard resources. However, if you use Custom Resource Definitions (CRDs) from an operator (like Prometheus or Istio), Argo CD only knows if the resource was created. It doesn't know if the operator actually succeeded in deploying the underlying components.

This means a Sync Wave might move to Wave 2 even if the Wave 1 CRD is still in a "Pending" or "Error" state.

Defining a Lua Health Check

Argo CD allows you to define health checks using Lua scripts in the argocd-cm ConfigMap in the argocd namespace.

Assume you have a custom resource called DatabaseInstance that has a status.phase field.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.apps.example.com/DatabaseInstance: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.phase == 'Ready' then
        hs.status = 'Healthy'
        hs.message = 'Database is ready'
        return hs
      end
      if obj.status.phase == 'Failed' then
        hs.status = 'Degraded'
        hs.message = 'Database failed to provision'
        return hs
      end
    end
    hs.status = 'Progressing'
    hs.message = 'Waiting for database to be ready'
    return hs

Apply the change and restart the argocd-application-controller:

kubectl apply -f argocd-cm.yaml
kubectl rollout restart deployment argocd-application-controller -n argocd

Now, Argo CD will wait for the DatabaseInstance to reach the Ready phase before marking the resource as Healthy.

Step 6: Managing Sync Windows and Maintenance Periods

In enterprise environments, automated deployments are often prohibited during "Freeze Periods" (e.g., Black Friday). You still want GitOps to track changes, but you don't want them applied to the cluster.

Argo CD doesn't have a built-in calendar, but you can implement this using labels and automation.

The Label-Based Freeze Approach

Add a label sync-window: frozen to your application.

kubectl label app guestbook-production sync-window=frozen

Create a simple automation (via GitHub Action or CronJob) that toggles the automated sync policy.

The "Freeze" Script:

# Disable auto-sync during freeze
argocd app set guestbook-production --sync-policy manual

The "Unfreeze" Script:

# Enable auto-sync after freeze
argocd app set guestbook-production --sync-policy automated

For a more sophisticated approach, use an external controller that watches for these labels and modifies the Application spec. This ensures the cluster remains untouched until the window opens.

Troubleshooting

Issue 1: Resource "Flickering" (The Sync Loop)

A resource constantly switches between "Synced" and "Out of Sync." This typically happens when a controller (like an HPA or Service Mesh) modifies the resource after Argo CD applies it.

The Fix: Use ignoreDifferences to tell Argo CD to ignore fields managed by other controllers.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Issue 2: Pruning Deleted Critical Resources

You accidentally deleted a namespace or a critical Secret in Git, and Argo CD pruned it from the cluster, causing an outage.

The Fix: Use the prune safety override. Annotate specific resources to prevent them from being pruned, regardless of the application-level policy.

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

Issue 3: Sync Wave Hanging

A sync wave is stuck in "Progressing" and refuses to move to the next wave.

The Fix: Check the health of the resource in the current wave. If it's a Job, ensure it is actually completing. If you implemented a custom health check, ensure the Lua script isn't returning Progressing indefinitely due to a typo in the status field.

kubectl describe job db-migration -n guestbook

FAQ

Q: Does prune: true delete resources in namespaces not managed by Argo CD?
A: No. Argo CD only prunes resources that are tracked within the specific Application's scope and managed by that application.

Q: Can I have different sync waves for different clusters?
A: Yes. Since sync waves are defined as annotations on the manifests themselves, you can use Kustomize or Helm to apply different annotations based on the target environment (e.g., a longer warmup wave in production than in dev).

Q: What happens if a Sync Hook fails?
A: If a PreSync hook fails, Argo CD will stop the sync process and mark the application as "Degraded," preventing the deployment of potentially broken code.

Q: Is Replace=true safe for all resources?
A: Not always. Since it deletes and recreates the resource, any fields not defined in your Git manifest (like some dynamically assigned annotations or labels) will be lost. Use it only for resources with immutable fields.

Conclusion

Moving from manual synchronization to advanced sync policies separates a "demo" GitOps setup from a production-grade platform. By implementing automated pruning and self-healing, you eliminate configuration drift and ensure Git is the absolute source of truth. Sync waves and hooks bring the orchestration capabilities of traditional CI/CD pipelines into the declarative world of Kubernetes.

In this tutorial, we've covered:

Enabling prune and selfHeal to maintain cluster hygiene.
Using Replace=true to handle immutable Kubernetes fields.
Orchestrating complex deployments with Sync Waves and Hooks.
The critical distinction between Git reverts and Argo CD rollbacks.
Extending Argo CD's intelligence with custom Lua health checks.
Managing deployment freezes using sync windows.

Your next steps should be to audit your current Application manifests. Identify resources managed by other controllers and apply ignoreDifferences to stop sync flickering. Then, map out your application dependencies and assign sync waves to ensure your databases always precede your APIs.

This concludes our deep dive into Argo CD and our series on GitOps automation. For those looking to further their expertise in site reliability, our guide on /interview/senior-sre-interview-questions-answers-for-2026 provides insight into how these patterns are evaluated in professional settings.

Deploy an EKS Cluster with Terraform

DevOps Start — Tue, 14 Apr 2026 15:10:58 +0000

This tutorial was originally published on devopsstart.com. Learn how to automate the deployment of a production-ready EKS cluster using Terraform modules!

This tutorial walks you through deploying a production-ready Amazon EKS cluster using Terraform.

Step 1: Set Up the Project Structure

Create a new Terraform project:

mkdir eks-cluster && cd eks-cluster
touch main.tf variables.tf outputs.tf

Step 2: Configure the AWS Provider

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  default = "us-west-2"
}

variable "cluster_name" {
  default = "my-eks-cluster"
}

Step 3: Create the VPC

EKS needs a VPC with public and private subnets:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["${var.region}a", "${var.region}b", "${var.region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

Step 4: Deploy the EKS Cluster

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.30"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_groups = {
    default = {
      instance_types = ["t3.medium"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
    }
  }
}

Step 5: Apply and Connect

terraform init
terraform plan
terraform apply

aws eks update-kubeconfig --name my-eks-cluster --region us-west-2
kubectl get nodes

You should see your two worker nodes in Ready state.

Cleanup

To avoid ongoing costs:

terraform destroy

Troubleshooting

Nodes not joining the cluster? Check that the node group subnets have NAT gateway access and the correct IAM roles are attached.

kubectl connection refused? Run aws eks update-kubeconfig again and verify your AWS credentials.

How to Automate Terraform Reviews with GitHub Actions

DevOps Start — Tue, 14 Apr 2026 15:00:51 +0000

Stop review fatigue and catch IaC security risks before they hit production. This guide was originally published on devopsstart.com.

Introduction

Reviewing Infrastructure as Code (IaC) is a different beast compared to reviewing application logic. When you review a Java or Python PR, you're looking for bugs, race conditions or inefficient algorithms. When you review Terraform, you're looking for "blast radius". A single misplaced character in a resource name or an incorrectly configured count variable can trigger a terraform destroy on a production database, leading to catastrophic downtime.

Despite the stakes, most DevOps teams suffer from "Review Fatigue". Human reviewers often spend 80% of their time pointing out trivial issues: missing tags, inconsistent indentation or hardcoded region strings. By the time they get to the actual architectural flaws—like an S3 bucket missing encryption or a security group open to 0.0.0.0/0—they're mentally exhausted. This creates a dangerous gap where critical infrastructure risks slip into production because the reviewer was too busy complaining about trailing commas.

The goal of this tutorial is to eliminate that toil. You'll learn how to integrate CodeRabbit with GitHub Actions to automate the first pass of your Terraform reviews. CodeRabbit isn't just a generic LLM wrapper; it's designed to understand the context of your repository. By the end of this guide, you'll have an AI-powered reviewer that flags security vulnerabilities, enforces naming conventions and analyzes your terraform plan output to warn you before you accidentally delete your entire VPC.

To make this work, we'll be using Terraform v1.7.0 and GitHub Actions. If you're managing state for a growing team, ensure you've already implemented /blog/terraform-state-locking-a-guide-for-growing-teams to avoid state corruption during these automated workflows.

Prerequisites

Before starting the integration, you need a specific set of tools and access levels. If you're missing any of these, the automation will fail during the handshake between GitHub and the AI engine.

First, you need a GitHub repository containing Terraform HCL code. This repository must be hosted on GitHub (Cloud or Enterprise) because CodeRabbit integrates directly via the GitHub App ecosystem. You should have administrative access to the repository to install apps and configure GitHub Action secrets.

Second, you need a CodeRabbit account. While there is a free trial, you'll need an active account to generate the integration keys required for the AI to read your pull requests.

Third, you must have Terraform v1.7.0 or later installed locally if you plan to test the HCL changes before pushing. We recommend using a version manager like tfenv to ensure consistency across your team.

Finally, a basic grasp of YAML syntax is required. You won't be writing complex scripts, but you'll be editing a .coderabbit.yaml configuration file. This file acts as the "brain" of your AI reviewer, telling it whether to be a strict security auditor or a helpful mentor.

Checklist of requirements:

GitHub Account with Repository Admin permissions.
CodeRabbit Account.
Terraform v1.7.0+.
A functioning remote backend (S3, GCS or Azure Blob) for your Terraform state.

Overview

In this tutorial, we are building an automated AI-driven guardrail system for your infrastructure. The objective is to move the "boring" parts of the code review—syntax, tagging and basic security—from the human reviewer to the AI.

The architecture works like this: A developer pushes a branch and opens a Pull Request (PR). This triggers a GitHub Action that runs terraform plan. Simultaneously, the CodeRabbit GitHub App intercepts the PR event. It reads the diff of the HCL files and the output of the terraform plan. Using a customized .coderabbit.yaml file, the AI applies your specific organization's rules (for example, "All AWS resources must have a Project tag").

The AI then posts a series of line-by-line comments on the PR. If it sees an S3 bucket without public_access_block, it won't just say "this is wrong"; it will provide the exact HCL snippet needed to fix it.

The real power comes from the synergy between the static code analysis and the execution plan. While static analysis can see that a resource is being changed, the terraform plan tells the AI if that change will cause a "Replacement" (destroy and recreate). The AI can then warn the reviewer: "Warning: Changing the name attribute of this RDS instance will cause a complete recreation of the database, leading to data loss."

By the end of this setup, your human reviewers will only need to focus on the "Why" (the architectural intent) rather than the "How" (the syntax and basic security).

Step 1: Installing the CodeRabbit GitHub App

The first step is to bridge the gap between your code and the AI engine. CodeRabbit operates as a GitHub App, which means it doesn't require you to manage complex SSH keys or manual webhooks.

Navigate to the GitHub Marketplace or the CodeRabbit dashboard.
Click "Install" and select the specific repositories you want the AI to monitor. Do not install it on all repositories unless you want AI comments on every single project in your organization.
Grant the necessary permissions. CodeRabbit requires "Read and Write" access to Pull Requests and "Read" access to repository contents. This allows the AI to see the diffs and post suggestions.

Once installed, you can verify the connection by opening a test PR in one of the selected repositories. You should see a CodeRabbit bot join the conversation shortly after the PR is created.

Step 2: Configuring the `.coderabbit.yaml` for Terraform

Generic AI reviews are often too noisy or too vague. To make the AI a "Senior DevOps Engineer," you need to give it a set of instructions. This is done via a .coderabbit.yaml file placed in the root of your repository.

This file defines the "Persona" and the "Instructions". For Terraform, you want the AI to prioritize stability, security and cost over "clever" code.

Create a file named .coderabbit.yaml in your root directory with the following content:

language: "hcl"
reviews:
  profile: "infrastructure_expert"
  instructions: |
    You are a Senior Platform Engineer specializing in Terraform v1.7.0.
    Your goal is to ensure infrastructure is secure, scalable and maintainable.

    Strictly enforce the following rules:
    1. Tagging: Every resource must have 'Environment', 'Owner' and 'Project' tags.
    2. Security: Flag any security group rule that allows 0.0.0.0/0 on port 22 or 3389.
    3. State Management: Ensure no local state is used; remote backends are mandatory.
    4. Naming: Resources must use kebab-case (e.g., 'web-server-prod').
    5. S3 Buckets: Must have 'versioning' enabled and 'public_access_block' configured.
    6. Secrets: Flag any hardcoded passwords, API keys or tokens. Suggest using AWS Secrets Manager or HashiCorp Vault.
    7. DRY Principle: Suggest the use of modules or `for_each` if the same resource is repeated more than 3 times.

  review_style: "concise"
  focus:
    - security
    - cost_optimization
    - maintainability

In this configuration, the instructions block is where the magic happens. By specifying "S3 Buckets must have versioning enabled", you've turned a generic LLM into a specialized Terraform auditor. When the AI sees an aws_s3_bucket resource without a corresponding aws_s3_bucket_versioning resource, it will now trigger a warning.

Step 3: Setting up the GitHub Actions Workflow

While CodeRabbit handles the AI review of the code, you still need the actual terraform plan to provide context. The AI is much more effective when it knows exactly what Terraform intends to do to your real-world infrastructure.

Create a file at .github/workflows/terraform-review.yml:

name: "Terraform AI Review"

on:
  pull_request:
    branches:
      - main

jobs:
  terraform-plan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.0

      - name: Terraform Init
        run: terraform init
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Terraform Plan
        id: plan
        run: terraform plan -no-color > plan_output.txt
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Upload Plan for AI
        uses: actions/upload-artifact@v4
        with:
          name: tf-plan
          path: plan_output.txt

This workflow does three critical things:

It initializes the environment using the official hashicorp/setup-terraform action.
It generates a plan and redirects the output to a text file (plan_output.txt). We use -no-color because AI models struggle with ANSI color codes.
It uploads the plan as an artifact. CodeRabbit can be configured to ingest these artifacts or read the logs from the Action to understand the impact of the change.

If you are deploying complex clusters, you might want to combine this with a tutorial on /tutorials/deploy-eks-cluster-with-terraform to ensure your plan outputs are structured correctly for the AI to parse.

Step 4: Configuring AI Personas for Infrastructure

Beyond the .coderabbit.yaml file, you can refine how the AI communicates. You don't want the AI to sound like a bot; you want it to sound like a colleague who has seen a hundred production outages.

In the CodeRabbit dashboard, you can adjust the "System Prompt" or the "Persona". For infrastructure, I recommend a "Conservative Auditor" persona. This persona is programmed to be pessimistic. It assumes that if a change looks risky, it probably is.

For example, if you change a vm_size in Azure or an instance_type in AWS, a "helpful" AI might say "Great, you're upgrading the CPU!". A "Conservative Auditor" AI will say "Warning: Changing the instance type may cause a reboot of the instance. Ensure your application handles this gracefully or perform this during a maintenance window."

To implement this, add a context section to your .coderabbit.yaml:

  context: |
    Always assume the environment is production unless the branch name contains 'dev'.
    Prioritize availability over cost.
    If a change causes a resource replacement, mark the comment as 'Critical'.

This prevents the AI from suggesting "cost-saving" measures that might degrade performance in a production environment.

Step 5: The Workflow in Action (The "Bad PR" Test)

To verify your setup, let's intentionally create a "Bad PR". This will test if the AI is actually following the rules we set in Step 2.

Create a new branch and add the following "anti-pattern" code to your main.tf:

# BAD PRACTICE: Hardcoded secret
resource "aws_db_instance" "database" {
  allocated_storage    = 20
  engine               = "mysql"
  instance_class       = "db.t3.micro"
  username             = "admin"
  password             = "supersecret123" # AI should flag this
  skip_final_snapshot  = true
}

# BAD PRACTICE: Wide open security group
resource "aws_security_group" "allow_ssh" {
  name        = "allow_ssh"
  description = "Allow SSH inbound traffic"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # AI should flag this
  }
}

# BAD PRACTICE: Missing mandatory tags
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  # Missing Environment, Owner and Project tags
}

Push this code and open a Pull Request. Within seconds, you should see the following behavior:

Secret Detection: CodeRabbit will highlight the password = "supersecret123" line. It will comment: "Critical: Hardcoded secret detected. Please use AWS Secrets Manager or a variable marked as sensitive = true."
Security Audit: It will flag the cidr_blocks = ["0.0.0.0/0"] for port 22. It will suggest: "Security Risk: SSH is open to the world. Restrict this to your company's VPN CIDR range."
Policy Enforcement: It will note that the aws_instance.web resource is missing the mandatory tags defined in your .coderabbit.yaml. It will provide a code block:

   tags = {
     Environment = "prod",
     Owner       = "platform-team",
     Project     = "web-app"
   }

Step 6: Integrating with `terraform plan` for Risk Analysis

The most powerful feature of this setup is the AI's ability to read the terraform plan output. Static analysis can tell you that you changed a line of code, but only the plan tells you if that change will destroy a resource.

Consider this scenario: You change the name of an S3 bucket in your HCL.

Static analysis sees: bucket = "my-old-bucket" $\rightarrow$ bucket = "my-new-bucket". This looks like a simple string change.

However, the terraform plan output will say:
-/+ destroy and then create replacement

When CodeRabbit analyzes this plan output, it will post a high-priority warning:
"🚨 Destructive Change Detected: Changing the bucket name will cause Terraform to destroy the existing S3 bucket and create a new one. This will result in the loss of all existing data in my-old-bucket unless you have a backup or use the terraform state mv command."

This is the difference between a generic AI and an IaC-aware AI. To ensure this works, make sure your GitHub Action outputs the plan in a way that the AI can access. If you use a tool like tfplan or terraform-pr, the AI can read the JSON output of the plan, which is even more precise.

Step 7: Verification and Iteration via AI Chat

One of the biggest frictions in code reviews is the "ping-pong" of comments.
Reviewer: "Can you change this to a module?"
Developer: "Why?"
Reviewer: "Because we have five of these."
Developer: "Okay, I'll do it."

With CodeRabbit, you can use the "Chat with AI" feature directly in the PR. Instead of waiting for a human, the developer can ask the AI to implement the suggestion.

For example, if the AI suggests using a module for repeated resources, the developer can reply to the comment:
@coderabbitai please rewrite these three aws_instance resources into a single module call using for_each.

The AI will then generate the updated HCL code, including the module definition and the call. The developer can simply click "Commit Suggestion" to apply the change. This reduces the PR cycle time from hours (or days) to minutes.

Troubleshooting

Even with a straightforward setup, you'll likely encounter a few common issues. Here is how to solve them.

AI is too noisy (Too many comments)

If the AI is flagging things that are actually acceptable in your environment, your .coderabbit.yaml is too vague.
Fix: Be more specific in your instructions. Instead of saying "Flag security issues," say "Flag security group rules that allow 0.0.0.0/0, but ignore these rules if the resource is tagged as PublicLoadBalancer."

AI cannot see the Terraform Plan

If the AI is giving generic advice but not mentioning resource destruction, it's not reading your GitHub Action output.
Fix: Ensure your GitHub Action is running before the AI completes its review. You can use the workflow_run trigger or ensure the AI is configured to wait for the terraform-plan job to finish. Check that the plan output is not being truncated by GitHub's log limits (which is why uploading as an artifact is preferred).

"Permission Denied" during Terraform Init

Your GitHub Action might fail at the terraform init step.
Fix: This is almost always a secret management issue. Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined in your GitHub Repository Secrets. If you're using OIDC (which is recommended), use the aws-actions/configure-aws-credentials action instead of static keys.

AI suggests outdated Terraform syntax

LLMs are trained on data that might be a year old. It might suggest resource "aws_s3_bucket_policy" syntax that has changed in the latest AWS provider.
Fix: In your .coderabbit.yaml, explicitly state the provider versions. Add: "Use AWS Provider v5.0+ syntax. Do not use deprecated attributes like acl inside the aws_s3_bucket resource; use the aws_s3_bucket_acl resource instead."

FAQ

Does CodeRabbit have access to my actual AWS/Azure credentials?
No. CodeRabbit only has access to the code diffs and the text output of the terraform plan generated by GitHub Actions. Your cloud credentials remain securely stored in GitHub Secrets and are only used by the GitHub Runner.

Can I use this with Terraform Cloud or Spacelift?
Yes. Instead of running terraform plan in a GitHub Action, you can configure the CI/CD tool to post the plan output as a comment on the PR. CodeRabbit will then analyze that comment as part of its review process.

Will the AI automatically merge my PRs?
No. CodeRabbit is a reviewer, not a merger. It provides suggestions and flags risks, but the final decision to approve and merge always stays with your human engineers.

How does this handle different environments (Dev vs Prod)?
You can handle this by using different .coderabbit.yaml profiles or by adding logic to your context block (as shown in Step 4) that tells the AI to treat branches like main or prod with higher strictness than feature/* branches.

Conclusion

By automating your Terraform reviews with GitHub Actions and CodeRabbit, you've shifted the burden of quality control from your most expensive resource—your senior engineers—to an AI that doesn't get tired or overlook a missing tag.

You've implemented a system that:

Enforces organization-wide tagging and naming standards automatically.
Catches "low-hanging fruit" security vulnerabilities before they reach a human.
Analyzes terraform plan outputs to prevent accidental data loss through resource replacement.
Accelerates the development loop by allowing developers to iterate with the AI via chat.

The next step for your team is to expand these rules. Start by auditing your last ten production incidents. For every incident that was caused by a misconfiguration (e.g., a missing lifecycle { prevent_destroy = true } block), add a corresponding rule to your .coderabbit.yaml.

As your infrastructure grows, you might also want to explore GitOps patterns to further automate the deployment of these reviewed changes. For a complete pipeline, check out /tutorials/how-to-set-up-argo-cd-gitops-for-kubernetes-automation to learn how to sync your approved Terraform changes directly to your clusters.

Now, go delete those hardcoded secrets and start automating your guardrails.

Fix Kubernetes CrashLoopBackOff: Root Causes & Diagnosis

DevOps Start — Tue, 14 Apr 2026 14:50:43 +0000

Dealing with the dreaded CrashLoopBackOff in your cluster? This comprehensive guide, originally published on devopsstart.com, walks you through diagnosing root causes and implementing prevention strategies.

Problem: What CrashLoopBackOff actually means

When you see CrashLoopBackOff in your kubectl get pods output, you aren't looking at a specific error but a state. It is a symptom. It means your container is crashing repeatedly and Kubernetes is attempting to restart it.

To prevent your cluster from hammering a failing application and wasting CPU cycles, Kubernetes implements an exponential backoff delay. The first restart happens almost immediately. If it crashes again, Kubernetes waits 10 seconds, then 20, 40 and so on, up to a maximum of five minutes. This is why a pod might appear "stuck" for several minutes even after you've pushed a fix.

Understanding this mechanism is critical for production SREs. If you wait for a pod to recover while it's in the backoff phase, you are wasting time. You should diagnose the cause using the official Kubernetes documentation as a baseline for pod lifecycles.

Root Causes of Pod Crashes

Most CrashLoopBackOff events fall into one of three buckets. I've seen these fail in clusters with >50 nodes where configuration drift becomes the primary culprit.

1. Configuration and Environment Failures

This is the most common cause during new deployments. The application starts, looks for a required environment variable or a mounted Secret/ConfigMap, finds it missing and throws an unhandled exception. Because the process exits with a non-zero code, Kubernetes marks it as failed.

2. Resource Constraints (OOMKilled)

When a container exceeds its defined memory limit, the Linux kernel invokes the Out-Of-Memory (OOM) killer. Kubernetes catches this and reports the status as OOMKilled. This happens either because the limits are set too low for the application's baseline needs or because of a memory leak that consumes available RAM over time. In Java applications, this is frequently caused by the JVM heap size (-Xmx) being larger than the Kubernetes memory limit.

3. Application and Dependency Failures

Modern applications often use "fail-fast" logic. If the app cannot connect to its database, Redis cache or an external API during the startup sequence, it will exit immediately. If your liveness probes are too aggressive, Kubernetes might kill a healthy pod that is simply taking too long to boot, creating a loop.

Solution: Step-by-Step Production Diagnosis

When a production pod is crashing, do not guess. Follow this systematic triage process.

Step 1: Check the High-Level Status

Start by identifying the exact state and the restart count.

kubectl get pods

Expected Output:

NAME                            READY   STATUS             RESTARTS   AGE
api-gateway-7d4f5b9c-abc12      0/1     CrashLoopBackOff   14         12m

Step 2: Inspect the Events and Exit Codes

The describe command tells you why Kubernetes stopped the container. Look specifically for the Last State section.

kubectl describe pod api-gateway-7d4f5b9c-abc12

Search for the Containers: block. You will see an entry similar to this:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    137
  Started:      Mon, 01 Jan 2024 10:00:00 +0000
  Finished:     Mon, 01 Jan 2024 10:00:05 +0000

Exit Code Cheat Sheet:

0: The app finished its task and exited cleanly. If this is a long-running service, your entrypoint is wrong.
1: Generic application crash (check logs for stack traces).
137: OOMKilled. The container used more memory than its limit.
139: Segmentation fault (memory corruption or binary incompatibility).
143: Graceful termination (SIGTERM) that took too long and was killed.

Step 3: The Golden Rule of Logs

Running kubectl logs <pod> on a crashing pod usually shows the logs of the current (newly started) container, which might be empty if the crash happens instantly. To see why the previous instance died, use the --previous flag.

kubectl logs api-gateway-7d4f5b9c-abc12 --previous

If the logs indicate a bad image version, you should perform a rollback immediately to restore service before spending hours debugging.

Step 4: The "Sleep Infinity" Hack for Deep Debugging

If logs are empty (e.g., the crash happens before the logger initializes), you need to get inside the container. You cannot exec into a crashing pod because it isn't running.

Override the container command in your deployment YAML to keep the container alive regardless of the app's state:

spec:
  containers:
  - name: api-gateway
    image: api-gateway:v1.2.0
    command: ["sh", "-c", "sleep infinity"]

Apply this change, then execute a shell to manually run your application binary and observe the crash in real-time:

kubectl apply -f deployment.yaml
kubectl exec -it api-gateway-7d4f5b9c-abc12 -- /bin/sh
# Once inside the pod:
/app/start-server.sh

Prevention: Stopping the Loop

To prevent CrashLoopBackOff from hitting production, implement these guardrails.

1. Right-Sized Resources

Use a Vertical Pod Autoscaler (VPA) in staging to find the actual memory usage. Set your requests close to the average usage and limits with a reasonable buffer. For example, if a Go microservice consistently uses 120MiB, set requests to 128MiB and limits to 256MiB. This reduces the likelihood of OOMKilled events by ensuring the scheduler places the pod on a node with actual available capacity.

2. Graceful Startup with Startup Probes

Implement a startupProbe. This tells Kubernetes to ignore liveness and readiness probes until the application has finished its initial boot sequence. This prevents Kubernetes from killing a pod that is simply performing a heavy database migration or cache warm-up.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

This configuration gives the app 300 seconds to start before the liveness probe takes over.

3. Configuration Validation

Use tools like kube-score or a CI pipeline that validates ConfigMap and Secret existence before triggering a rollout. For complex multi-cluster environments, implementing GitOps patterns via Argo CD or Flux can help ensure configuration consistency across environments, reducing "it worked in staging" failures.

FAQ

Q: Why does my pod stay in CrashLoopBackOff even after I fixed the config?
A: Because of the exponential backoff. Kubernetes waits longer between each restart attempt. You can force a fresh start by deleting the pod: kubectl delete pod <pod-name>.

Q: Can a CrashLoopBackOff be caused by the node itself?
A: Yes. If the node is under extreme disk pressure or PID pressure, the container runtime might fail to start the container, leading to a crash loop. Check kubectl describe node <node-name> for "Pressure" conditions.

Q: How do I distinguish between an application crash and a Kubernetes-initiated kill?
A: Look at the Reason in kubectl describe pod. Error usually implies the application exited with a non-zero code, while OOMKilled explicitly means the kernel killed the process for exceeding memory limits.

Conclusion and Next Steps

CrashLoopBackOff is a signal that your application is failing its basic environment or resource requirements. The fastest path to resolution is identifying the exit code and inspecting the --previous logs.

Your next steps:

Audit your current production deployments for pods missing startupProbes.
Review your limits vs requests to ensure you aren't over-committing memory on your nodes.
Implement a standardized "debug" command override in your developer handbook to speed up the "Sleep Infinity" process during incidents.

Kubernetes for Beginners: Deploy Your First Application

DevOps Start — Tue, 14 Apr 2026 14:45:40 +0000

New to container orchestration? This practical guide, originally published on devopsstart.com, will help you demystify Kubernetes and deploy your first app using Minikube.

If you're looking to scale your applications, improve reliability, and automate deployment workflows, you've likely heard of Kubernetes. Often abbreviated as K8s, it's the de-facto standard for container orchestration, helping organizations manage their containerized workloads with unparalleled efficiency. But for many, getting started with Kubernetes can feel like staring at a complex alien spaceship control panel.

Don't fret. This article is your practical, no-nonsense guide to demystifying Kubernetes. We'll cut through the jargon, explain the core concepts in plain language, and get you hands-on with a local cluster to deploy your very first application. By the end, you'll have a solid foundation and the confidence to explore this powerful platform further.

What is Kubernetes? Understanding Container Orchestration

At its core, Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications. Think of it as an operating system for your data center, but specifically designed for containers.

Before Kubernetes, managing applications deployed as hundreds or thousands of individual containers across multiple servers was a Herculean task. Imagine you have 50 services, each running 10 instances, requiring updates, scaling, and self-healing capabilities. Manually orchestrating this complexity quickly becomes impossible. This is where container orchestration tools like Kubernetes step in.

The Problems Kubernetes Solves

Kubernetes tackles several critical challenges in modern software development:

Deployment and Updates: It automates the process of rolling out new features or fixes without downtime. Need to update your application? Kubernetes can replace instances one by one, ensuring service continuity and allowing for easy rollbacks if issues arise.
Scaling: Demand spikes? Kubernetes can automatically scale your application up or down by adding or removing container instances based on CPU usage, custom metrics, or predefined schedules. This eliminates the need for manual server provisioning.
Self-Healing: If a container crashes, a server fails, or an application becomes unresponsive, Kubernetes can automatically restart the container, reschedule it to a healthy node, or even replace the failed server. It's designed for resilience and high availability.
Load Balancing and Service Discovery: Kubernetes automatically distributes incoming network traffic across multiple healthy instances of your application, preventing any single instance from becoming overloaded. It also provides service discovery, allowing containers to find and communicate with each other using logical names rather than hardcoding unstable IP addresses.
Resource Management: It efficiently manages and allocates computing resources (CPU, memory) across your cluster. This ensures containers get what they need to perform optimally without wasting valuable capacity.
Portability: Kubernetes isn't tied to a specific cloud provider or infrastructure. You can run the same application configuration on AWS, Azure, GCP, on-premises data centers, or even on your laptop, offering true hybrid and multi-cloud capabilities.

In essence, Kubernetes abstracts away the underlying infrastructure complexities, allowing developers and operations teams to focus more on building and delivering applications rather than constantly managing servers. My take? It's the biggest game-changer in infrastructure since virtualization. If you're running anything in containers in production, you need Kubernetes.

Understanding Core Kubernetes Concepts: Pods, Deployments & Services

Before we get our hands dirty, let's briefly touch upon the fundamental building blocks of Kubernetes. Don't worry about the YAML yet; just grasp the idea behind each component.

1. Nodes: The Foundation

Imagine your Kubernetes cluster as a fleet of computers. Each computer in this fleet is called a Node.

Worker Nodes: These are the machines (physical or virtual) where your actual containerized applications run. They execute the workloads and are managed by the control plane.
Control Plane (formerly Master Node): This is the brain of the cluster. It makes global decisions about the cluster (e.g., scheduling Pods, detecting and responding to cluster events), maintains the desired state, and manages the worker nodes. In a production setup, the control plane is typically distributed across multiple machines for high availability. For local development, like with Minikube, it often runs on a single machine or even within a VM.

2. Pods: The Smallest Deployable Unit

A Pod is the smallest and most fundamental unit you can deploy in Kubernetes. Think of a Pod as a tightly-knit group of one or more containers that share network, storage, and lifecycle resources.

Why Pods, not just containers? While most Pods contain a single application container, some applications might need a "sidecar" container to perform auxiliary tasks like logging, data synchronization, or proxying. The Pod ensures these co-located containers are scheduled together on the same node and share their environment.
Analogy: If a Docker container is like a single LEGO brick, a Pod is a small, carefully assembled LEGO model (e.g., a car with wheels and an engine) that you can then place on a larger LEGO city (your Node).

Pods are ephemeral; they come and go. When a Pod dies (e.g., due to a crash or node failure), Kubernetes doesn't try to revive that specific Pod instance. Instead, it creates a new Pod to replace it, ensuring the desired state is maintained.

3. Deployments: Managing Your Pods

Directly managing individual, ephemeral Pods is tedious and doesn't scale. That's where Deployments come in. A Deployment is a higher-level object that manages the creation and lifecycle of a set of identical Pods.

What it does: You define a desired state for your application (e.g., "I want 3 replicas of my Nginx Pod running, using this image"). The Deployment controller then constantly monitors the cluster to ensure that this desired state is always met. If a Pod crashes, the Deployment will automatically create a new one to maintain the replica count. Deployments also handle rolling updates and rollbacks seamlessly.
Analogy: If a Pod is a single worker, a Deployment is the HR department. You tell HR you need "3 customer service reps for the web app," and HR (the Deployment) makes sure there are always 3 reps working, hiring replacements if someone leaves or gets sick.

Deployments are crucial for scaling, rolling updates, and rollbacks.

4. Services: Connecting to Your Applications

Pods are born and die, and their IP addresses change frequently. How do external users or other applications reliably communicate with your application? Services solve this fundamental networking challenge.

What it does: A Service provides a stable network endpoint (a consistent IP address and port) for a set of Pods. It acts as an internal load balancer, distributing incoming traffic across the healthy Pods associated with it based on labels.
Analogy: A Service is like a stable phone number for your customer service department (Deployment). Even if individual reps (Pods) come and go, the main number (Service IP) remains the same, and calls are routed to whoever is available and healthy.

There are different types of Services, each with a distinct purpose:

ClusterIP: Exposes the Service on an internal IP address within the cluster. This makes the service only reachable from within the cluster, ideal for internal microservices communication.
NodePort: Exposes the Service on a static port on each Node's IP address. This makes the service accessible from outside the cluster using <NodeIP>:<NodePort>, suitable for development or simple external access.
LoadBalancer: Exposes the Service externally using a cloud provider's load balancer. This type automatically provisions an external load balancer (e.g., AWS ELB, Azure Load Balancer, GCP Load Balancer) and assigns it a public IP. This only works on cloud-managed Kubernetes clusters.
ExternalName: Maps the Service to the contents of the externalName field (e.g., a DNS name), by returning a CNAME record. This is used to make an external service (like a database outside the cluster) accessible as if it were internal.

Prerequisites for Your Kubernetes Journey

Before diving into Kubernetes, you should have a basic understanding of:

Containers and Docker: What they are, how to build a simple Docker image, and how to run a container locally. This is fundamental; Kubernetes orchestrates containers, so knowing how they work is a must.
Command Line Interface (CLI): Comfort with basic shell commands (like cd, ls, mkdir, echo) is essential as you'll be interacting with Kubernetes primarily via kubectl.
YAML Syntax: While we'll start simple, Kubernetes heavily relies on YAML files for defining resources. Familiarity with its indentation and key-value pairs will be very helpful as you progress.

Tools for Local Kubernetes Development

To follow along, you'll need a local Kubernetes environment. I recommend Minikube for beginners because it's lightweight and easy to set up. Alternatively, if you already use Docker Desktop, it includes a Kubernetes cluster.

Minikube: A tool that runs a single-node Kubernetes cluster inside a virtual machine (VM) on your laptop. It's excellent for learning and local development, mimicking a real cluster on a smaller scale.
Docker Desktop (with Kubernetes enabled): If you're already using Docker Desktop for Mac or Windows, you can enable Kubernetes directly from its settings. It provides a full-featured, single-node cluster that integrates well with your Docker environment.

For this guide, we'll proceed with Minikube.

Setting Up a Local Kubernetes Cluster with Minikube

Let's get Minikube up and running.

Step 1: Install a Hypervisor

Minikube runs Kubernetes inside a VM. You'll need a VM driver such as VirtualBox, HyperKit (macOS), KVM (Linux), or Hyper-V (Windows). VirtualBox is a popular cross-platform choice.

VirtualBox Installation (if you don't have one): Download and install VirtualBox from https://www.virtualbox.org/wiki/Downloads.

Step 2: Install `kubectl`

kubectl (pronounced "kube-control" or "kube-cuddle") is the command-line tool for running commands against Kubernetes clusters. It's your primary interface for interacting with the cluster.

On macOS (using Homebrew):
If you don't have Homebrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Then install kubectl:
```
brew install kubectl
```
On Windows (using Chocolatey):
If you don't have Chocolatey: Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
Then install kubectl:
```
choco install kubernetes-cli
```

On Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Verify kubectl installation by checking its version:

kubectl version --client

Expected output (versions may vary, but should be similar):

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Step 3: Install Minikube

On macOS (using Homebrew):
```
brew install minikube
```
On Windows (using Chocolatey):
```
choco install minikube
```

On Linux:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

Verify Minikube installation:

minikube version

Expected output (versions may vary, but should be similar):

minikube version: v1.32.0
commit: 18b262b90bc77543265d5069b2d3851b9e6f32e9

Step 4: Start Minikube

Now, let's fire up your local Kubernetes cluster. This might take a few minutes for the first time as Minikube downloads necessary components and sets up the VM.

minikube start --driver=virtualbox

(If you prefer Docker Desktop's built-in Kubernetes, just enable it in Docker Desktop settings and skip minikube start. Ensure kubectl is configured to point to it, which Docker Desktop usually handles automatically.)

Expected output (truncated, but showing key steps):

😄  minikube v1.32.0 on Darwin 14.3.1 (arm64)
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.28.3 preload image minikube-v1.28.3 ...
    > minikube-v1.28.3.tar: 609.43 MiB / 609.43 MiB [===================] 100.00%
🔥  Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

You now have a running Kubernetes cluster!

Step 5: Check Cluster Status

You can verify your cluster is running and that kubectl is connected correctly:

kubectl cluster-info

Expected output (IP addresses and ports will vary):

Kubernetes control plane is running at https://192.168.59.100:8443
CoreDNS is running at https://192.168.59.100:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

You can also see the nodes in your cluster, which should show your single minikube node:

kubectl get nodes

Expected output:

NAME       STATUS   ROLES           AGE     VERSION
minikube   Ready    control-plane   5m20s   v1.28.3

The minikube node is Ready, indicating your cluster is healthy and operational.

Deploying Your First Application on Kubernetes

Now for the exciting part: deploying an application! We'll deploy a simple Nginx web server.

Quick Deployment Tests: `kubectl run` and `kubectl create deployment`

For quick, ad-hoc tests, kubectl offers commands to create resources directly without YAML. It's important to understand how they've evolved:

Creating a single Pod with `kubectl run`

In modern Kubernetes (kubectl v1.18+), kubectl run is primarily used to create a single Pod (and implicitly a ReplicaSet to manage it). This is useful for quickly testing an image or running a temporary command.

kubectl run nginx-single-pod --image=nginx:1.25.3 --port=80

This command tells Kubernetes to create a Pod named nginx-single-pod using the nginx:1.25.3 Docker image and to open port 80.

Expected output:

pod/nginx-single-pod created

You can verify it by running kubectl get pods. Note that while a ReplicaSet might be created in the background to manage this Pod, it's not a full Deployment resource. If you want a full Deployment, read the next section.

Creating a Deployment with `kubectl create deployment`

For a proper, managed application where you want Kubernetes to ensure a certain number of Pods are always running, you use a Deployment. While kubectl run used to create Deployments in older Kubernetes versions, the clear and direct way now is kubectl create deployment.

# This creates a Deployment resource, which then manages Pods.
kubectl create deployment nginx-app --image=nginx:1.25.3

This command creates a Deployment named nginx-app that uses the nginx:1.25.3 image. By default, it will create one replica (one Pod).

Expected output:

deployment.apps/nginx-app created

You can verify the Deployment and its Pod(s):

kubectl get deployment nginx-app
kubectl get pods -l app=nginx-app

Let's clean up these quickly created resources before moving to the recommended YAML method.

kubectl delete pod nginx-single-pod
kubectl delete deployment nginx-app

Expected output:

pod "nginx-single-pod" deleted
deployment.apps "nginx-app" deleted

Deploying with YAML: The Recommended Kubernetes Approach

While kubectl create deployment is quicker, defining your resources in YAML files is the standard and recommended practice for real-world scenarios. It allows for version control, clearer definitions, and easier management of complex applications.

First, ensure any previous nginx-app deployment is deleted:

kubectl delete deployment nginx-app --ignore-not-found=true

Now, create a file named nginx-deployment.yaml with the following content:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2 # We want 2 instances of our Nginx application
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.3 # Using a specific Nginx image version for stability
        ports:
        - containerPort: 80 # The port Nginx listens on inside the container

Explanation of the YAML:

apiVersion: apps/v1: Specifies the API version for the resource. apps/v1 is the current stable version for Deployments.
kind: Deployment: Defines that we are creating a Deployment resource.
metadata.name: nginx-deployment: A unique name for our Deployment within the Kubernetes namespace.
metadata.labels.app: nginx: Labels are key-value pairs used to organize and select resources. This label identifies all resources related to our Nginx application.
spec.replicas: 2: This is crucial! It tells Kubernetes to maintain two identical Pods for this application. If one Pod crashes or is terminated, Kubernetes will automatically create a new one to maintain this desired count.
spec.selector.matchLabels.app: nginx: This selector tells the Deployment controller which Pods it manages. It looks for Pods with the app: nginx label. This linkage is vital.
spec.template: This defines the template for the Pods that the Deployment will create. Any Pod created by this Deployment will conform to this template.
spec.template.metadata.labels.app: nginx: Labels for the Pods themselves, matching the selector above to ensure they are managed by this Deployment.
spec.template.spec.containers: An array defining the containers within each Pod. A Pod can contain multiple containers, though typically it's just one main application container.
- name: nginx: The unique name of our container within the Pod.
image: nginx:1.25.3: The Docker image to use for this container. Always use specific versions (e.g., :1.25.3) in production; avoid latest to ensure consistent deployments.
ports.containerPort: 80: The port that the Nginx application listens on inside the container.

Now, apply this YAML definition to your cluster:

kubectl apply -f nginx-deployment.yaml

Expected output:

deployment.apps/nginx-deployment created

Exposing Your Application with a Service

Your Nginx Deployment is running, but how do we access it from outside the cluster? We need a Service to provide a stable network endpoint.

Create a file named nginx-service.yaml:

# nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx # Selects Pods with the label app: nginx
  ports:
    - protocol: TCP
      port: 80 # The port the Service itself will listen on (inside the cluster)
      targetPort: 80 # The port the Pod container is listening on
  type: NodePort # This type exposes the service on a port on each node

Explanation of the Service YAML:

apiVersion: v1: API version for Services.
kind: Service: Defines that we are creating a Service resource.
metadata.name: nginx-service: A unique name for our Service.
spec.selector.app: nginx: This is the crucial link! The Service uses this label selector to find the Pods managed by our nginx-deployment. Any Pod with the label app: nginx will be a target for this Service.
spec.ports: Defines the network ports for the Service.
- port: 80: This is the port number the Service itself will expose inside the cluster. Other services within the cluster can access it via nginx-service:80.
- targetPort: 80: This is the port on the container that the Service will forward traffic to. In this case, Nginx is listening on port 80.
spec.type: NodePort: We use NodePort here so Minikube can expose the Service on a specific port accessible from your host machine. Kubernetes automatically picks an available port (usually in the 30000-32767 range) on each node.

Apply the Service YAML:

kubectl apply -f nginx-service.yaml

Expected output:

service/nginx-service created

Now you have a fully deployed and exposed Nginx application!

Essential `kubectl` Commands for Kubernetes Beginners

Let's learn how to inspect your deployed resources using kubectl. These commands will be your everyday tools.

1. `kubectl get`: View Resources

This is your most frequent command to see what's running in your cluster.

Get all Deployments in the current namespace:

kubectl get deployments

Output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           3m30s

(READY 2/2 means 2 out of 2 desired Pods are running and ready to serve traffic.)

Get all Pods in the current namespace (including system Pods if you use -A for all namespaces):
```
kubectl get pods
```
Output (Pod names will have a random suffix from the ReplicaSet):
```
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-7f98d9f485-8s92p   1/1     Running   0          3m45s
nginx-deployment-7f98d9f485-l4n9k   1/1     Running   0          3m45s
```
(Note the long, auto-generated names for Pods, indicating they are managed by a Deployment.)

Get all Services in the current namespace:

kubectl get services

Output (the NodePort will vary for you, usually in the 30000-32767 range):

NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP        18m
nginx-service   NodePort    10.106.124.120   <none>        80:30000/TCP   5m

(Look at nginx-service. It has a CLUSTER-IP for internal access and a NodePort, for example 30000, for external access.)

Get all commonly used resources in the default namespace:
```
kubectl get all
```
(This command shows Deployments, Pods, Services, and ReplicaSets related to the default namespace.)

2. `kubectl describe`: Get Detailed Information

When you need more verbose details about a specific resource, describe is your friend. It provides information about resource status, events, and configuration.

Describe the Nginx Deployment:

kubectl describe deployment nginx-deployment

Output (truncated, but includes events, replica status, pod template definition):

Name:                   nginx-deployment
Namespace:              default
CreationTimestamp:      Thu, 29 Feb 2024 10:30:15 -0800
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
...
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx:1.25.3
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deployment-7f98d9f485 (2/2 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  5m3s   deployment-controller  Scaled up replica set nginx-deployment-7f98d9f485 to 2

Describe one of your Nginx Pods:

(Replace nginx-deployment-7f98d9f485-8s92p with one of your actual Pod names from kubectl get pods)
```
kubectl describe pod nginx-deployment-7f98d9f485-8s92p
```
This will show extensive details including events, container status, IP address, node assignment, resource limits, and more.

3. `kubectl logs`: View Container Logs

To debug an application, you often need to see its logs. This command fetches logs from a specific container within a Pod.

View logs for one of your Nginx Pods:
(Replace with your Pod name)

kubectl logs nginx-deployment-7f98d9f485-8s92p

Output (Nginx startup logs):

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to execute files in order:
/docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
/docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

4. Accessing Your Nginx Application

Since we used a NodePort Service with Minikube, you can easily access your application from your host machine. Minikube makes this straightforward:

minikube service nginx-service

This command will automatically open your web browser to the correct URL (e.g., http://192.168.59.100:30000). You should see the "Welcome to nginx!" page served by your containerized application.

5. Cleaning Up Your Resources

When you're done experimenting, it's good practice to delete the resources you created to keep your cluster tidy. This deletes the Deployment, its associated Pods, and the Service.

kubectl delete -f nginx-deployment.yaml
kubectl delete -f nginx-service.yaml

Expected output:

deployment.apps "nginx-deployment" deleted
service "nginx-service" deleted

To stop or completely delete your Minikube cluster:

minikube stop # Stops the VM but keeps the configuration for later reuse.
# OR
minikube delete # Deletes the VM and all Kubernetes configuration, freeing up disk space.

Next Steps: Expanding Your Kubernetes Learning Journey

Congratulations! You've successfully set up a local Kubernetes cluster, deployed a containerized application, exposed it via a Service, and learned essential kubectl commands. This is a monumental first step into the world of container orchestration!

Kubernetes is a vast ecosystem, and this is just the tip of the iceberg. Here's a roadmap for your continued learning:

Deep Dive into YAML: Understand the full structure of Kubernetes resource definitions. Learn about different API versions, common fields, and best practices for writing maintainable YAML manifests.
Storage (Volumes): Learn how to make your application data persistent using various types of Volumes (e.g., hostPath, PersistentVolumeClaim, StorageClass). This is crucial for stateful applications.
Networking (Advanced): Explore more advanced networking concepts like Kubernetes Ingress controllers (for robust HTTP/HTTPS routing, SSL termination, and virtual hosts), Network Policies (for controlling traffic between Pods), and Container Network Interface (CNI) plugins.
Configuration Management (ConfigMaps & Secrets): Learn how to externalize your application configurations using ConfigMaps and securely manage sensitive data like API keys, database credentials, and certificates using Secrets.
Helm: A package manager for Kubernetes that simplifies deploying and managing complex applications. It allows you to define, install, and upgrade even the most intricate Kubernetes applications using "charts." It's an indispensable tool in any serious K8s environment.
Monitoring and Logging: Integrate with popular tools like Prometheus (for metrics collection) and Grafana (for visualization), and centralized logging solutions (e.g., Fluentd, Elasticsearch, Kibana, Loki) for robust observability.
Cloud-Managed Kubernetes: Once comfortable with local K8s, explore managed services like Amazon EKS, Azure AKS, or Google GKE. These services handle the operational burden of managing the Kubernetes control plane for you, allowing you to focus on your applications.
CI/CD Integration: Learn how to integrate Kubernetes deployments into your continuous integration and continuous delivery pipelines, enabling automated, fast, and reliable software releases.

Remember, the best way to learn is by doing. Experiment, break things, and fix them. The Kubernetes community is huge and incredibly supportive, so don't hesitate to seek help when you get stuck.

FAQ

Q1: What's the difference between a Pod and a Container?

A container (like a Docker container) is a single, isolated process or set of processes with its own filesystem, CPU, and memory limits. A Pod, in Kubernetes, is the smallest deployable unit and can contain one or more containers. These containers within a Pod share the same network namespace (meaning they share an IP address and port space) and can share storage volumes. While you can't directly deploy a single container to Kubernetes, you always deploy a Pod that then runs your container(s).

Q2: Why not just use Docker Compose for orchestration?

Docker Compose is excellent for defining and running multi-container Docker applications on a single host. It's perfect for local development or small, single-server deployments where you manage a few containers. Kubernetes, on the other hand, is designed for distributing and orchestrating containerized applications across a cluster of many machines. It provides advanced features like automatic scaling, self-healing, rolling updates, and intelligent resource scheduling across multiple nodes that Docker Compose doesn't offer. If you need true high availability, scalability, and resilience across a distributed infrastructure, Kubernetes is the powerful solution.

Q3: Is Kubernetes only for large companies?

Absolutely not. While Kubernetes shines in large-scale, complex environments, its benefits in terms of reliability, automation, and portability are valuable for projects and teams of all sizes. Even a small team or individual developer can leverage Kubernetes to streamline deployments, ensure their applications are robust, and simplify operations, often by starting with a managed cloud service to reduce the initial operational overhead. The learning curve is real, but the investment often pays off quickly in terms of efficiency and stability.

Q4: What's the cost of running Kubernetes?

The cost varies significantly depending on your setup. Running a local Minikube cluster on your laptop is free (minus electricity). Running a self-managed Kubernetes cluster on bare metal or VMs will incur infrastructure costs (servers, networking, storage) and significant operational overhead (managing the control plane, upgrades, security patches). Cloud-managed Kubernetes services (EKS, AKS, GKE) typically charge for the underlying compute resources (worker nodes, storage, network egress) and sometimes a small fee for the control plane itself. The biggest cost factor often comes down to the expertise required to manage it effectively, whether that's in-house talent or external consultants.

Q5: How is Kubernetes related to Docker?

Docker is primarily a containerization technology used to package applications and their dependencies into portable containers. Kubernetes is an orchestration platform that manages and deploys these Docker (or other OCI-compliant) containers at scale across a cluster of machines. You can think of Docker as the engine that creates the individual cars (containers), and Kubernetes as the sophisticated traffic controller and fleet manager that ensures all the cars are running efficiently on the right roads, scaling them up or down, and rerouting them if there are issues.

Conclusion

You've just taken your first concrete steps into the world of Kubernetes, deploying a real application on a functional cluster. This foundational knowledge of Pods, Deployments, Services, and kubectl commands is essential for anyone looking to master modern infrastructure and embrace cloud-native development.

While Kubernetes has a reputation for having a steep learning curve, the benefits it offers in terms of scalability, reliability, and automation are transformative for application management. Don't be intimidated by its perceived complexity; approach it incrementally, focusing on one core concept at a time. The hands-on experience you've gained today is far more valuable than hours of theoretical reading.

Your next actionable steps should be to:

Revisit the nginx-deployment.yaml and nginx-service.yaml files. Try changing the number of replicas in the Deployment, or the Nginx image version, and re-apply them (kubectl apply -f filename.yaml) to see how Kubernetes updates your application with zero downtime.
Experiment with kubectl delete deployment <name> and kubectl delete service <name> to understand the cleanup process and how Kubernetes removes associated resources.
Dive into the official Kubernetes documentation. It's incredibly comprehensive, well-maintained, and a fantastic resource for deepening your understanding.
Explore the Minikube dashboard: Run minikube dashboard in your terminal to open a web UI for your cluster. This provides a visual overview of your deployments, pods, and other resources.

Keep building, keep learning, and before you know it, you'll be confidently orchestrating your applications like a seasoned pro.

Testing in Production: Guide to Progressive Delivery

DevOps Start — Tue, 14 Apr 2026 14:40:35 +0000

This guide was originally published on devopsstart.com. Learn how to decouple deployment from release to reduce risk using progressive delivery strategies.

Introduction

You've spent weeks polishing a feature in a staging environment that is a mirror image of production. The tests pass, the QA team gives the thumbs up, and the deployment is scheduled for 2:00 AM. Then, the moment the code hits live traffic, everything collapses. A database deadlock occurs because the production dataset is 1,000 times larger than staging. A race condition emerges because of a specific traffic pattern that only exists in the wild. You realize that your "perfect" staging environment was a lie.

The hard truth of modern distributed systems is that production is the only environment that truly matters. Trying to replicate the complexity of a live global cluster in a pre-production environment is a losing game of whack-a-mole. To solve this, elite engineering teams have shifted toward Progressive Delivery. This isn't about being reckless with user data; it's about acknowledging that the only way to truly verify a change is to test it against real traffic, but in a way that minimizes the blast radius.

In this guide, you'll learn how to decouple deployment from release, implement canary strategies, and build the observability loops required to make testing in production safer than traditional releases. You'll move from a mindset of preventing all failures to one of rapid detection and recovery.

The Fallacy of Staging and the Shift to MTTR

Staging environments are often treated as the ultimate safety net, but they are fundamentally flawed. They suffer from "environment drift," where the configuration, data volume and network topology slowly diverge from production. Even if you use a Terraform testing pyramid to ensure your infrastructure is consistent, you cannot simulate the unpredictability of human users or the sheer volume of a production database.

When you rely solely on pre-production testing, you are optimizing for Mean Time Between Failures (MTBF). You're trying to ensure that a crash never happens. In a complex microservices architecture, this is impossible. I've seen this fail in clusters with >50 nodes where the network jitter alone creates failure modes that simply don't exist in a 3-node staging environment.

Instead, the industry is shifting toward optimizing Mean Time to Recovery (MTTR). The goal is no longer "zero bugs," but "zero prolonged outages."

To achieve this, you must stop treating a "deployment" (the act of moving binaries to a server) as a "release" (the act of exposing a feature to a user). By separating these two events, you can push code to production in a dormant state, verify its health with internal users and then gradually roll it out. This requires a fundamental change in how you handle your application logic. You no longer write code that is either "on" or "off"; you write code that is conditionally active based on a runtime toggle.

For example, consider a new pricing algorithm. Instead of replacing the old one, you wrap the new logic in a feature flag.

# Example using a conceptual feature flag client (e.g., Unleash or LaunchDarkly)
import feature_flags

def calculate_price(order):
    # The feature flag is checked at runtime, not compile time
    # This allows instant kill-switching without a redeploy
    if feature_flags.is_enabled("new_pricing_engine", user_id=order.user_id):
        return new_pricing_logic(order)

    return legacy_pricing_logic(order)

def new_pricing_logic(order):
    # New logic that might have a bug under high load
    return order.total * 0.95

def legacy_pricing_logic(order):
    return order.total

In this scenario, the code is deployed to 100% of your servers, but the risk is 0% until you flip the switch for a small subset of users.

Implementing Canary Releases with Service Mesh

While feature flags handle application logic, Canary Releases handle the network. A canary release involves routing a small percentage of live traffic to a new version of your service while the majority remains on the stable version. If the canary version shows an increase in 5xx errors or latency spikes, the traffic is instantly routed back to the stable version.

To do this effectively at scale, you need a service mesh or a sophisticated ingress controller. Using Istio v1.21, you can define a VirtualService that splits traffic based on weights. This allows you to test the "plumbing" of your application (memory leaks, connection pool exhaustion, CPU spikes) which feature flags often miss.

Here is how you configure a 90/10 traffic split between the stable and canary versions of a service.

# Istio VirtualService for Canary Traffic Splitting
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: product-page-route
spec:
  hosts:
  - product-page.example.com
  http:
  - route:
    - destination:
        host: product-page-service
        subset: v1
      weight: 90
    - destination:
        host: product-page-service
        subset: v2
      weight: 10

To make this work, you also need a DestinationRule to define the subsets based on Kubernetes labels.

# Istio DestinationRule to define version subsets
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: product-page-destination
spec:
  host: product-page-service
  subsets:
  - name: v1
    labels:
      version: v1.0.0
  - name: v2
    labels:
      version: v1.1.0

Once this is applied, you monitor your telemetry. If you see the canary version (v2) throwing errors, you don't need to perform a full redeployment. You simply update the VirtualService weight back to 100/0.

Advanced Safety: Shadow Traffic and the Safety Matrix

Canary releases are great, but they still expose real users to potential bugs. For high-risk changes, such as a database migration or a critical security patch, you should use Shadow Traffic (also known as Dark Launching).

Shadowing mirrors live traffic. When a request hits your production environment, the load balancer sends it to the stable version (which returns the response to the user) and asynchronously sends a copy of that request to the new version. The new version processes the request, but its response is discarded. You compare the results of the stable version and the shadow version in your logs. If the shadow version produces a different result or crashes, you've found a bug without a single user ever seeing an error page.

Because different changes carry different risks, you shouldn't use the same testing method for everything. Use this Safety Matrix to decide your approach:

Change Type	Risk Level	Recommended Method	Primary Goal
UI/UX tweak, CSS change	Low	Feature Flags	User feedback, A/B testing
New API endpoint, Minor logic	Medium	Canary Release	Performance, Error rates
Database Schema change, Core Engine	High	Shadow Traffic	Data correctness, Latency
Global Config change	Critical	Feature Flags + Canary	Blast radius control

Imagine you are migrating from a legacy SQL query to a new NoSQL implementation. A canary release is too risky because a bug could corrupt production data. Instead, you shadow the traffic. You send the request to both the SQL and NoSQL paths. You log the results of both. If the NoSQL path returns a "null" where the SQL path returned a "user_id," you know your migration logic is flawed.

This approach requires high-cardinality observability. You cannot rely on a simple "CPU usage" graph. You need distributed tracing (e.g., Jaeger or Honeycomb) to see exactly which request failed in the shadow path and why. You need to be able to query: "Show me all requests where the shadow response differed from the production response by more than 5%."

Best Practices for Testing in Production

Transitioning to progressive delivery is as much a cultural shift as it is a technical one. If you don't have the right guardrails, "testing in production" becomes a euphemism for "breaking things for users."

Automate the Rollback Loop: Do not rely on a human to watch a dashboard and click "rollback." Link your monitoring system (e.g., Prometheus) to your deployment tool (e.g., ArgoCD). If the 99th percentile latency for the canary version exceeds 500ms for more than two minutes, the system should automatically revert the traffic weight to 0%. This can reduce the window of impact from hours to seconds.
Define Strict Error Budgets: Establish a Service Level Objective (SLO). If your error budget for the month is 0.1% and a canary release consumes 0.05% of that budget in ten minutes, stop the rollout immediately. This removes the emotional tension between developers wanting to move fast and SREs wanting stability.
Start with Internal "Dogfooding": Your first "canary" should always be your own employees. Use headers or cookie-based routing to ensure that only users with an @company.com email address hit the new version.
Keep Feature Flags Short-Lived: Feature flags introduce technical debt. Once a feature is 100% rolled out and stable, create a ticket to remove the flag logic from the code. A codebase littered with old if (flag_enabled) statements becomes an unmaintainable nightmare.
Invest in High-Cardinality Metrics: Standard metrics tell you that something is wrong. High-cardinality metrics (including user_id, region and version_id) tell you who is affected. Without this, you cannot effectively limit the blast radius.

FAQ

Is testing in production just a fancy way of saying "we don't test our code"?
No. Testing in production is the final stage of a rigorous pipeline. You still run unit tests, integration tests and contract tests in CI. Progressive delivery addresses the "unknown unknowns" that only appear under real-world load and state, which no amount of pre-production testing can fully uncover.

How do I handle database migrations with canary releases?
Database changes are the hardest part of progressive delivery. You must use "expand and contract" patterns. First, add the new column or table (Expand) while keeping the old one. Deploy the code that writes to both but reads from the old. Then, migrate the data. Finally, deploy the code that reads from the new and delete the old column (Contract). Never perform a destructive database change in a single deployment.

What happens if a shadow test modifies data?
Shadow traffic must be read-only. If the service you are shadowing performs writes, you must use a "mock" or "shadow" database that mimics production but doesn't affect real users. Alternatively, use a transactional wrapper that always rolls back the transaction at the end of the shadow request.

Can I use this approach for a small team with limited resources?
Yes. You don't need a full service mesh like Istio to start. You can start with a simple feature flag library in your code or a basic weighted load balancer at the DNS level. The mindset shift—decoupling deployment from release—is free and provides immediate value.

Conclusion

Testing in production is not about recklessness; it is about precision. By accepting that staging is an imperfect proxy for reality, you can implement strategies like feature flags, canary releases and traffic shadowing to reduce the blast radius of any given failure. The transition from optimizing for MTBF to optimizing for MTTR allows your team to deploy more frequently with significantly less anxiety.

To get started, don't try to overhaul your entire pipeline overnight. Start with one low-risk service. Implement a basic feature flag for a UI change, then move to a 5% canary rollout for a backend API. Once you have the observability in place to detect failures in seconds rather than hours, you'll find that the "safest" way to deploy is to do it progressively in the environment where it actually matters.

Your next steps are clear: identify your most unstable service, set up a basic traffic split and define your first automated rollback trigger. Stop fearing production and start using it as your most accurate testing tool.

Kubernetes Test Automation: Implementing a Shift-Left Strategy

DevOps Start — Tue, 14 Apr 2026 14:35:31 +0000

Stop letting a shared staging environment bottleneck your deployments. This guide, originally published on devopsstart.com, shows you how to implement a shift-left strategy using ephemeral Kubernetes namespaces.

Introduction

Most DevOps teams treat their staging environment like a crowded subway car during rush hour. Everyone is trying to push their features into one shared space, leading to deployment queues, "flaky" tests caused by state contamination and the inevitable "it worked in staging but failed in prod" nightmare. When five different developers deploy five different versions of a microservice to the same namespace, you aren't testing your code; you're testing your ability to manage chaos. This is the shared staging bottleneck, and it's the primary reason feedback loops in Kubernetes pipelines stretch from minutes to days.

To solve this, you need a shift-left strategy. Shift-left testing isn't just about writing unit tests earlier; it's about moving the entire environment lifecycle into the CI pipeline. By utilizing ephemeral, Kubernetes-native environments, you can provide every single Pull Request (PR) with its own isolated namespace. This ensures that tests run against a clean slate, removing the noise of other developers' changes.

In this article, you'll learn how to architect a Kubernetes test automation strategy that replaces the shared staging monolith with dynamic preview environments, manages database state without slowing down your pipeline and automates the teardown process to keep your cloud bill under control.

The Shared Staging Bottleneck and the K8s Testing Pyramid

The traditional approach to testing in Kubernetes involves a linear path: Dev $\rightarrow$ Staging $\rightarrow$ Production. The problem is that "Staging" becomes a dumping ground. When a test fails in a shared environment, you don't know if it's because of your code, a configuration change someone else made ten minutes ago or a database record that was deleted by another team. This environment drift creates a false sense of security or, more often, a culture of ignoring "flaky" tests.

To break this cycle, you must adapt the traditional testing pyramid for the cloud-native era. In a Kubernetes-native world, the pyramid shifts from focusing on code coverage to focusing on infrastructure and integration boundaries.

At the base, you have unit tests running in the CI runner. Above that are integration tests where the application runs in a container alongside its dependencies (like Redis or PostgreSQL) using tools like Testcontainers. The critical middle layer is the ephemeral environment: a fully functional, short-lived instance of your stack deployed into a dedicated Kubernetes namespace. Finally, the top is reserved for smoke tests and progressive delivery in production.

For those managing the infrastructure beneath these tests, adopting a rigorous approach to the underlying code is vital. Just as you test your application, you should follow a dedicated framework for testing infrastructure as code to ensure your VPCs and clusters are stable before the application pods even land.

When you implement this pyramid, your CI pipeline should follow this sequence:

# Step 1: Run unit tests in the CI runner
go test ./... -v

# Step 2: Run integration tests using Testcontainers or Docker Compose
# This validates the app can talk to its immediate dependencies
docker-compose up -d db redis
go test ./integration/...
docker-compose down

# Step 3: Deploy to an ephemeral K8s namespace for E2E tests
# Creating a unique namespace per PR (e.g., pr-123)
kubectl create namespace pr-123
helm install pr-123 ./charts/my-app \
  --namespace pr-123 \
  --set image.tag=sha-abc123 \
  --wait

By shifting the heavy lifting to ephemeral namespaces, you eliminate the queue. Developers no longer wait for "their turn" to use staging. They get a unique URL, run their E2E suite and merge when the green checkmark appears.

Implementing Ephemeral Environments with GitOps

The most effective way to manage these dynamic environments is through a GitOps workflow. Instead of having a CI script manually run kubectl apply, you use a tool like ArgoCD (v2.11.0) to synchronize the state of a Git repository with your cluster. In a "Namespace-per-PR" pattern, the CI pipeline creates a new folder in a "preview" Git repository or updates a Helm values file that tells ArgoCD to spin up a new application instance.

The magic happens when you combine Kubernetes namespaces with a dynamic Ingress controller. By using a wildcard DNS record (e.g., *.preview.example.com), you can map pr-123.preview.example.com directly to the service in the pr-123 namespace. This provides an instant, shareable link for QA and stakeholders to verify the feature in a live environment.

Here is a configuration example of how to structure a Helm values file for an ephemeral environment deployment:

# values-pr-123.yaml
global:
  environment: preview
  prNumber: "123"

ingress:
  enabled: true
  # Dynamic hostname based on PR number
  hostname: pr-123.preview.example.com

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

# Use a lightweight version of the database for previews
database:
  type: postgres
  version: "16.2"
  storageSize: 1Gi

To deploy this via a CI pipeline, you would execute:

# Deploy the application using Helm v3.14.0
helm upgrade --install my-app-pr-123 ./charts/my-app \
  --namespace pr-123 \
  -f values-pr-123.yaml \
  --wait --timeout 5m

# Verify the deployment is healthy
kubectl get pods -n pr-123

This approach ensures that the environment is an exact replica of production architecture but isolated in terms of traffic and data. If the deployment fails, it doesn't take down the rest of the team. You can simply delete the namespace and start over. For more details on how Kubernetes handles these resource definitions, refer to the official Kubernetes Documentation.

Managing Test Data and the Lifecycle Teardown

The "Achilles' heel" of ephemeral environments is the database. You cannot spin up a 500GB production database clone for every PR; it's too slow and too expensive. Instead, you have three viable options: synthetic data generators, lightweight anonymized snapshots or a "Golden Image" database.

The "Golden Image" approach is usually the winner for speed. You maintain a separate, sanitized database image that contains the minimum viable dataset needed for tests to pass. When the ephemeral environment starts, you deploy this image as a Kubernetes StatefulSet.

# db-snapshot.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-db
  namespace: pr-123
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app: test-db
  template:
    metadata:
      labels:
        app: test-db
    spec:
      containers:
      - name: postgres
        image: my-registry.com/test-db-snapshot:v1.2.0 # Pre-seeded sanitized data
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_PASSWORD
          value: "password123"

Creating these environments is easy, but cleaning them up is hard. If you forget to delete a namespace, you'll find your cluster exhausted of IP addresses and your cloud bill skyrocketing. You should never rely on developers to manually run kubectl delete namespace.

The professional solution is a TTL (Time-to-Live) controller. You can implement this using a simple Kubernetes Custom Resource Definition (CRD) or a lightweight operator that watches for a ttl annotation. If a namespace has an annotation preview.example.com/ttl: "24h", the controller deletes the namespace once the timer expires.

If you don't have time to build an operator, integrate the teardown into your Git event pipeline:

# Triggered by a "PR Closed" or "PR Merged" event in GitHub/GitLab
echo "Cleaning up environment for PR 123..."
kubectl delete namespace pr-123 --wait=false

# Verify deletion (may take a few minutes due to finalizers)
kubectl get ns pr-123

By automating the teardown, you treat your infrastructure as truly disposable. This allows you to be more aggressive with your testing strategy, knowing that the cost of a leaked environment is capped by the TTL.

Best Practices for K8s Test Automation

Implementing a shift-left strategy requires a mindset shift. You are no longer managing a "server"; you are managing a "factory" that produces environments. Follow these guidelines to keep your pipeline stable:

Use Resource Quotas Strictly: Ephemeral environments are prone to "resource leakage." Apply a ResourceQuota to every preview namespace. This prevents a single buggy PR from consuming all the CPU on your nodes and starving other teams.
Avoid Persistent Volumes (PVs) where possible: Use emptyDir or lightweight ephemeral disks for test databases. Mounting network storage (like AWS EBS) adds significant latency to the environment spin-up time and often leads to "volume already attached" errors during rapid teardowns.
Implement Readiness Probes: Your E2E tests must not start the second helm install finishes. Use readinessProbes in your pods and a "wait-for-it" script in your pipeline to ensure the application is actually accepting traffic before the test suite kicks off.
Shift Observability Left: Don't wait for production to see your logs. Deploy a lightweight Prometheus and Loki instance that aggregates data from all preview namespaces. If a test fails, you should be able to jump straight to the logs of that specific PR without hunting through a centralized logging system.
Automate the Merge Gate: Set your GitHub/GitLab settings to require a "passed" status from the ephemeral environment suite before the Merge button is enabled. This transforms your tests from a "suggestion" into a "requirement."

Once your ephemeral environments are stable, you can begin exploring progressive delivery patterns to handle the final 5% of risk that can only be caught by real user traffic.

FAQ

How do I handle secret management in ephemeral environments?

Don't store secrets in your Git repository, even for test environments. Use a tool like HashiCorp Vault or AWS Secrets Manager. The best pattern is to use the External Secrets Operator (ESO). You define a SecretStore at the cluster level and each ephemeral namespace gets an ExternalSecret object that fetches the necessary keys based on the PR number or a generic "test" profile.

Isn't spinning up a whole namespace for every PR too expensive?

It depends on the scale. If you have 100 developers pushing 5 PRs a day, you could have 500 namespaces. To mitigate cost, use a dedicated "Preview Cluster" with Cluster Autoscaler enabled and use Spot Instances (AWS) or Preemptible VMs (GCP). Since these environments are disposable, a node termination is a minor inconvenience. I've seen this reduce preview environment costs by up to 70% compared to on-demand instances.

How do I deal with external dependencies (like a 3rd party API) that aren't in K8s?

You cannot spin up a new instance of Stripe or Twilio for every PR. In these cases, use API mocking. Tools like Prism (for OpenAPI) or WireMock can be deployed as pods within your ephemeral namespace. Your application's configuration is then pointed to the mock service instead of the real API. This ensures your tests are deterministic and don't trigger real-world side effects.

What happens if the `kubectl delete namespace` command hangs?

Kubernetes namespaces sometimes get stuck in a Terminating state, usually because of finalizers on resources that can't be deleted. To fix this, you need a cleanup script that identifies the blocking resource (often a stubborn PV or a custom resource) and patches the finalizer to null. This is a common operational headache that requires a robust cleanup job in your CI/CD pipeline.

Conclusion

Moving away from a shared staging environment is the most impactful change you can make to your Kubernetes delivery pipeline. By implementing a shift-left strategy centered on ephemeral namespaces, you replace the anxiety of "breaking staging" with the confidence of isolated, reproducible testing. You've learned how to structure a K8s testing pyramid, automate environment creation via GitOps and manage the critical lifecycle of test data and teardowns.

The transition isn't instant. Start by automating the creation of a single preview namespace for a high-traffic service. Once you've nailed the "create $\rightarrow$ test $\rightarrow$ destroy" loop, expand it to the rest of your microservices.

Your immediate next step should be to audit your current staging environment: identify the top three causes of "flaky" tests and determine if isolation via a dedicated namespace would have prevented them. Once you have that data, begin implementing the TTL controller to ensure your experiment doesn't break your budget.

GitOps Testing Strategies: Validate Deployments with ArgoCD

DevOps Start — Tue, 14 Apr 2026 14:30:26 +0000

Stop relying on 'blind syncs' and start validating your GitOps deployments. This guide, originally published on devopsstart.com, shows you how to bridge the gap between CI and CD using ArgoCD.

Introduction

You've likely experienced the "blind sync" nightmare. Your CI pipeline is green, your unit tests passed, and your GitHub Action successfully merged the PR into the main branch. ArgoCD sees the change, syncs the manifest to the cluster, and reports a healthy status because the pods are running. Then, five minutes later, your monitoring alerts scream. The application is crashing because of a missing environment variable or a database schema mismatch that only manifests at runtime.

This happens because there is a fundamental gap between CI testing and GitOps validation. Traditional CI tests the artifact (the image), but GitOps manages the state (the manifest). When you treat the Git sync as the final step of your pipeline, you're essentially deploying and hoping for the best.

In this guide, you'll learn how to bridge this gap by implementing a closed-loop validation strategy. We will move beyond simple "sync and pray" deployments by integrating shift-left manifest validation, ArgoCD sync hooks for pre-deployment checks, and Argo Rollouts for automated metric-based analysis. By the end, you'll know how to ensure that a "Healthy" status in ArgoCD actually means your application is functioning correctly for your users.

Closing the GitOps Testing Gap with Shift-Left Validation

The first step to preventing broken deployments is to stop them from ever reaching your Git repository. In a GitOps workflow, the Git repo is the single source of truth. If a developer commits a manifest with a typo in the API version or a missing required field, ArgoCD will try to apply it, fail, and leave your cluster in a "Degraded" state.

To fix this, you must implement shift-left manifest validation. This means moving the validation of your Kubernetes YAMLs into the CI pipeline, before the merge occurs. You should not rely on the Kubernetes API server to tell you that your YAML is invalid. Instead, use tools like kube-linter (v0.14.0) or kubeval to enforce security policies and schema correctness.

For example, you can integrate kube-linter into a GitHub Action to catch common misconfigurations, such as running containers as root or missing resource limits.

# Install kube-linter v0.14.0
curl -L https://github.com/stackrox/kube-linter/releases/download/v0.14.0/kube-linter_linux_amd64.tar.gz | tar xz
sudo mv kube-linter /usr/local/bin/

# Run linter against your manifests directory
kube-linter lint path/to/manifests/

If a developer forgets to define CPU limits, the output will look like this:

path/to/manifests/deployment.yaml:12: Error: containers should have CPU limits defined

By failing the build here, you prevent the "blind sync" from even starting. This approach is a core part of a broader Kubernetes test automation strategy, ensuring that only syntactically and logically sound manifests are promoted to the environment. It transforms your Git repository from a place where "any YAML goes" into a curated set of validated configurations.

Implementing Pre-Sync Hooks for Runtime Readiness

Even a syntactically perfect manifest can fail if the environment isn't ready. A classic example is the database migration. If your application pod starts before the database schema is updated, the app will crash-loop, causing a deployment failure that ArgoCD might struggle to recover from automatically.

ArgoCD (v2.10.0) solves this using Sync Waves and Hooks. Sync Waves allow you to order the application of resources, while Hooks allow you to run specific jobs at certain points in the sync process (PreSync, Sync, PostSync).

To handle database migrations, you should use a PreSync hook. This ensures the migration job completes successfully before ArgoCD attempts to update the Deployment. If the PreSync job fails, ArgoCD stops the sync process entirely, preventing the broken application version from ever reaching the pods.

Here is a production-ready example of a migration job using a PreSync hook:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate-job
  annotations:
    # This tells ArgoCD to run this job before syncing other resources
    argocd.argoproj.io/hook: PreSync
    # This ensures the job is deleted after it succeeds to keep the cluster clean
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: my-app-migration:v1.2.3
        command: ["/bin/sh", "-c", "python manage.py migrate"]
        envFrom:
        - secretRef:
            name: db-credentials
      restartPolicy: OnFailure
  backoffLimit: 3
  # Prevent the hook from hanging indefinitely
  activeDeadlineSeconds: 300

By using argocd.argoproj.io/hook: PreSync, you create a synchronous gate in an otherwise asynchronous GitOps process. You can find more detailed configuration options in the official ArgoCD documentation.

The trade-off here is deployment speed. Adding hooks increases the time it takes for a change to go from Git to "Healthy." However, the cost of a few extra minutes is negligible compared to the cost of a production outage caused by a failed schema migration.

Advanced Validation with Argo Rollouts and AnalysisTemplates

Once the pods are running, the "Healthy" status in ArgoCD only means the Liveness and Readiness probes passed. It doesn't mean the application is actually working. A pod can be "Ready" but returning 500 errors for every single request because of a bad configuration in the application logic.

To solve this, you need progressive delivery. Instead of a hard cut-over, you use Argo Rollouts (v1.6.0) to implement Canary deployments combined with AnalysisTemplates. An AnalysisTemplate allows you to define a set of metrics (usually from Prometheus) that must remain within a certain threshold during the rollout. If the error rate spikes, Argo Rollouts automatically rolls back the deployment without human intervention.

First, define the AnalysisTemplate to check for HTTP 500 errors:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    successCondition: result[0] >= 0.95
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(irate(http_requests_total{service="{{args.service-name}}", status!~"5.*"}[2m]))
          /
          sum(irate(http_requests_total{service="{{args.service-name}}"}[2m]))

Next, integrate this into your Rollout strategy:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-web-app
spec:
  replicas: 5
  strategy:
    canary:
      analysis:
        templates:
        - templateName: success-rate
      steps:
      - setWeight: 20
      - pause: { duration: 5m }
      - setWeight: 50
      - pause: { duration: 5m }

In this configuration, Argo Rollouts shifts 20% of traffic to the new version and then monitors the Prometheus query for five minutes. If the success rate drops below 95%, the rollout is marked as failed and instantly reverts to the stable version. This is the gold standard for testing in production via progressive delivery, as it limits the "blast radius" of a bad deployment.

The real power here is the automated feedback loop. You aren't waiting for a user to report a bug; the infrastructure is observing its own health and taking corrective action in real-time.

Best Practices for GitOps Testing

Implementing these tools is only half the battle. To make your GitOps testing loop robust, follow these operational patterns:

Decouple Configuration from Code: Never store environment-specific secrets in your Git manifests. Use an external secret manager (like HashiCorp Vault or AWS Secrets Manager) and integrate it with the External Secrets Operator. This ensures that your testing templates remain generic while the actual values are injected at runtime.
Implement Automated Smoke Tests: Don't rely solely on metrics. Create a Tekton pipeline (v0.50.0) that triggers a suite of smoke tests (e.g., using Playwright or Postman) immediately after ArgoCD signals a successful sync. If the smoke tests fail, the pipeline should trigger an API call to ArgoCD to roll back the application.
Version Everything: Ensure your images use specific SHA tags or semantic versions, not latest. If you use latest, ArgoCD may not detect a change in the image, and your testing loop will be bypassed entirely.
Set Up Alerting for Sync Failures: Use ArgoCD Notifications to send alerts to Slack or Microsoft Teams when a sync fails or a hook crashes. A failed PreSync job is a critical event that requires immediate developer attention.
Test Your Rollbacks: A rollback mechanism is useless if it's never tested. Periodically induce a failure in a staging environment to ensure that your AnalysisTemplates actually trigger the rollback as expected.

FAQ

How do I handle test data without polluting production manifests?

The best approach is to use Kustomize overlays. Create a base directory for your core manifests and an overlays/test directory for your testing environment. In the test overlay, you can add specific ConfigMaps for mock API endpoints or test database connection strings. When ArgoCD syncs the test environment, it merges the base with the test overlay, ensuring production remains clean.

What happens if a PreSync hook hangs indefinitely?

By default, a Kubernetes Job will run until it completes or hits the backoffLimit. To prevent a hanging hook from blocking your pipeline, always define an activeDeadlineSeconds in your Job spec. This tells Kubernetes to terminate the pod if it runs longer than a specified time, which then allows ArgoCD to mark the sync as failed.

Can I run integration tests between the PreSync and Sync phases?

Yes, but it requires a slightly different architecture. Since the PreSync hook runs before the main application pods are updated, you cannot test the new application code. To run integration tests against the new code before it hits 100% of users, you must use Argo Rollouts. Run your tests during the pause steps of the Canary deployment, targeting the canary service endpoint.

Should I use Tekton or GitHub Actions for post-sync validation?

If your tests require access to internal cluster resources (like a private database or an internal API), Tekton is superior because it runs natively inside your Kubernetes cluster. If your tests are purely external (like hitting a public URL), GitHub Actions is simpler to manage. For high-compliance environments, Tekton is the preferred choice for security.

Conclusion

The "blind sync" is a common failure point in GitOps, but it is entirely avoidable. By shifting manifest validation left with kube-linter, managing dependencies with ArgoCD PreSync hooks, and automating runtime validation with Argo Rollouts AnalysisTemplates, you turn your deployment process from a leap of faith into a scientific process.

The key is to stop treating the Git sync as the end of the pipeline. Instead, view the sync as the beginning of the validation phase. Your goal should be to create a closed loop where the system observes its own health and reacts automatically to regressions.

To get started, take these three immediate steps: first, add a linter to your CI pipeline to catch schema errors. Second, move your database migrations into a PreSync hook. Finally, identify your top three critical business metrics and implement an AnalysisTemplate to monitor them during your next deployment. This transition will significantly reduce your Mean Time to Recovery (MTTR) and increase your overall deployment confidence.

Forem: DevOps Start

Essential kubectl Commands Cheat Sheet

Pod Management

Deployments

Services and Networking

Debugging

Context and Config

Debug Kubernetes CrashLoopBackOff in 30 Seconds

The Problem

The Solution

Combine with describe for the full picture

Common Exit Codes

Why It Works

Rapid Rollback: `kubectl set image` for Urgent Fixes

Introduction

Understanding kubectl set image

The Urgent Rollback Scenario

Identifying the Previous Image Tag

Executing the kubectl set image Rollback

Important Considerations

Conclusion

How to Set Up Argo CD GitOps for Kubernetes Automation

Introduction

Prerequisites

Overview

Step 1: Installing Argo CD

Step 2: Initial Access and Authentication

Step 3: Connecting Your Git Repository

Step 4: Creating Your First GitOps Application

Component Breakdown

Step 5: Handling Configuration Drift

Step 6: Scaling with ApplicationSets

Step 7: Integrating the Full CI/CD Pipeline

Example GitHub Action snippet for manifest update

Troubleshooting

1. Application stuck in "Progressing" status

2. Repository Connection Failed

3. "OutOfSync" loop

FAQ

Conclusion

How to Configure Advanced Argo CD Sync Policies for GitOps

Prerequisites

Overview

Step 1: Implementing Automated Pruning and Self-Healing

Understanding Pruning and Self-Healing

Configuration

Testing the Policy

Step 2: Mastering Advanced Sync Options

Using Replace=true

ApplicationSet Level Policies

Step 3: Implementing Sync Waves and Hooks

Sync Waves

Sync Hooks

Step 4: Designing Robust Rollback Strategies

Strategy A: The Git-based Rollback (Pure GitOps)

Strategy B: The Argo CD UI/CLI Rollback (Emergency Fast-Track)

The Decision Matrix

Step 5: Implementing Custom Health Checks

Defining a Lua Health Check

Step 6: Managing Sync Windows and Maintenance Periods

The Label-Based Freeze Approach

Troubleshooting

Issue 1: Resource "Flickering" (The Sync Loop)

Issue 2: Pruning Deleted Critical Resources

Issue 3: Sync Wave Hanging

FAQ

Conclusion

Deploy an EKS Cluster with Terraform

Step 1: Set Up the Project Structure

Step 2: Configure the AWS Provider

Step 3: Create the VPC

Step 4: Deploy the EKS Cluster

Step 5: Apply and Connect

Cleanup

Troubleshooting

How to Automate Terraform Reviews with GitHub Actions

Introduction

Prerequisites

Overview

Step 1: Installing the CodeRabbit GitHub App

Understanding `kubectl set image`

Executing the `kubectl set image` Rollback

Step 2: Configuring the `.coderabbit.yaml` for Terraform

Step 6: Integrating with `terraform plan` for Risk Analysis

Step 2: Install `kubectl`

Quick Deployment Tests: `kubectl run` and `kubectl create deployment`

Creating a single Pod with `kubectl run`

Creating a Deployment with `kubectl create deployment`

Essential `kubectl` Commands for Kubernetes Beginners

1. `kubectl get`: View Resources

2. `kubectl describe`: Get Detailed Information

3. `kubectl logs`: View Container Logs