Forem: DevOps Start

How to Build a Developer Control Plane with Backstage

DevOps Start — Tue, 05 May 2026 14:25:21 +0000

Looking to reduce cognitive load for your engineering teams? This tutorial, originally published on devopsstart.com, walks you through building a developer control plane using Backstage.

An Internal Developer Platform (IDP) is a centralized control plane that gives your development teams a paved road for building, deploying and managing software. Instead of managing dozens of different tools and CLIs, developers get a single, curated interface for everything from creating a new microservice to checking its CI/CD status or viewing its documentation. This tutorial shows you how to build a foundational IDP using Backstage.io, the open-source framework for building developer portals created by Spotify and now a CNCF graduated project.

You will learn to set up a Backstage application, populate its software catalog, integrate GitHub Actions to view pipeline runs and create a software template that lets developers scaffold new services in minutes.

What is an Internal Developer Platform?

An Internal Developer Platform (IDP) is a layer built on top of your existing DevOps toolchain that exposes your infrastructure and tooling through a simplified, self-service interface. It codifies best practices and organizational standards into "golden paths", enabling developers to create and manage applications without needing deep expertise in Kubernetes, Terraform or complex CI/CD configurations.

Backstage is the leading open-source project for building IDPs. It provides a pluggable frontend and backend that act as a central hub. It's not a replacement for tools like Jenkins, Argo CD or Grafana. Instead, it integrates with them, presenting their information and actions within a unified system. This approach turns a complex, distributed toolchain into a cohesive and discoverable platform. An IDP reduces cognitive load on developers by abstracting away the underlying complexity of cloud-native infrastructure, letting them focus on writing code instead of fighting with tooling.

Prerequisites

To follow this tutorial, you need a few tools installed on your local machine.

Node.js: Backstage is a TypeScript/JavaScript application. You need Node.js v18.x or v20.x. This guide uses v20.11.1. You can use a tool like nvm to manage Node versions.
Yarn: Backstage uses Yarn v1 for package management. After installing Node.js, you can install it globally:
```
npm install -g yarn
```


shell
* **Docker:** The Backstage backend runs in a Docker container during local development to connect to a PostgreSQL database. Ensure Docker Desktop or an equivalent is installed and running.
* **`npx`:** This command-line tool is included with `npm` (which comes with Node.js) and is used to run the Backstage app creation script without a global installation.
* **A GitHub Account and Personal Access Token (PAT):** Backstage integrates with GitHub to discover components for the software catalog and display CI/CD information. You need a GitHub account and a PAT with the `repo` scope to allow Backstage to read repository information and workflow runs. You can create a token in your GitHub settings under `Developer settings > Personal access tokens > Tokens (classic)`.

## Step 1: Scaffold a New Backstage App

The fastest way to get started is with the Backstage CLI's `create-app` command. This script scaffolds a complete monorepo with a frontend, a backend and all the necessary configuration to run locally.

First, run the interactive creator using `npx`:



```bash
npx @backstage/create-app@latest

The script will prompt you for an application name. Let's call it dev-control-plane:

? Enter a name for the app [required] dev-control-plane

This process takes 5-10 minutes depending on your network speed, as it clones the template, installs all npm dependencies and sets up the basic structure.

Once it's finished, navigate into the new directory:

cd dev-control-plane

The directory structure looks like this:

.
├── app-config.yaml         # Main configuration file for your app
├── catalog-info.yaml       # Registers this app in its own catalog
├── lerna.json
├── package.json            # Root package.json for the monorepo
├── packages/
│   ├── app/                # The frontend application (React)
│   └── backend/            # The backend application (Node.js/Express)
└── yarn.lock

Now, start the application. The backend and frontend run as separate processes.

yarn dev

This command starts the backend on port 7007 and the frontend on port 3000. After a minute or two of compilation, your web browser should automatically open to http://localhost:3000.

You now have a running, albeit empty, Backstage application. The initial view shows an example catalog with a few components. The next step is to clear these examples and populate the catalog with your own services.

Step 2: Configure the Software Catalog

The Software Catalog is the heart of Backstage. It's a centralized system for tracking ownership and metadata for all your software, including microservices, libraries, websites and machine learning models. Backstage discovers these components by ingesting catalog-info.yaml files from your Git repositories.

For this example, you will need a sample GitHub repository containing a catalog-info.yaml file. You can create a new public repository named sample-service or use one of your existing projects. Throughout this guide, replace your-org with your actual GitHub username or organization name.

Create a catalog-info.yaml file in the root of that repository:

# In your-org/sample-service/catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service
  description: A sample service for the Backstage catalog.
  annotations:
    github.com/project-slug: your-org/sample-service
spec:
  type: service
  lifecycle: experimental
  owner: user:guest
  system: examples

This file contains several key fields:

apiVersion and kind: Define the entity type. Component is the most common kind, representing a piece of software.
metadata.name: A unique identifier for the component within Backstage.
metadata.annotations: Provides external identifiers. The github.com/project-slug annotation is crucial for plugins like GitHub Actions to find the correct repository.
spec.type: The type of component, for example, service, website, or library.
spec.lifecycle: The current maturity stage, such as experimental, production, or deprecated.
spec.owner: Specifies who owns this component. This is often a team or user group. For now, we'll use the default guest user.

Now, tell your Backstage application to find this file. Open app-config.yaml in the root of your dev-control-plane project and find the catalog.locations section. Replace the example rules with a single entry pointing to your repository.

# in app-config.yaml

catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  rules:
    - allow: [Component, API, Resource, Group, User, System, Domain, Template, Location]
  locations:
    # Remove the example locations and add this one:
    - type: url
      target: https://github.com/your-org/sample-service/blob/main/catalog-info.yaml

Restart your yarn dev process for the changes to take effect. When Backstage starts up, it will fetch this YAML file, process it and add the sample-service component to the catalog. You can now see it on the main page. This declarative, "as-code" approach to catalog management is powerful because the catalog stays in sync with your source code, and ownership information is version-controlled right alongside the service itself.

Step 3: Integrate a CI/CD Plugin (GitHub Actions)

Seeing a list of services is useful, but the real power of an IDP comes from integrating operational data. Let's add the GitHub Actions plugin to display CI/CD status directly on the component page in Backstage.

Add GitHub Integration Configuration

First, configure Backstage to authenticate with the GitHub API. This requires the Personal Access Token (PAT) you created earlier.

Open app-config.yaml and add the following integrations section. You may also need to add the top-level github key if it's not present.

# in app-config.yaml

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}

# This key may already exist. If so, just ensure the token is set.
github:
  token: ${GITHUB_TOKEN}

We're using an environment variable GITHUB_TOKEN to avoid committing secrets to version control. When you run yarn dev, you'll need to export this variable.

export GITHUB_TOKEN="your_classic_github_pat_here"
yarn dev

Production Gotcha: For a production deployment, you would use a secret management system like HashiCorp Vault or AWS Secrets Manager to inject this token, not an environment variable on your local machine. Proper secret handling is critical. Tools like the GitHub Actions Security scanner can help you detect accidentally committed secrets. For a deep dive, check out our guide on how to stop secret leaks in CI/CD.

Install and Configure the Plugin

Next, install the GitHub Actions plugin package in your frontend app.

cd packages/app
yarn add @backstage/plugin-github-actions

Now, you need to add the plugin's UI component to the entity page, which displays detailed information about a single component. Open the file packages/app/src/components/catalog/EntityPage.tsx.

Import the plugin components, then modify the cicdContent constant to conditionally render the GitHub Actions view.

// in packages/app/src/components/catalog/EntityPage.tsx

// ... other imports
import {
  EntityGithubActionsContent,
  isGithubActionsAvailable,
} from '@backstage/plugin-github-actions';
import { Grid, Card, CardContent } from '@material-ui/core'; // Ensure Grid is imported
import { EntitySwitch } from '@backstage/plugin-catalog'; // Ensure EntitySwitch is imported

// ...

const cicdContent = (
  <Grid container spacing={3} alignItems="stretch">
    <EntitySwitch>
      <EntitySwitch.Case if={isGithubActionsAvailable}>
        <Grid item sm={12}>
          <EntityGithubActionsContent />
        </Grid>
      </EntitySwitch.Case>

      <EntitySwitch.Default>
        <Grid item>
          <Card>
            <CardContent>
              No CI/CD provider available for this entity.
            </CardContent>
          </Card>
        </Grid>
      </EntitySwitch.Default>
    </EntitySwitch>
  </Grid>
);

This code uses EntitySwitch to conditionally render the GitHub Actions content only if the component has the necessary github.com/project-slug annotation.

After saving the file, the dev server should automatically reload. Navigate to your sample-service component in the catalog. You should now see a "CI/CD" tab, and inside it, a view of the recent GitHub Actions workflow runs for that repository. A developer can now see if their last commit passed its tests without leaving Backstage.

Step 4: Create a Software Template with the Scaffolder

One of the most powerful features of Backstage is the Software Scaffolder. It allows you to create templates for new projects, enforcing best practices and setting up everything a developer needs automatically.

Let's create a template that scaffolds a new Node.js "hello world" service, complete with a Dockerfile, a catalog-info.yaml file and registration in a new GitHub repository.

Create the Template Definition

First, create a new directory for your templates at the root of your dev-control-plane project.

mkdir -p templates/nodejs-service

Inside this directory, create a template.yaml file. This file defines the template's metadata and the input parameters it requires from the user.

# in templates/nodejs-service/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-service-template
  title: Node.js Service
  description: Creates a simple Node.js service with Docker.
spec:
  owner: user:guest
  type: service

  # These parameters are used to gather user-provided information.
  parameters:
    - title: Component Details
      required:
        - component_id
        - owner
      properties:
        component_id:
          title: Name
          type: string
          description: Unique name of the component
          ui:field: EntityNamePicker
        description:
          title: Description
          type: string
          description: A description for this component
        owner:
          title: Owner
          type: string
          description: Owner of the component
          ui:field: OwnerPicker
          ui:options:
            allowedKinds:
              - Group
    - title: Repository Location
      required:
        - repoUrl
      properties:
        repoUrl:
          title: Repository Location
          type: string
          ui:field: RepoUrlPicker
          ui:options:
            allowedHosts:
              - github.com

  # These steps are executed in order.
  steps:
    - id: fetch-base
      name: Fetch Base
      action: fetch:template
      input:
        url: ./content
        values:
          component_id: ${{ parameters.component_id }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          repoUrl: ${{ parameters.repoUrl }}

    - id: publish
      name: Publish
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: This is ${{ parameters.description }}
        repoUrl: ${{ parameters.repoUrl }}

    - id: register
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: '/catalog-info.yaml'

  # The output of a successful template run.
  output:
    links:
      - title: Repository
        url: ${{ steps.publish.output.remoteUrl }}
      - title: Open in catalog
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}

Create the Template Content

Next, create a content subdirectory within templates/nodejs-service. This will hold the skeleton files for our new service.

mkdir templates/nodejs-service/content

Inside templates/nodejs-service/content, create the following files. These are Handlebars templates, where {{ ... }} expressions will be replaced by user-provided values.

catalog-info.yaml.hbs:

# templates/nodejs-service/content/catalog-info.yaml.hbs
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.component_id | dump }}
  description: ${{ values.description | dump }}
  annotations:
    github.com/project-slug: ${{ values.repoUrl | parseRepoUrl | pick('owner') }}/${{ values.repoUrl | parseRepoUrl | pick('repo') }}
    backstage.io/techdocs-ref: dir:.
spec:
  type: service
  lifecycle: experimental
  owner: ${{ values.owner | dump }}

index.js.hbs:

// templates/nodejs-service/content/index.js.hbs
const express = require('express');
const app = express();
const port = 8080;

app.get('/', (req, res) => {
  res.send('Hello from ${{ values.component_id }}!');
});

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`);
});

package.json.hbs:

// templates/nodejs-service/content/package.json.hbs
{
  "name": "${{ values.component_id }}",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "express": "^4.18.2"
  }
}

Dockerfile.hbs:
For an in-depth guide on creating efficient Dockerfiles, see our article on Docker multi-stage builds.

# templates/nodejs-service/content/Dockerfile.hbs
FROM node:20-slim

WORKDIR /usr/src/app

COPY package*.json ./
RUN npm install --omit=dev

COPY . .

EXPOSE 8080
CMD [ "node", "index.js" ]

Register the Template

Finally, add the template to your app-config.yaml so Backstage can find it.

# in app-config.yaml
catalog:
  locations:
    # ... your other locations
    - type: file
      target: ../../templates/nodejs-service/template.yaml
      rules:
        - allow: [Template]

Restart your yarn dev server. Go to http://localhost:3000/create. You should now see your "Node.js Service" template. Clicking "Choose" will take you to a form where you can enter the component name, owner and desired GitHub repository location.

When you click "Create", the Scaffolder will:

Render the template files with your inputs.
Create a new repository in your GitHub account.
Push the rendered files to the new repository.
Register the new component in the Backstage catalog.

You've now automated the creation of new services, ensuring they all start from a standardized, compliant baseline.

Step 5: Integrate Documentation with TechDocs

The final piece of our control plane is centralized documentation. Backstage's TechDocs feature renders Markdown documentation stored alongside your code directly within the Backstage UI.

Configure TechDocs

TechDocs requires a backend plugin and a location to store the generated documentation site. For local development, it can use a local generator and storage directory. This configuration is usually present by default in new Backstage applications.

Open app-config.yaml and ensure the techdocs section is configured for local development.

# in app-config.yaml
techdocs:
  builder: 'local' # Can be 'local' or 'external'
  generator:
    runIn: 'docker' # 'docker' or 'local'
  publisher:
    type: 'local' # 'local' or 'googleGcs' or 'awsS3' or 'azureBlobStorage'

Add Documentation to a Service

Let's add documentation to the sample-service we created earlier. In that service's repository, do the following:

Add the TechDocs annotation to catalog-info.yaml. This tells Backstage where to find the documentation source. The dir:. value means "look in the current directory".

# in your-org/sample-service/catalog-info.yaml
metadata:
  # ... other metadata
  annotations:
    # ... other annotations
    backstage.io/techdocs-ref: dir:.


yaml

2. **Create an `mkdocs.yml` file** in the root of the repository. This is the configuration file for MkDocs, the static site generator TechDocs uses.



    ```yaml
    # in your-org/sample-service/mkdocs.yml
    site_name: 'Sample Service Documentation'
    nav:
      - Home: index.md

Create a /docs directory and add an index.md file inside it.

mkdir docs
echo "# Sample Service\n\nThis is the main documentation page." > docs/index.md


yaml

Commit and push these changes to your repository.

Navigate to your `sample-service` component in Backstage and click the "Docs" tab. The first time, you may need to wait a few minutes for Backstage to generate the documentation site. Once it's ready, you'll see your rendered Markdown file. You now have a single place where any developer can find up-to-date, version-controlled documentation for any service.

## Troubleshooting Common Issues

When setting up Backstage for the first time, you might run into a few common problems.

### CORS Errors

**Symptom:** The Backstage frontend fails to load data from the backend, and you see Cross-Origin Resource Sharing (CORS) errors in your browser's developer console.
**Fix:** Ensure your `app-config.yaml` has the correct `backend.cors.origin` setting for local development:



```yaml
# in app-config.yaml
backend:
  # ...
  cors:
    origin: http://localhost:3000
    methods: [GET, POST, PUT, DELETE, PATCH, OPTIONS]
    credentials: true

GitHub Auth Fails

Symptom: The GitHub Actions plugin shows an error, or the Scaffolder fails at the "publish" step with an authentication error.
Fix:

Verify the Token: Double-check that you've exported the GITHUB_TOKEN environment variable in the same terminal session where you run yarn dev.
Check Scopes: Ensure your GitHub Personal Access Token (classic) has the repo scope. For creating new repositories via the Scaffolder, it may also need the workflow scope.
Check Organization Settings: If publishing to a GitHub organization, it may have settings that restrict PAT access or require third-party application approval.

Catalog Import Fails

Symptom: A component you added to catalog.locations doesn't appear in the UI.
Fix:

Check the URL: Make sure the target URL in app-config.yaml points directly to the raw catalog-info.yaml file on your Git provider.
Validate YAML: Use a YAML linter to check for syntax errors in your catalog-info.yaml. Indentation errors are common.
Check Backend Logs: The Backstage backend logs (from the yarn dev command) will often show detailed error messages about why a location failed to be ingested. Look for lines containing Catalog-Processor or error.

You have now built the foundation of a powerful Internal Developer Platform. You've created a central place for service discovery, integrated real-time operational data, automated new service creation and centralized documentation. This is the core of a "developer control plane" that can significantly improve your team's productivity and standardize your engineering practices. From here, you can explore hundreds of other plugins for tools like Argo CD, Kubernetes and Grafana to build out a truly comprehensive platform.

Supply Chain Security Proxy: Move Beyond Vulnerability Scanning

DevOps Start — Tue, 28 Apr 2026 14:15:14 +0000

This article was originally published on devopsstart.com. Learn why relying solely on CVE scanning is a reactive strategy and how to implement a security proxy to proactively secure your software supply chain.

Vulnerability scanning is a reactive failure state, not a security strategy.

Most organizations treat Software Composition Analysis (SCA) as their primary defense against supply chain attacks. They plug in a scanner, wait for it to find a known CVE, and then assign a Jira ticket to a developer to update a library. This approach assumes that the vulnerability is already known and indexed in a database. It ignores the window of time between a malicious package upload and its discovery, and it does nothing to prevent zero-day supply chain attacks like dependency confusion or typosquatting.

If you rely solely on scanners, you are documenting how you were breached rather than preventing the attack. To secure the perimeter, you must implement a supply chain security proxy that controls the ingress of every byte of third party code before it touches your build server.

The Detection Gap

Reliance on scanning creates a dangerous detection gap. When a malicious actor uploads a package to npm or PyPI that mimics a popular library (typosquatting), there is often a several hour or even several day lag before scanners flag that specific version. In a modern CI/CD pipeline, that package is pulled, built, and deployed to production in minutes. Your secrets are exfiltrated before the scanner alerts you.

Consider dependency confusion. An attacker discovers the name of an internal corporate package, such as corp-auth-lib. They upload a malicious package with the same name but a higher version number to the public npm registry. Without a security proxy, the build agent sees the higher version on the public registry and pulls it instead of the internal one. A scanner won't stop this because the package isn't vulnerable in the CVE sense; it is performing exactly as the attacker intended.

I have seen this play out in environments with over 500 microservices where the scan and fix treadmill became a full time job for three engineers. They spent 40 hours a week chasing low severity CVEs while the actual architectural hole (direct internet access for build agents) remained open. By shifting focus from detecting a fire to controlling who enters the building, you eliminate entire classes of attacks. A security proxy acts as a mandatory checkpoint. If a package isn't on the allow list or fails a provenance check, it never enters the environment. This is the difference between a smoke detector and a locked door.

For those managing complex pipelines, this shift is similar to how you might secure Terraform PRs with an architecture firewall to prevent configuration drift. Instead of checking if the infrastructure is broken after the apply, you validate the intent before execution.

Balancing Velocity and Governance

The most common pushback from developers is that a security proxy kills velocity. The Request a Package workflow is often viewed as a bureaucratic nightmare. Developers argue that forcing every dependency through a manual approval process slows down feature delivery, especially during the inner loop of development where npm install is critical for prototyping.

This argument is partially correct. If you implement a security proxy as a manual ticket system where a security officer must click Approve on every version bump, you create a bottleneck that developers will eventually bypass. They will use personal hotspots or tunnel out of the build environment just to get work done. The friction of a poorly implemented proxy is a security risk because it encourages shadow IT.

The solution is to automate the governance. A modern security proxy should be a policy engine, not a manual gate. For example, you can set a policy that allows any package that has been public for more than 30 days, has more than 1,000 downloads, and is signed by a trusted vendor. This allows 95% of requests to pass through automatically while flagging high risk, brand new packages for a quick human review. The goal is to move from Allow All to Automated Governance.

When Scanning Still Wins

There are specific contexts where a security proxy is overkill. For very small teams (under 10 engineers) or early stage startups building a Proof of Concept (PoC), the operational overhead of maintaining a private registry like Artifactory or Nexus v3.x can outweigh the risk. At this scale, the attack surface is small and the priority is finding product market fit, not building a SLSA Level 4 compliant supply chain.

Scanning also remains superior for identifying vulnerabilities in code you have already mirrored. A proxy prevents the ingress of bad code, but it cannot predict when a previously safe library is suddenly found to have a critical flaw. When Log4Shell hit, the problem wasn't that the library was newly introduced, it was that an existing, trusted library had a critical flaw. In that case, a proxy provides no protection for existing deployments. You still need a robust SCA tool to scan your current Bill of Materials (SBOM) and identify where the vulnerable version is running.

For teams using fully managed serverless build environments where they have zero control over the network layer, a proxy is technically impossible to implement. These teams must rely on shift left scanning and strict dependency pinning in their lockfiles.

Implementing the Proxy Architecture

To move beyond scanning, you need a centralized gateway that acts as a policy enforcement point.

The Architectural Pattern

A supply chain security proxy sits between your build agents (GitHub Actions, GitLab Runners, Jenkins) and the public registries (Docker Hub, npm, PyPI). Instead of the build agent calling docker pull, it calls docker pull proxy.corp.com/image.

The proxy performs the following checks in order:

Identity: Is the request coming from an authenticated build agent?
Allow-list: Is this package/version approved for use in this project?
Integrity: Does the checksum match the known good version?
Provenance: Is there a signed attestation proving this was built in a trusted environment?

Hardening the Image Pipeline

For container images, the proxy should integrate with Sigstore/Cosign. You don't trust the tag latest or even a specific version; you verify the signature.

# Verifying an image signature using Cosign v2.2.4
cosign verify --key cosign.pub ghcr.io/my-org/my-app:v1.2.0

If verification fails, the proxy blocks the pull. To take this further, enforce SLSA framework requirements. A SLSA attestation is a signed piece of metadata that tells you how the artifact was built. If the attestation shows the image was built on a developer's laptop rather than a hardened CI runner, the proxy rejects it.

Stopping Dependency Confusion

To kill dependency confusion, configure your proxy to use Virtual Repositories with strict resolution orders. In a tool like JFrog Artifactory v7.x, you create a virtual repository that aggregates a local (private) repo and a remote (public) repo.

Configure the resolution order so that the local repository is searched first. More importantly, implement Exclusion Patterns. If a package starts with corp-, the proxy must be configured to never check the public registry for that pattern.

# Conceptual Proxy Policy for Dependency Resolution
policies:
  - pattern: "corp-*"
    action: "BLOCK_EXTERNAL"
    reason: "Internal packages must never be resolved from public registries"
  - pattern: "*"
    action: "ALLOW_EXTERNAL"
    condition: "age > 30d && downloads > 1000"

The Chain of Trust: Proxy to Admission Controller

The proxy is only the first half of the battle. The second half is ensuring that the Proxy-Approved status follows the artifact to the cluster. This is where the proxy integrates with a Kubernetes Admission Controller like Kyverno or OPA Gatekeeper.

The workflow:

Ingress: Proxy pulls node:18-alpine, verifies the signature, and caches it.
Attestation: The proxy (or a separate CI step) signs the image with a Security-Approved key.
Deployment: A developer tries to deploy the image to GKE or EKS.
Enforcement: The Admission Controller intercepts the request and checks for the Security-Approved signature.

If a developer tries to bypass the proxy by pointing their deployment to a public image on Docker Hub, the Admission Controller blocks the pod.

# Example Kyverno Policy to enforce proxy-signed images
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-proxy-signature
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-image-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "proxy.corp.com/*"
          attestors:
            - entries:
                - keys:
                    publicKeys:
                      - key: "ssh-rsa AAAAB3Nza... [your-proxy-public-key]"

This creates a complete chain of trust. The proxy ensures only vetted code enters the building, and the Admission Controller ensures only vetted code runs. If you see pods failing in your cluster, use guides on Kubernetes troubleshooting to determine if it was a signature mismatch or a network failure.

The Quarantine Zone

Moving to a proxy requires a shift in how developers interact with dependencies. The most successful implementations use a Quarantine Zone. When a developer requests a new library, the proxy pulls it into a restricted, isolated mirror. It is then automatically scanned for malware and analyzed for suspicious signals, such as a package created 2 hours ago that tries to access /etc/shadow.

If the package passes the automated gauntlet, it is promoted to the Approved repository. This allows developers to get tools quickly while keeping the production build environment sterile.

Implement Dependency Pinning as a hard requirement. Using ranges like ^1.2.0 in your package.json or requirements.txt is an invitation for disaster. The proxy should be configured to alert or block builds that do not use strict version pinning (for example, 1.2.3). This prevents stealthy updates where a vendor pushes a malicious version that fits within your range, bypassing your initial vetting.

To maintain this at scale, integrate the request process into an Internal Developer Platform (IDP) built with Backstage, allowing developers to Request a Package via a UI form that triggers the automated quarantine pipeline.

Operationalizing the Proxy

Do not flip the switch for the entire company at once. You will break every build in the organization. Instead, follow this three step rollout:

Transparent Mode: Deploy the proxy and configure build agents to use it, but set all policies to Log Only. This provides a baseline of every dependency currently used across the org. You will likely find thousands of dependencies you didn't know existed.
Caching Mode: Enable mirroring and caching. Ensure that if the public registry goes down, your builds still work. This provides immediate value to developers through faster builds and makes them allies in the security mission.
Enforcement Mode: Start blocking the most dangerous patterns first (for example, dependency confusion patterns) before moving to strict signature verification and SLSA attestations.

The operational cost of maintaining this infrastructure is non-trivial. You need high availability for your registry, as it is now a single point of failure for all deployments. Use a distributed storage backend and ensure your proxy is scaled horizontally across multiple availability zones.

Scanning is a useful tool for auditing, but it is a weak defense mechanism. By implementing a supply chain security proxy, you stop reacting to CVEs and start controlling your perimeter. You move the security boundary from the end of the pipe (the cluster) to the start of the pipe (the registry). When you combine a proxy with signature verification and a Kubernetes admission controller, you create a hardened pipeline where untrusted code simply cannot execute.

Secure Terraform PRs with an Architecture Firewall

DevOps Start — Fri, 24 Apr 2026 14:15:37 +0000

Stop the 'merge and pray' workflow! This guide was originally published on devopsstart.com and explores how to implement an automated architecture firewall for your Terraform PRs using OPA.

Introduction

An architecture firewall is a governance layer integrated into your CI/CD pipeline that automatically blocks infrastructure changes violating security or organizational standards before they reach your environment. Unlike a network firewall that filters packets, this firewall filters Pull Requests (PRs). It transforms your infrastructure requirements from passive documentation in a wiki into active, executable code that cannot be ignored.

In this article, you will learn how to move beyond the "merge and pray" workflow by implementing Policy as Code (PaC). We will explore the technical bridge between a terraform plan and automated validation using tools like Open Policy Agent (OPA) and Checkov. You'll discover how to create a pipeline that converts Terraform plans to JSON, evaluates them against strict guardrails and provides immediate feedback to developers via PR comments. By the end, you will have a strategy to enforce encryption, restrict public access and prevent accidental resource deletion without slowing down your engineering velocity. This approach aligns with modern Terraform testing best practices, ensuring that your cloud footprint remains secure by design rather than by chance.

Why Manual PR Reviews Fail the Architecture Test

Relying solely on human peer reviews to catch security holes is a recipe for a production outage. In high-velocity environments, reviewers suffer from fatigue. When a developer submits a PR with 500 lines of HCL, a reviewer might miss a single 0.0.0.0/0 in a security group or a missing encryption_enabled = true flag on an S3 bucket. Humans are great at reviewing logic and intent, but they are terrible at consistently auditing thousands of lines of configuration against a 50-page security compliance PDF.

The "Merge and Pray" workflow creates a dangerous gap where "architectural drift" occurs. This happens when the actual state of your cloud deviates from your intended security posture because a few "small" exceptions were merged over time. To solve this, you need an automated gate that operates on the terraform plan output. This plan is the only source of truth because it represents exactly what Terraform intends to do, accounting for variables, modules and the current state of the cloud.

For example, if you are using Terraform v1.9.0+, you can generate a machine-readable plan that your architecture firewall can analyze. This removes the ambiguity of reviewing the .tf files alone, which doesn't show the final resolved values.

# Generate the binary plan file
terraform plan -out=tfplan

# Convert the binary plan to JSON for policy evaluation
terraform show -json tfplan > tfplan.json

By shifting the audit from the code to the plan, you ensure that the firewall sees the final result, not just the intent. This is the foundation of a robust governance strategy.

Implementing the Policy as Code Engine

To build an architecture firewall, you must choose a Policy as Code (PaC) engine. For simple, industry-standard checks, tools like Checkov or TFLint are excellent because they come with hundreds of pre-built policies. However, for complex organizational logic (such as "Production databases must be deployed in three availability zones and have a specific naming convention"), you need a general-purpose policy engine like Open Policy Agent (OPA). OPA uses a language called Rego to query JSON data.

The technical flow is straightforward: your CI pipeline runs the plan, converts it to JSON and pipes that JSON into OPA. If the Rego policy returns a "deny" result, the CI pipeline fails and the PR is blocked from merging. This turns your security requirements into a unit test for your infrastructure.

Below is a practical example of a Rego policy that prevents any AWS S3 bucket from being created without server-side encryption.

package terraform.analysis

import future.keywords.if

# Default allow unless a violation is found
default allow = true

# Violation: S3 bucket without encryption
deny[msg] if {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket"

    # Check if server_side_encryption_configuration is missing or empty
    # In Terraform JSON, 'after' contains the planned state
    not resource.change.after.server_side_encryption_configuration
    msg := sprintf("Security Violation: S3 bucket %s must have encryption enabled", [resource.address])
}

To run this against your plan in a GitHub Action or GitLab CI runner using OPA v0.60.0, you would execute:

# Run OPA evaluation and capture the deny rules
opa eval -i tfplan.json -d policy.rego "data.terraform.analysis.deny"

If the output contains any messages, the firewall has triggered and the build should fail. This process provides a mathematical guarantee that no unencrypted bucket ever reaches production.

Real-World Application: Preventing Production Catastrophes

A common production nightmare is the accidental deletion of a critical resource, such as a primary database or a core VPC, due to a renaming error or a module refactor. A manual reviewer might not realize that changing a resource name in Terraform results in a "destroy and recreate" action. An architecture firewall can catch this by analyzing the actions array in the terraform plan JSON.

By writing a policy that flags any delete action on resources tagged as critical, you create a safety net. This doesn't mean you can never delete things; it means you must explicitly acknowledge the risk, perhaps through a "break-glass" label on the PR or a manual override from a Lead Architect.

Consider this scenario: a developer changes the name of an RDS instance to match a new naming convention. Terraform sees this as deleting the old DB and creating a new one. Without a firewall, the PR looks like a simple string change. With a firewall, the system sees a delete action on an aws_db_instance and blocks it.

Here is how you would implement a "Protection" rule in Rego to block deletions of production databases:

package terraform.analysis

deny[msg] if {
    resource := input.resource_changes[_]
    resource.type == "aws_db_instance"

    # Check if the action includes 'delete'
    resource.change.actions[_] == "delete"

    # Only apply this to production environments by checking input variables
    input.variables.environment == "prod"

    msg := sprintf("CRITICAL FAILURE: Attempting to delete production database %s. This action is blocked by the Architecture Firewall.", [resource.address])
}

Integrating this into your workflow requires a tight loop. You can use automation for Terraform reviews to post these specific error messages directly as comments on the offending line of the PR. This transforms the "No" from the security team into a helpful, automated suggestion from the platform.

Best Practices for Architecture Guardrails

Implementing a firewall can create friction if handled poorly. If every PR is blocked by 50 different warnings, developers will find ways to bypass the system. Use these strategies to balance security with velocity.

Distinguish Between Warnings and Failures. Not every policy should block a merge. Use "Advisory" levels for things like "missing cost-center tag" (warning) and "Critical" levels for "open SSH port" (hard fail). This prevents the firewall from becoming a nuisance.
Version Your Policies. Treat your Rego or Checkov policies like application code. Store them in a separate Git repository, version them and test them against a suite of "known-bad" Terraform plans to ensure no regressions in your security posture.
Provide Remediation Guidance. A failure message like Policy violation: SEC-01 is useless. Your firewall should return Security Violation: Port 22 is open to 0.0.0.0/0. Please restrict this to the corporate VPN range (10.x.x.x).
Implement an Exception Process. There will always be a legitimate reason to break a rule. Create a standardized way to grant exceptions, such as requiring a specific metadata tag (exception_id = "SEC-123") that the policy engine is programmed to ignore.
Shift Left with Local Pre-commit Hooks. Don't make the CI pipeline the first time a developer sees a failure. Provide a pre-commit configuration using tools like terraform-docs and checkov so they can catch errors on their local machine.

FAQ

Does the architecture firewall replace the need for manual peer reviews?

No, it augments them. The firewall handles the "objective" checks (security, compliance, syntax) so that human reviewers can focus on the "subjective" checks (architecture design, business logic and efficiency). It removes the tedious parts of the review process, allowing engineers to have higher-level discussions about the implementation.

Which tool should I choose: OPA, Checkov, or Sentinel?

If you are using Terraform Cloud/Enterprise, Sentinel is the native choice and offers the deepest integration. If you need a free, industry-standard scanner that works out-of-the-box with minimal configuration, go with Checkov. If you have complex, custom business logic that spans multiple cloud providers and requires a powerful query language, Open Policy Agent (OPA) is the gold standard. I have seen mature platform teams use Checkov for general security and OPA for custom organizational guardrails.

How do I prevent the firewall from slowing down my deployment pipeline?

Running terraform plan and opa eval typically adds less than 60 seconds to a pipeline. To further optimize, you can run these checks in parallel with other tests. Additionally, by implementing local pre-commit hooks, you reduce the number of failed CI runs, meaning the pipeline only handles "clean" code.

Can I use this to manage costs?

Yes, this is a powerful use case. You can write policies that analyze the resource_changes for expensive instance types. For example, you can block any PR that attempts to spin up an aws_instance of type p4d.24xlarge unless the project has a specific "high-compute" approval tag. This prevents "bill shock" by catching expensive mistakes before the resources are actually provisioned.

Conclusion

Building an architecture firewall is about shifting your mindset from "trusting the reviewer" to "trusting the system." By implementing Policy as Code, you ensure that your security standards are consistently applied across every single PR, regardless of who is reviewing it. This creates a scalable governance model that allows your platform team to support hundreds of developers without becoming a bottleneck.

To get started, don't try to automate your entire security handbook at once. Start with the "low hanging fruit": block public S3 buckets and open SSH ports. Once your team is comfortable with the automated feedback loop, gradually introduce more complex architectural rules.

Your next steps are to install OPA or Checkov, integrate a terraform show -json step into your GitHub Actions or GitLab CI and write your first "deny" rule. This transition from manual oversight to automated guardrails is the defining characteristic of a mature Platform Engineering organization.

Local LLM for Log Analysis: Privacy-First Debugging with Ollama

DevOps Start — Fri, 24 Apr 2026 14:05:27 +0000

Stop sending sensitive production logs to the cloud. This guide, originally published on devopsstart.com, shows you how to build a privacy-first debugging stack using Ollama and Llama 3.

Introduction

Sending production logs to cloud AI APIs is a non-starter for any serious SRE in a regulated industry. The answer to maintaining security while gaining AI capabilities is to shift the inference engine to your own hardware. By deploying a local LLM stack using Ollama and Llama 3, you can perform semantic log analysis and root cause diagnosis without a single byte of data leaving your secure perimeter.

Whether you are in fintech, healthcare, or govtech, the "Compliance Wall" is real. You cannot risk leaking PII, session tokens, or internal IP addresses to a third party, even with "Enterprise" privacy agreements. You can find the fundamental concepts of managing these workloads in the official Ollama documentation, which provides the framework for running open-source models locally.

Why this take

Most organizations try to solve the privacy problem with PII Redaction scripts before sending logs to a cloud provider. This is a flawed strategy. Regular expressions and basic NER (Named Entity Recognition) models always miss something. A leaked credit card number or a proprietary internal URL in a stack trace can trigger a compliance audit that costs your company millions. The only way to guarantee zero leakage is to ensure the data never leaves the air-gapped environment or the VPC.

In a production environment with over 500 microservices, the sheer volume of logs makes manual grep-ing impossible. I have seen teams spend six hours correlating logs across three different namespaces just to find a single timeout. A local LLM, when fed a curated slice of logs, can identify the behavioral pattern of a failure in seconds. For example, a sequence of 200 OK responses that occur in an impossible order often indicates a logic bug that regex-based monitors will never catch.

Consider the operational reality of a CrashLoopBackOff. Instead of manually running kubectl logs and kubectl describe and trying to map them in your head, you can pipe the output directly into a local model. When you are Fixing Kubernetes CrashLoopBackOff in Production, the bottleneck is usually the cognitive load of parsing verbose Java or Go stack traces. A local LLM reduces this load by summarizing the failure point immediately.

The cost of cloud tokens for log analysis is astronomical. Logs are verbose. If you send 10MB of logs to a cloud LLM for every incident, your monthly bill will skyrocket. Running a 7B or 8B parameter model on a dedicated GPU node costs nothing but the electricity and the initial hardware investment.

The strongest counter-argument

The most common pushback against local LLMs is the "Hardware Tax." Critics argue that the VRAM requirements for acceptable performance are too high for a standard developer laptop or a typical DevOps jump box. It is true that running a 70B parameter model requires multiple A100s or H100s to be performant, which is an unreasonable ask for a local debugging setup. If you try to run a large model on a CPU with 16GB of RAM, the tokens per second will be so slow that you might as well go back to using grep.

There is also the issue of Context Window limitations. A production log file can be several gigabytes, while most local models have a context window ranging from 8k to 128k tokens. You cannot simply upload a log file to Ollama and ask what happened. You have to implement a pre-processing pipeline to slice the logs, filter out the noise, and feed the model only the relevant window surrounding the timestamp of the error. This adds architectural complexity that a simple API call to OpenAI does not have.

However, these arguments ignore the reality of model quantization. Using 4-bit quantization (GGUF format), you can run a Llama 3 8B model on a machine with as little as 8GB of VRAM with negligible loss in reasoning capability for log analysis. For DevOps tasks, you do not need the creative writing abilities of a 175B parameter model; you need a model that understands stack traces and Kubernetes events.

Exceptions where cloud LLMs still win

There are specific scenarios where a local LLM is the wrong tool. If you are a tiny startup with zero regulatory constraints and no dedicated hardware, the overhead of managing an Ollama instance is a distraction. In those cases, the speed of onboarding a cloud API outweighs the privacy risks.

Cloud LLMs also win when you need cross-domain knowledge at an extreme scale. If your log error is caused by a very obscure bug in a niche third party library that was updated two weeks ago, a cloud model trained on the most recent web crawl might have the answer. A local model's knowledge is frozen at the time of its training.

Additionally, if your team requires a collaborative, multi-user interface with complex permissioning and auditing for every single prompt, building that on top of Open WebUI requires more effort than using a managed SaaS platform. For the senior SRE who needs to diagnose a production outage in a secure environment, these advantages are irrelevant.

Implementing the Privacy-First Stack

To move from theory to production, use Ollama for the backend, Llama 3 (8B) for the reasoning, and Open WebUI for the interface.

Installing the Engine

On a Linux workstation with an NVIDIA GPU, install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, pull the Llama 3 model. I recommend the 8B version for most log tasks as it balances speed and accuracy.

ollama run llama3:8b

The Log Pipeline Architecture

You cannot dump a 1GB log file into the model. You must use a pipeline. The most effective flow is: Log Source → Grep/Awk Filter → Local LLM.

For example, if you are debugging an OOMKilled pod, first extract the relevant events. If you have already followed the steps to Debug OOMKilled Pods in Kubernetes, you know that the describe output is more valuable than the application logs.

Use this bash script to automate the extraction and analysis:

# Extract the last 100 lines of logs and the pod description
kubectl describe pod my-app-6f7d8-abc > pod_desc.txt
kubectl logs my-app-6f7d8-abc --tail=100 > pod_logs.txt

# Combine them into a prompt file
echo "Act as a Senior SRE. Analyze the following Kubernetes pod description and logs to find the root cause of the failure. Focus on memory limits and exit codes." > prompt.txt
cat pod_desc.txt >> prompt.txt
echo "--- LOGS ---" >> prompt.txt
cat pod_logs.txt >> prompt.txt

# Pipe the prompt to Ollama
cat prompt.txt | ollama run llama3:8b

Prompt Engineering for DevOps

Generic prompts yield generic answers. To get production-ready insights, give the LLM a persona and a specific constraint.

Bad Prompt: "What is wrong with these logs?"

Good Prompt:
"Act as a Site Reliability Engineer specializing in Java Spring Boot applications. I am providing a heap dump summary and the last 50 lines of the application log. Identify if this is a Memory Leak or a sudden spike in traffic. Provide the answer in a bulleted list: 1. Root Cause, 2. Evidence from logs, 3. Recommended fix."

When dealing with complex orchestration issues, such as those found when you Fix CrashLoopBackOff in Kubernetes Pods, use this template:

Persona: Kubernetes Expert
Context: Pod is in CrashLoopBackOff.
Task: Analyze the 'Last State' termination message and the current logs.
Constraint: Ignore health check failures; focus on application-level exceptions.
Logs: [Insert Logs Here]

Hardware Requirements and Performance

The sweet spot for local log analysis is a machine with 24GB of VRAM (like an RTX 3090 or 4090). This allows you to run the 8B model with a massive context window or even experiment with the 70B model using heavy quantization.

Component	Minimum (Fast)	Recommended (Pro)	Note
GPU	NVIDIA RTX 3060 (12GB)	NVIDIA RTX 4090 (24GB)	VRAM is the primary metric
RAM	16GB	64GB	Used for offloading if VRAM fills
Storage	50GB SSD	200GB NVMe	Models are large (4GB to 40GB each)
OS	Ubuntu 22.04	Ubuntu 24.04	Best driver support for CUDA

If you are forced to run on CPU (Apple Silicon M2/M3 is an exception and works great), expect a drop from 50 tokens per second to about 3 to 5 tokens per second. This is acceptable for asynchronous log analysis but frustrating for interactive chatting.

Semantic Anomaly Detection vs. Regex

Standard observability tools like Splunk or ELK rely on indices and keyword searches. If you search for "Error", you find errors. But what if the system is failing silently?

Example: A payment gateway returns 200 OK for every request, but the response body says {"status": "pending", "reason": "timeout"}. A regex monitor sees the 200 and stays green. A local LLM can be prompted to look for logical contradictions:

"Analyze these logs for 'silent failures'. Look for cases where the HTTP status is 200 but the response body indicates a failure or a timeout."

This move from syntactic analysis (looking for patterns) to semantic analysis (understanding meaning) is the real power of the local LLM. It allows you to find the unknown unknowns that you didn't know to write a regex for.

Log Streamlining and Noise Reduction

One of the biggest costs in DevOps is Log Bloat. We store terabytes of INFO logs that we never read. You can use a local LLM as a pre-processor to summarize logs before they are even archived.

By running a small, fast model like Mistral v0.3, you can create a Log Summarizer that takes 1,000 lines of verbose debug logs and converts them into three sentences:

The application started successfully.
It attempted to connect to the database three times and failed.
It entered a sleep state for 30 seconds.

This reduces the cognitive load on the human engineer and can potentially reduce storage costs if you only archive the summaries and a sampled percentage of the raw logs.

Local LLMs are the only viable path for secure, privacy-first debugging in highly regulated environments. While the hardware requirements are higher than using a cloud API, the trade-off is a total elimination of PII leakage risk and the removal of per-token costs. Start by installing Ollama on a GPU-enabled jump box, select a 4-bit quantized Llama 3 model, and begin piping your kubectl outputs into it to reduce your mean time to resolution (MTTR).

How to Build AI Agents for Kubernetes Deployments

DevOps Start — Thu, 23 Apr 2026 14:15:34 +0000

Ever wanted an AI that doesn't just explain Kubernetes errors but actually helps you fix them? This guide, originally published on devopsstart.com, walks through building autonomous K8s agents using MCP, Kagent, and K8sGPT.

Introduction

AI agents for Kubernetes deployments are autonomous systems that follow an "Observe → Reason → Act" loop to resolve cluster issues without manual intervention. While a standard LLM can explain what a CrashLoopBackOff is, a true agent can detect the error, pull the logs, analyze the stack trace, cross-reference it with recent Git commits, and propose a specific PR to fix the environment variable causing the crash.

Building these agents requires moving beyond simple prompting and into "tool use" or "function calling." You are essentially giving an LLM a set of specialized skills (API wrappers) that allow it to interact with your cluster, your GitOps pipeline, and your observability stack. In this guide, you will learn how to architect these skills using the Model Context Protocol (MCP) and frameworks like Kagent and K8sGPT to automate the most tedious parts of Kubernetes operations.

For a deep dive into the foundational concepts of managing the pods these agents will be monitoring, see the guide on Kubernetes for Beginners: Deploy Your First Application.

Prerequisites

Before starting this tutorial, you need a functioning Kubernetes environment and the necessary API access for the LLM. I recommend a development cluster (Kind or Minikube) or a staging namespace in a cloud provider like GKE or EKS to avoid accidental production outages.

You will need the following tools installed on your local machine:

kubectl v1.30+: The standard Kubernetes CLI.
Helm v3.14+: For managing the agent's dependencies.
Python 3.11+: Most agent frameworks, including Kagent and LangChain, require modern Python.
An OpenAI API Key (GPT-4o) or Anthropic API Key (Claude 3.5 Sonnet): Agents require high-reasoning models to avoid hallucinations during tool selection.
K8sGPT v0.12+: For the diagnostic skill set implementation.

You should also have a basic understanding of Kubernetes RBAC. Agents operate as identities within the cluster, and giving them cluster-admin privileges is a security risk. You will need to be comfortable creating ServiceAccounts and RoleBindings to enforce the principle of least privilege.

Overview

In this tutorial, we are building a "Deployment Guardian" agent. This isn't a monolithic script, but a modular system capable of three specific skills:

Automated Diagnostics: Using K8sGPT to scan for misconfigurations and interpreting those errors using an LLM.
Resource Right-Sizing: Analyzing pod resource usage and suggesting updates to the Horizontal Pod Autoscaler (HPA).
GitOps Sync Validation: Monitoring ArgoCD application health and triggering syncs when drifts are detected.

The core of this architecture relies on the Model Context Protocol (MCP). MCP is an open standard that decouples the LLM from the specific implementation of the tool. Instead of writing a custom wrapper for every single kubectl command, MCP allows you to expose a standardized "server" that tells the LLM exactly what tools are available, what arguments they take, and what the expected output format is.

By the end of this guide, you will have an agent that provides the root cause and the exact YAML change needed to fix a deployment, integrated directly into your operational workflow. For those managing the underlying infrastructure of these clusters, understanding how to Deploy an EKS Cluster with Terraform provides the necessary context for where these agents actually reside.

Step 1: Architecting the Agent Loop

Before writing code, you must understand how the agent thinks. A standard LLM request is a linear path: Prompt → Response. An agent loop is circular.

When you ask an agent to "Fix the failing deployment in the staging namespace," it performs the following sequence:

Observation: The agent calls a tool (for example, get_pod_status) to see which pods are failing.
Reasoning: It observes three pods in CrashLoopBackOff and reasons that it needs logs to understand the root cause.
Action: It calls get_pod_logs for one of the failing pods.
Observation: The logs show a java.lang.NullPointerException related to a missing database URL.
Reasoning: It checks the ConfigMap to see if the environment variable is defined.
Action: It calls get_configmap.
Final Response: It concludes the environment variable is missing and suggests the specific kubectl patch command or Git PR.

To implement this, you can use a framework like Kagent, which is built on AutoGen. It treats the "DevOps Engineer" as one agent and the "Kubernetes Cluster" as a tool-providing environment.

Step 2: Implementing the Tooling Layer with MCP

The Model Context Protocol (MCP) is the primary mechanism for production-grade agents. Instead of hardcoding functions into your Python script, you run an MCP server that exposes your Kubernetes API.

First, install the MCP SDK for Python:

pip install mcp

Now, create a simple MCP server that provides a "skill" to get pod events. This is more efficient than giving the LLM raw kubectl access because you can filter the output to only include errors, which reduces token usage and hallucination risk.

# k8s_mcp_server.py
from mcp.server.fastmcp import FastMCP
import subprocess

mcp = FastMCP("K8s-Guardian")

@mcp.tool()
def get_pod_errors(namespace: str) -> str:
    """Fetches only Warning events for pods in a specific namespace."""
    cmd = ["kubectl", "get", "events", "-n", namespace, "--field-selector", "type=Warning"]
    result = subprocess.run(cmd, capture_output=True, text=True)

    if result.returncode != 0:
        return f"Error fetching events: {result.stderr}"

    return result.stdout if result.stdout else "No warning events found."

if __name__ == "__main__":
    mcp.run()

To run this server:

python k8s_mcp_server.py

The LLM now sees get_pod_errors as a capability. When it encounters a deployment failure, it will autonomously decide to call this function rather than guessing. This architectural separation allows you to update the Python "skill" without changing the prompt of the LLM.

Step 3: Configuring Least-Privilege RBAC

Giving an AI agent a kubeconfig with cluster-admin is an unacceptable security risk. If the LLM hallucinates a command like kubectl delete ns --all, the agent will execute it.

You must create a dedicated ServiceAccount with a restricted Role. For our Deployment Guardian, the agent needs to read pods, events, and logs, but it should only be able to "patch" specific resources.

Create a file named agent-rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-ai-agent
  namespace: ai-ops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-read-write-role
  namespace: staging
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "events", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-read-write-binding
  namespace: staging
subjects:
- kind: ServiceAccount
  name: k8s-ai-agent
  namespace: ai-ops
roleRef:
  kind: Role
  name: agent-read-write-role
  apiGroup: rbac.authorization.k8s.io

Apply the configuration:

kubectl create namespace ai-ops
kubectl apply -f agent-rbac.yaml

To connect your agent to this identity, use a token-based approach or a projected volume if the agent runs inside the cluster. For local development, you can impersonate the ServiceAccount to verify permissions:

kubectl get pods -n staging --as=system:serviceaccount:ai-ops:k8s-ai-agent

Step 4: Integrating K8sGPT for Diagnostic Skills

While custom MCP tools are great for specific tasks, K8sGPT provides a powerful set of pre-built diagnostic skills. It scans your cluster for common issues and uses an LLM to explain them.

First, install the K8sGPT CLI:

brew install k8sgpt

Now, authenticate it with your LLM provider:

k8sgpt auth add --backend openai --model gpt-4o

To integrate K8sGPT into your agent's skill set, wrap the k8sgpt analyze command into a tool. This allows the agent to trigger a full cluster scan and reason over the results.

# Adding K8sGPT as a tool in our MCP server
@mcp.tool()
def analyze_cluster_health(namespace: str) -> str:
    """Runs a K8sGPT analysis on the namespace to find errors."""
    cmd = ["k8sgpt", "analyze", "--namespace", namespace]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

When you run this, the output provides a detailed analysis:

$ k8sgpt analyze --namespace staging
[!] Pod 'auth-service-6f7d' is in CrashLoopBackOff
Analysis: The pod is failing because the 'DB_PASSWORD' environment variable is missing.
The application expects this variable to be provided via a Secret.

The agent can now combine this high-level analysis with its own get_configmap tool to find where the secret is missing. This creates a tiered diagnostic approach: K8sGPT finds the "what," and the custom MCP tools find the "how" and "where." If you see these errors frequently, check the Fix Kubernetes CrashLoopBackOff in Production guide for manual remediation steps.

Step 5: Building the Resource Optimization Skill

Resource optimization requires the agent to observe metrics (via Prometheus or Metrics Server) and act on the Horizontal Pod Autoscaler (HPA).

To implement this, your agent needs a tool that can query the Metrics Server:

@mcp.tool()
def get_pod_resource_usage(pod_name: str, namespace: str) -> str:
    """Retrieves CPU and Memory usage for a specific pod."""
    cmd = ["kubectl", "top", "pod", pod_name, "-n", namespace]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

The agent's reasoning logic for optimization follows this pattern:

Trigger: The agent is asked to "Optimize the checkout-service."
Observation: It calls get_pod_resource_usage and sees the pod is consistently using 95% of its memory limit.
Observation: It calls kubectl get hpa and sees the HPA is targeting 50% CPU, but the bottleneck is actually memory.
Reasoning: The agent realizes the HPA should be updated to include memory metrics or the memory limit should be increased.
Action: It proposes a YAML change to the HPA definition.

For a detailed explanation of how HPA works to better tune your agent's prompts, read the Kubernetes HPA Deep Dive.

Step 6: Automating GitOps with ArgoCD Integration

An agent that runs kubectl patch directly creates "configuration drift." The source of truth must always be Git. Therefore, your agent's "Act" phase should target your GitOps tool.

If you are using ArgoCD, give your agent tools to interact with the ArgoCD API or the Git repository. First, ensure you have ArgoCD installed; if not, follow the How to Install Argo CD guide.

Now, create a tool that allows the agent to check the sync status of an application:

@mcp.tool()
def get_argocd_app_status(app_name: str) -> str:
    """Checks if an ArgoCD application is Synced and Healthy."""
    cmd = ["argocd", "app", "get", app_name]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return result.stdout

The "GitOps Loop" for the agent is:

Detect: The agent sees a pod failing in the cluster.
Diagnose: It finds that the image tag v1.2.0 has a bug.
Resolve: It searches for the latest stable image tag in the registry.
Act: Instead of running kubectl set image, it uses a GitHub API tool to create a Pull Request updating the image tag in the Git repository.
Verify: It monitors ArgoCD until the app shows as Synced and Healthy.

This workflow ensures the AI agent remains a part of the governed pipeline. You can learn more about managing these sync policies in the Advanced Argo CD Sync Policies tutorial.

Step 7: Implementing Safety Rails and Human-in-the-Loop (HITL)

To prevent "hallucination-driven outages," you must implement a safety layer between the agent's reasoning and the action.

1. The Dry-Run Constraint

Every tool that modifies the cluster must implement a --dry-run=server flag by default. The agent should first call the tool in dry-run mode and present the proposed change.

@mcp.tool()
def propose_deployment_patch(deployment_name: str, namespace: str, patch_yaml: str) -> str:
    """Proposes a change to a deployment using dry-run."""
    with open("patch.yaml", "w") as f:
        f.write(patch_yaml)

    cmd = ["kubectl", "patch", "deployment", deployment_name, "-n", namespace, "--patch-file", "patch.yaml", "--dry-run=server"]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return f"Proposed Change: {result.stdout}"

2. The Approval Gate (HITL)

The agent must not execute a patch or delete command without manual approval from a human operator, typically via a Slack bot or CLI prompt.

Agent: "I've found that the auth-service is OOMKilled. I propose increasing the memory limit from 256Mi to 512Mi. Should I apply this change? [Yes/No]"
Human: "Yes"
Agent: (Executes the actual patch command)

3. Policy-as-Code (Kyverno/OPA)

A cluster-level policy engine like Kyverno or OPA Gatekeeper should be the final line of defense. For example, a policy that prevents any resource from being deleted in the production namespace, regardless of the requester's identity.

Step 8: Testing and Validating Agent Performance

Treat your agent's skills like production code.

Unit Testing Tools

Test each MCP tool independently. If your get_pod_errors tool fails to parse kubectl output, the LLM will receive garbage and hallucinate a solution.

# Example test for the tool
python -c "from k8s_mcp_server import get_pod_errors; print(get_pod_errors('staging'))"

Scenario-Based Validation (Chaos Engineering)

Test your agent by intentionally breaking things in a sandbox:

Inject a Failure: Delete a Secret that a deployment needs.
Trigger Agent: Ask, "Why is the deployment failing?"
Evaluate:

Did it find the missing secret? (Correctness)
Did it suggest the right fix? (Accuracy)
Did it try to delete the namespace? (Safety)
How many tool calls did it take? (Efficiency)

Token Cost and Latency Tracking

Agents can be expensive. A complex diagnostic loop might call 10 different tools, sending significant context back to the LLM. Use tools like LangSmith or Arize Phoenix to trace the agent's thoughts. If the agent loops infinitely (calling the same tool repeatedly), refine the system prompt to include a "maximum tool call" limit.

Troubleshooting

The Agent "Loops" Infinitely

Symptom: The agent calls get_pod_status repeatedly for 20 turns.
Fix: Update the system prompt: "If a tool returns the same result twice, do not call it again. Instead, try a different diagnostic tool or ask the user for more information."

RBAC "Forbidden" Errors

Symptom: Error from server (Forbidden): pods "my-pod" is forbidden: User "system:serviceaccount:ai-ops:k8s-ai-agent" cannot get resource "pods/log".
Fix: Check your Role definition. pods and pods/log are different resources in Kubernetes. You must explicitly list pods/log in the resources section of your RBAC YAML.

Hallucinated CLI Flags

Symptom: The agent tries to run kubectl get pods --show-all-errors, which is not a real flag.
Fix: Be explicit in your MCP tool description. Instead of "Fetch pods," say "Fetch pods using the exact command kubectl get pods -n {namespace}."

Context Window Overflow

Symptom: The agent "forgets" the initial error after calling several tools.
Fix: Implement "summarization" in your tools. Instead of returning raw kubectl output, filter for the top 5 most relevant errors before sending the text to the LLM.

Conclusion

Building AI agents for Kubernetes is a shift from "writing scripts" to "designing capabilities." By utilizing the Model Context Protocol (MCP), you decouple your agent's reasoning from the underlying API calls, allowing you to iterate on "skills" without breaking the agent's logic.

We have moved from the basic "Observe → Reason → Act" loop to a production-ready architecture featuring least-privilege RBAC, GitOps integration via ArgoCD, and strict human-in-the-loop safety rails.

Actionable Next Steps:

Start Small: Implement one "read-only" skill (like the get_pod_errors tool) and run it in a local Kind cluster.
Secure the Perimeter: Apply the RBAC constraints before moving the agent to a shared development environment.
Implement the Gate: Add a manual approval step for any tool that uses kubectl patch or kubectl delete.
Monitor and Refine: Use a tracing tool to see where your agent is hallucinating and refine your tool descriptions accordingly.

How to Manage Multiple Azure Subscriptions in Terraform

DevOps Start — Mon, 20 Apr 2026 22:01:41 +0000

Managing Hub-and-Spoke architectures in Azure can be a challenge when dealing with multiple subscriptions. This guide, originally published on devopsstart.com, explains how to use Terraform provider aliases to streamline your deployments.

How to Manage Multiple Azure Subscriptions in Terraform

To deploy resources across multiple Azure subscriptions in a single Terraform configuration, you must use provider aliases. By default, the azurerm provider targets only one subscription based on your authentication context. To override this, you define multiple provider blocks, assigning an alias to each and specifying a unique subscription_id.

This pattern is essential for Hub-and-Spoke network architectures. In these environments, central shared services (like Azure Firewall or ExpressRoute) live in a Hub subscription, while application workloads reside in separate Spoke subscriptions. Without aliases, you would be forced to run separate Terraform states and pipelines for every single subscription, which makes cross-subscription networking a manual nightmare.

You can find the complete provider specification in the official Terraform Azure Provider documentation.

Implementing Provider Aliases

To start, you need to configure your providers.tf file. The provider without an alias becomes the default. Any provider with an alias must be explicitly called when defining a resource using the provider meta-argument.

# providers.tf

# Default provider (Spoke Subscription)
provider "azurerm" {
  features {}
  subscription_id = "00000000-0000-0000-0000-000000000000"
}

# Aliased provider (Hub Subscription)
provider "azurerm" {
  alias           = "hub"
  features {}
  subscription_id = "11111111-1111-1111-1111-111111111111"
}

When you create a resource, use the provider argument to tell Terraform which subscription to use. If you omit this, Terraform defaults to the primary provider.

# Deploy a VNet in the Hub subscription
resource "azurerm_virtual_network" "hub_vnet" {
  provider            = azurerm.hub
  name                = "hub-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = "eastus"
  resource_group_name = "hub-rg"
}

# Deploy a VNet in the Spoke subscription (default provider)
resource "azurerm_virtual_network" "spoke_vnet" {
  name                = "spoke-vnet"
  address_space       = ["10.1.0.0/16"]
  location            = "eastus"
  resource_group_name = "spoke-rg"
}

Cross-Subscription Data Referencing

A common production scenario involves fetching an existing resource ID from a Hub subscription to use as a property in a Spoke resource, such as creating a VNet peering. In my experience, this is where most "Resource Not Found" errors occur because the data block defaults to the wrong subscription.

# Fetch Hub VNet ID from the Hub subscription
data "azurerm_virtual_network" "hub_vnet_data" {
  provider            = azurerm.hub
  name                = "hub-vnet"
  resource_group_name = "hub-rg"
}

# Create peering in the Spoke subscription pointing to the Hub
resource "azurerm_virtual_network_peering" "spoke_to_hub" {
  name                      = "spoke-to-hub"
  resource_group_name       = "spoke-rg"
  virtual_network_name      = azurerm_virtual_network.spoke_vnet.name
  remote_virtual_network_id = data.azurerm_virtual_network.hub_vnet_data.id
}

By explicitly assigning provider = azurerm.hub to the data block, Terraform authenticates against the Hub subscription to retrieve the ID before attempting to create the peering in the Spoke subscription.

The Module Provider Gotcha

The biggest mistake engineers make with multi-subscription setups is assuming modules inherit aliases automatically. They do not. If you call a module and it contains azurerm resources, those resources will use the default provider regardless of where the module is called from.

To fix this, you must explicitly pass the aliased provider into the module using the providers map.

module "spoke_workload" {
  source = "./modules/workload"

  # Map the module's internal 'azurerm' provider to the 'hub' alias
  providers = {
    azurerm = azurerm.hub
  }

  vnet_id = data.azurerm_virtual_network.hub_vnet_data.id
}

Inside the module code, do not define a provider block. Just use the standard azurerm resource blocks; the mapping happens at the root level. This ensures your modules remain reusable across different environments. I have seen this fail in clusters with >50 nodes where a missed provider mapping caused a production workload to be deployed into a development subscription, leading to significant security audit failures. To maintain high reliability, consider testing your infrastructure as code.

Best Practices for Naming and Scale

Avoid generic names like azurerm.sub1 or azurerm.secondary. In a production environment with dozens of subscriptions, these names provide zero context and lead to configuration errors. Use functional names that describe the role of the subscription:

azurerm.hub
azurerm.shared_services
azurerm.prod_workload
azurerm.identity_mgmt

In environments with more than 50 subscriptions, managing these aliases in a single providers.tf file becomes brittle. At that scale, I recommend splitting your state files by subscription or using a wrapper tool. This reduces the blast radius of a single terraform apply and decreases the time spent in the "refreshing state" phase, which can otherwise take several minutes.

FAQ

Can I use the same Service Principal for multiple subscriptions?
Yes, as long as that Service Principal has the required RBAC roles (for example, Contributor) across all targeted subscriptions. Terraform handles the switching via the subscription_id field in the provider block.

Do I need to run az account set before running Terraform?
No. When you explicitly define subscription_id in the provider block, Terraform ignores the current active subscription in your Azure CLI session and targets the ID specified in the code.

Does using aliases increase the plan time?
Slightly. Terraform must establish separate API sessions for each provider instance. In very large environments, this can add 10 to 30 seconds to the refresh phase.

Conclusion

Using provider aliases is the only professional way to handle multi-subscription Azure deployments. By separating your Hub and Spoke configurations and explicitly passing providers to your modules, you eliminate the risk of deploying resources to the wrong environment.

Your next steps should be to:

Audit your current providers.tf and rename any generic aliases to functional names.
Check your module calls to ensure providers = { ... } is being used for all non-default subscriptions.
Implement data blocks to automate the linkage between Hub and Spoke resources.

GitHub Actions Security: How to Stop Secret Leaks in CI/CD

DevOps Start — Mon, 20 Apr 2026 21:46:31 +0000

Originally published on devopsstart.com, this guide explores how to eliminate static secrets and harden your GitHub Actions pipelines against credential theft.

Introduction

The fastest way to compromise a production environment isn't by hacking a firewall; it's by stealing a long-lived AWS Access Key leaked in a GitHub Actions log. Secret leakage in CI/CD pipelines is a systemic risk because these pipelines possess the "keys to the kingdom", allowing them to provision infrastructure, modify databases and push code to production.

When secrets leak, they typically happen through three vectors: accidental logging, compromised third-party actions or malicious pull requests from external contributors. To stop this, you must move from static secrets to identity-based authentication using OpenID Connect (OIDC) and implement a strict least-privilege model for your workflow permissions.

In this guide, you will learn how to implement OIDC, the danger of mutable version tags, and how to defend against "pwn-request" attacks. For those managing complex infrastructure, combining these security practices with how to automate terraform reviews with github actions ensures that security is baked into the code review process, not just the execution phase.

The Anatomy of a Secret Leak: Why Your Logs Aren't Safe

GitHub provides a built-in masking feature that replaces known secrets with asterisks (***) in the logs. However, this is a convenience feature, not a security boundary. Attackers can easily bypass masking by encoding the secret. If a developer runs echo $SECRET | base64, the resulting string is no longer the original secret and will not be masked. Any user with read access to the action run can decode it instantly.

Another common leak vector is the "debug dump". When a pipeline fails, developers often add run: env or run: printenv to debug the environment. This prints every single environment variable to the logs. While GitHub tries to mask the secrets, any variable that was dynamically generated or slightly modified during the build process will leak in plain text.

The most dangerous leak comes from the supply chain. If you use a third-party action like uses: some-random-user/setup-tool@v1, you are executing arbitrary code from that user's repository. If that account is compromised, the attacker can update the code in @v1 to curl your environment variables to an external server. Because the action runs with the GITHUB_TOKEN and any secrets you passed to it, the attacker gains full access without leaving a trace in your logs.

Moving from Static Secrets to OIDC

The industry standard for securing cloud access in CI/CD is OpenID Connect (OIDC). Long-lived IAM keys (the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pair) are liabilities because they never expire and are often stored as static GitHub Secrets. If these leak, they remain valid until you manually rotate them. OIDC replaces these static keys with short-lived, identity-based tokens.

With OIDC, GitHub Actions acts as an Identity Provider (IdP). When a workflow runs, it requests a JWT (JSON Web Token) from GitHub. The workflow then presents this token to the cloud provider (AWS, Azure or GCP). The cloud provider verifies the token's signature and checks if the "claims" (such as the repository name or the branch) match a pre-defined trust relationship. If they match, the provider issues a temporary security token, typically valid for one hour.

To implement this in AWS, you first create an IAM Role with a Trust Policy that trusts the GitHub OIDC provider. Then, use the official aws-actions/configure-aws-credentials action (v4). You must specify permissions: id-token: write in your YAML to allow the runner to request the JWT.

# Example: OIDC Authentication for AWS
name: Secure Deploy
on:
  push:
    branches: [ main ]

permissions:
  id-token: write # Required for requesting the JWT
  contents: read  # Required for checkout

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-role
          aws-region: us-east-1

      - name: Verify Identity
        run: aws sts get-caller-identity

The output of the last command shows the assumed role, not a static user. If this workflow is compromised, the attacker only has a temporary token that expires quickly, which reduces the blast radius significantly compared to static keys.

Hardening the Supply Chain: The Danger of Mutable Tags

Most DevOps engineers use version tags when referencing actions, such as uses: actions/checkout@v4. This looks clean, but it is a security anti-pattern. Tags in Git are mutable; a maintainer (or an attacker who has hijacked the account) can move the v4 tag to a different, malicious commit. You think you are using a trusted version, but the underlying code has changed without your knowledge.

To eliminate this risk, pin actions to a full-length commit SHA. A SHA is an immutable fingerprint of the code. If the code changes by a single character, the SHA changes. While this makes updating actions more tedious, it is the only way to guarantee that the code you audited is the code running today.

I have seen this fail in clusters with >50 nodes where a single compromised community action allowed an attacker to exfiltrate internal environment variables across dozens of repos. In a production environment with over 100 repositories, manually updating SHAs is a burden. Use a tool like Renovate Bot or Dependabot to automate these updates while keeping them pinned.

# UNSAFE: Using a mutable tag
# If the maintainer changes what @v4 points to, your pipeline is compromised.
- uses: actions/checkout@v4

# SAFE: Using a full-length commit SHA
# This code will NEVER change, regardless of what happens to the repository tags.
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

When pinning, always include a comment noting which version the SHA corresponds to. In clusters where security compliance is strict, such as those running on GKE Autopilot or hardened EKS nodes, this level of granularity is mandatory to pass SOC2 or ISO27001 audits.

Defending Against "Pwn-Requests" and Fork Attacks

One of the most overlooked vulnerabilities in GitHub Actions is the handling of Pull Requests from forks. By default, the pull_request event does not grant secrets to the runner for security reasons. However, developers often find this frustrating when they need to run integration tests that require a database key. To solve this, they use the pull_request_target event.

The pull_request_target event is extremely dangerous. Unlike pull_request, it runs in the context of the base branch (usually main) and has access to secrets. If you have a workflow triggered by pull_request_target that checks out the code from the PR branch and then runs a script, a malicious contributor can modify that script in their fork to echo $SECRET | base64. Since the workflow runs with the base branch's permissions, the attacker steals your production credentials.

To safely handle external contributions, never execute untrusted code from a fork while secrets are present. If you need to run tests on a PR, use the standard pull_request event and utilize "Environment" protections.

# DANGEROUS: Vulnerable to pwn-requests
on:
  pull_request_target:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4 # This checks out the PR code from the fork
      - run: npm install && npm test # The PR author can change 'npm test' to steal secrets
        env:
          API_KEY: ${{ secrets.API_KEY }}

The correct pattern is to require a manual approval from a maintainer before a workflow can access a protected environment's secrets. This creates a human-in-the-loop firewall that prevents automated credential theft.

Best Practices for CI/CD Hardening

To maintain a secure posture, implement these five practices across every repository in your organization.

Implement a Global Permissions Policy: Start every job with the most restrictive permissions. Use permissions: contents: read by default and only add id-token: write or packages: write when specifically required. This prevents a compromised action from deleting your repository.
Use Environment-Based Secrets: Do not put production secrets in the global "Repository Secrets" section. Create a "Production" environment and assign secrets there. This allows you to enforce "Required Reviewers", meaning no code can access production keys without a senior engineer's sign-off.
Automate Secret Scanning: Integrate Gitleaks or TruffleHog into your pipeline as a pre-commit hook or an initial CI step. These tools look for patterns (like AKIA... for AWS) and fail the build if a secret is detected in the commit history.
Avoid Secret Passing via Env: Instead of passing secrets as environment variables to every step, pass them only to the specific step that needs them. This minimizes the number of processes that have the secret in their memory space.
Rotate Credentials Every 90 Days: Even with OIDC, some legacy systems require static keys. Implement a strict rotation policy. If a key is not rotated regularly, a leak might go undetected for months, giving attackers a permanent backdoor.

FAQ

Does GitHub really mask all my secrets in the logs?

No. GitHub only masks the exact string stored in the secret. If your code transforms the secret (e.g., base64 encoding, URL encoding or splitting the string), the resulting output will not be masked. Never rely on masking as a primary security control.

Why is `pull_request_target` worse than `pull_request`?

pull_request runs in the context of the merge commit and has no access to secrets from the base repository. pull_request_target runs in the context of the base branch and has full access to secrets, meaning any code introduced by a contributor in a fork can access those secrets if the workflow executes that code.

Should I use OIDC for every single cloud provider?

Yes. Every major provider (AWS, Azure, GCP and HashiCorp Vault) now supports OIDC for GitHub Actions. Moving away from static JSON keys or CSV credential files reduces your operational overhead and eliminates the risk of "stale" credentials living in your repository settings.

Can I still use version tags like `@v4` if I use a private runner?

Yes, but it is still a bad practice. Even on a private runner, a compromised third-party action can exfiltrate data from your internal network or steal the GITHUB_TOKEN to modify your source code. The location of the runner does not protect you from supply chain attacks.

Conclusion

Securing GitHub Actions requires moving away from the "trust by default" mindset. The combination of OIDC for identity, SHA pinning for supply chain integrity and strict permissions blocks creates a defense-in-depth strategy. The most critical immediate step you can take is auditing your workflows for pull_request_target and replacing static cloud keys with OIDC roles.

Start by implementing these three actionable steps today: first, replace all v* tags with commit SHAs in your most critical deployment pipeline. Second, migrate your production cloud authentication to OIDC to eliminate long-lived keys. Third, configure GitHub Environments with mandatory reviewers for all production secrets. By shifting security left into your CI/CD configuration, you ensure that your pipeline is a tool for delivery, not a liability.

Cursor vs Copilot vs Cody: Best AI Editor for DevOps

DevOps Start — Mon, 20 Apr 2026 21:41:26 +0000

Choosing the right AI editor for DevOps is about more than just autocomplete—it's about codebase context. Originally published on devopsstart.com, this guide compares Cursor, Copilot, and Cody for IaC and Kubernetes workflows.

Introduction

Choosing an AI code assistant for DevOps isn't about who can write the cleanest Python function; it's about who understands the relationship between your variables.tf, your Helm charts and your GitHub Actions workflow. Most AI tools are built for application developers, which means they often fail when faced with the fragmented nature of infrastructure. If you've ever had Copilot suggest a deprecated Terraform provider or a Kubernetes API version that hasn't existed since 1.16, you know the "context problem" firsthand.

In this guide, you'll learn how to navigate the trade-offs between GitHub Copilot, Cursor and Sourcegraph Cody specifically through the lens of a Platform or DevOps engineer. We will dive into how each tool handles codebase indexing, how they manage the hallucinations common in YAML and HCL, and which one actually helps you reduce "time to first green build" in a complex CI/CD pipeline. By the end, you'll have a clear decision matrix to determine which tool fits your specific organizational scale, security requirements and infrastructure complexity. Whether you are managing a handful of scripts or a massive polyglot monorepo, the right choice depends on how the AI "sees" your architecture.

The Context Problem: Why General AI Fails DevOps

DevOps engineers don't write linear code; they build distributed systems. A single change in a Terraform module might require updates to a Kubernetes manifest and a corresponding change in a CI pipeline. Standard AI completions fail here because they typically rely on "active tab" context. If you are editing deployment.yaml but the relevant environment variable is defined in terraform/outputs.tf (which is closed), the AI is guessing based on generic internet patterns, not your actual architecture.

For example, imagine you are trying to reference a secret created by an ExternalSecrets operator. A generic AI will suggest a standard Kubernetes Secret syntax. A context-aware AI knows you are using ExternalSecret objects and will suggest the correct API group. This is the difference between a tool that saves you five seconds of typing and a tool that prevents a production outage. To solve this, tools have moved toward Retrieval-Augmented Generation (RAG), which indexes your local or remote files to provide actual project awareness. You can read more about the complexities of managing these environments in the Kubernetes v1.36 Features, Deprecations & Upgrade Guide to see why version-specific context is so critical.

Consider this scenario: you need to add a new resource to a Terraform module that already has a strict naming convention and specific tagging requirements defined in a separate locals.tf file.

# locals.tf
locals {
  common_tags = {
    Environment = var.env
    Project     = "Phoenix"
    ManagedBy   = "Terraform"
  }
}

# main.tf
# You start typing: resource "aws_s3_bucket" "logs" {
# A context-blind AI suggests: tags = { Name = "logs" }
# A context-aware AI suggests: tags = local.common_tags

When the AI knows your locals.tf exists, it stops hallucinating generic tags and starts following your internal standards. This eliminates the manual "copy-paste" cycle that often leads to inconsistent infrastructure and failed compliance checks.

Cursor: The AI-Native Powerhouse for IaC

Cursor is not a plugin; it is a fork of VS Code. This architectural choice is a game changer for DevOps engineers because it allows the AI to integrate deeply with the IDE's indexing engine. While Copilot feels like a sophisticated autocomplete, Cursor feels like a pair programmer that has actually read your entire repository. It uses a local index of your files, meaning when you ask it to "Add a new environment to the staging cluster," it scans your existing .tfvars and kustomize overlays to mirror the pattern exactly.

For those managing complex Terraform projects, Cursor's @Codebase feature is indispensable. You can prompt the AI to analyze the relationship between different modules without opening every file. This is particularly useful when you are implementing Terraform Testing Best Practices and need the AI to generate test cases based on the actual resource dependencies. In clusters with >100 nodes, where naming conventions are strict and dependencies are deep, this level of indexing prevents the "hallucinated resource" error that plagues plugin-based assistants.

Here is how you would actually use Cursor to refactor a Kubernetes manifest to use a new ConfigMap source:

# In Cursor, you use the Cmd+K (or Ctrl+K) interface.
# Prompt: "@Codebase update all deployments in /k8s/overlays/prod to use the 
# new configmap-v2 defined in configmap.yaml"

# Cursor identifies all files in the directory and applies the change:
# Before:
# configMapRef:
#   name: app-config
# After:
# configMapRef:
#   name: app-config-v2

The magic here is that Cursor doesn't just find and replace text; it understands that the configMapRef is a Kubernetes object property. It maintains the indentation of your YAML (which is the bane of every DevOps engineer's existence) and ensures that the change is consistent across all target files. This removes the tedious manual verification usually required after a bulk edit.

Sourcegraph Cody: Mastering the Enterprise Monorepo

While Cursor excels at local indexing, Sourcegraph Cody is designed for the enterprise scale. Many Platform teams work in massive polyglot monorepos where the Terraform code is in one directory, the Go-based operator is in another and the documentation is in a separate Wiki or GitHub Pages site. Cody's strength lies in its ability to pull context from remote repositories and external documentation via the Sourcegraph index.

Cody is the "Enterprise Context King" because it doesn't just look at your open files; it looks at your entire organization's knowledge graph. If your company has a proprietary way of handling VPC peering or a specific wrapper around Pulumi, Cody can be configured to prioritize those internal patterns over generic public documentation. This is vital for SOC2 or HIPAA compliant environments where "following the internal standard" is not a suggestion, but a legal requirement.

Imagine you are tasked with updating a CI pipeline using a custom internal GitHub Action that isn't documented on the public web.

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Internal Deploy
        uses: my-corp/deploy-helper@v2 # Cody knows this action exists in your org
        with:
          cluster_id: ${{ secrets.CLUSTER_ID }}
          # Cody suggests the 'environment' input because it indexed 
          # the 'deploy-helper' repo in the same organization.
          environment: 'production'

By indexing the my-corp/deploy-helper repository, Cody provides suggestions for inputs and outputs that GitHub Copilot would simply guess. This reduces the need to constantly switch between your editor and the internal documentation browser. For teams implementing GitOps Testing Strategies, Cody can help bridge the gap between the ArgoCD configuration and the underlying Kubernetes manifests by tracing the logic across different repositories.

Comparing AI Performance on YAML and HCL

When it comes to Infrastructure as Code (IaC), the biggest risk is the "confidently wrong" suggestion. HCL (HashiCorp Configuration Language) and YAML are whitespace-sensitive and schema-dependent. GitHub Copilot is generally the fastest for simple snippets, but it is the most prone to hallucinating API versions. For example, it might suggest apiVersion: extensions/v1beta1 for an Ingress resource, which has been deprecated for years.

Cursor and Cody perform better here because they can be anchored to specific versions of your codebase. If your project specifies Terraform v1.7.0 in a .terraform-version file, Cursor is more likely to suggest syntax compatible with that version. In a head-to-head comparison for generating a complex Kubernetes NetworkPolicy, Cursor typically wins on formatting, while Cody wins on referencing your existing network architecture.

Let's look at a practical comparison of how these tools handle a request to create a Kubernetes Service of type LoadBalancer with specific cloud annotations for AWS.

# Prompt: "Create a LoadBalancer service for the 'api' deployment with AWS NLB annotations"

# Copilot: Often gives a generic LoadBalancer without the specific 
# service.beta.kubernetes.io/aws-load-balancer-type: nlb annotation.

# Cursor: Checks your other services, sees you use 'nlb-ip' mode, and suggests:
# annotations:
#   service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"
#   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"

# Cody: References the official AWS Load Balancer Controller docs (if indexed)
# and suggests the most current annotation for your specific K8s version.

The "hallucination risk" in Kubernetes is particularly high because the API evolves so rapidly. A tool that relies on a training set from 2022 will lead you toward deprecated fields. A tool that uses RAG to look at your current kubectl version or your manifest files will guide you toward the current standard.

Best Practices for AI-Driven DevOps

To get the most out of these tools without introducing security vulnerabilities or infrastructure drift, you must treat AI output as a "proposed change" rather than "final code." Follow these guidelines to maintain stability.

Use Version Pinning in Prompts: Never just ask for a "Terraform script." Specify the version. Use prompts like "Using Terraform v1.7.x and the AWS provider v5.0, create a VPC..." This forces the AI to narrow its search space and reduces the likelihood of deprecated syntax.
Verify with Static Analysis: AI is great at writing code but terrible at verifying it. Always pipe AI-generated HCL through terraform validate and YAML through kube-linter or datree. This catches the small indentation errors that AI frequently introduces.
Context-Seed Your Prompts: In Cursor or Cody, explicitly tag the files that define your architecture. Instead of "Fix this error," use "@variables.tf @main.tf fix the mismatch in the subnet ID." This provides the RAG engine with a direct path to the answer.
Sanitize Secrets Before Indexing: Ensure your .gitignore is robust. While most modern AI editors respect .gitignore, double-check that you aren't indexing .terraform.lock.hcl or temporary state files that might contain sensitive metadata.
Iterative Refinement: Start with a high-level architecture prompt, then drill down into specific resources. Asking an AI to "Write my entire EKS cluster" usually results in a mess. Ask it to "Define the VPC," then "Define the EKS cluster using that VPC," and finally "Add the node groups."

FAQ

Which AI editor is the most secure for corporate code?

Sourcegraph Cody generally leads in enterprise security because it offers robust controls over where data is stored and how it is indexed. For organizations with strict data residency requirements, Cody's ability to run on-premises or in a private cloud is a major advantage. Cursor and Copilot have "Privacy Modes" that promise not to train on your data, but for SOC2/HIPAA environments, the transparency of Cody's indexing layer is typically more acceptable to security auditors.

Can these tools actually replace writing Terraform by hand?

No, and attempting to do so is dangerous. AI is excellent at boilerplate (creating 10 similar S3 buckets) and translation (converting a Helm chart to a Kustomize overlay), but it cannot reason about your business logic or the cost implications of a specific instance type. Use AI to handle the "syntax toil" while you handle the "architectural intent."

How do I stop the AI from suggesting deprecated Kubernetes APIs?

The best way is to provide a "source of truth" file in your repository. Create a K8S_STANDARDS.md file that lists your cluster version and preferred API versions. In Cursor or Cody, refer to this file using @K8S_STANDARDS.md in your prompt. This overrides the AI's general training data with your specific project requirements.

Does using a fork like Cursor break my VS Code extensions?

Since Cursor is a fork of VS Code, it is compatible with almost all VS Code extensions. You can import your existing themes, keybindings and plugins (like the HashiCorp Terraform extension) directly. The primary difference is the built-in AI layer, which replaces the need for a separate Copilot plugin.

Conclusion

The transition from "AI as a plugin" to "AI as an environment" is the most significant shift in DevOps productivity since the rise of GitOps. GitHub Copilot remains a solid choice for generalists who want a low-friction experience. However, for the specialized needs of a Platform Engineer, Cursor's local codebase indexing provides a level of precision in HCL and YAML that plugins cannot match. For those operating at a massive corporate scale, Sourcegraph Cody's remote context capabilities make it the only viable choice for navigating polyglot monorepos.

Your next step should be a two-week trial: install Cursor for your local feature development to see if the @Codebase indexing reduces your context-switching. Simultaneously, if you are in a large team, evaluate Cody's ability to index your internal documentation. Once you've chosen your tool, integrate a static analysis step into your CI pipeline to ensure that AI-generated speed doesn't come at the cost of production stability. Stop fighting with YAML indentation and start leveraging the context of your entire architecture.

Build an Internal Developer Platform with Backstage and

DevOps Start — Mon, 20 Apr 2026 21:36:22 +0000

Stop the 'ticket-ops' madness! This guide, originally published on devopsstart.com, shows you how to combine Backstage and Crossplane to build a true self-service Internal Developer Platform.

Introduction

Stop forcing your developers to learn the intricacies of cloud provider consoles or struggle with 500-line Terraform modules just to get a database. The gap between raw infrastructure and developer productivity is where "ticket ops" thrives, slowing down deployment cycles and frustrating engineers. To solve this, you need an Internal Developer Platform (IDP) that abstracts infrastructure complexity into a self-service experience.

An IDP allows developers to provision resources via a simplified interface without needing to be cloud experts. In this guide, you will learn how to build a production-ready IDP by combining Backstage and Crossplane. Backstage acts as your front-end portal, providing a unified interface for service discovery and software templates. Crossplane serves as the back-end control plane, turning Kubernetes into a universal API for managing cloud resources.

By the end of this article, you will understand the architecture required to move from manual Infrastructure as Code (IaC) to a scalable Infrastructure as a Service (IaaS) model. You'll see exactly how to map a button click in a UI to a live AWS RDS instance via GitOps, reducing the cognitive load on your developers while maintaining strict governance for your platform team. For more on managing the underlying clusters, you can check out Kubernetes for Beginners: Deploy Your First Application.

The Architecture: Connecting Backstage to Crossplane

Building an IDP isn't about one tool; it's about the pipeline. The most common mistake is trying to connect Backstage directly to a cloud API. That is a security nightmare and lacks auditability. Instead, use a GitOps-driven control plane architecture. In this flow, Backstage doesn't "create" the infrastructure; it "requests" it by committing a manifest to Git.

The sequence works as follows: a developer selects a "Provision Postgres" template in the Backstage Scaffolder. Backstage then triggers a commit of a simple YAML file to a Git repository. An automated GitOps controller, such as ArgoCD, detects this change and syncs the manifest to a Kubernetes cluster. Inside that cluster, Crossplane v1.14.x sees the new Custom Resource (CR) and communicates with the cloud provider's API to provision the actual resource.

This ensures that your Git history is the single source of truth, which is critical for compliance and disaster recovery. To ensure these deployments are handled reliably, you should learn How to Set Up Argo CD GitOps for Kubernetes Automation.

The "connective tissue" here is the YAML schema. Backstage must output a manifest that exactly matches the CompositeResourceDefinition (XRD) you've defined in Crossplane. If the Scaffolder outputs db_size: small but Crossplane expects storageClass: small, the request will hang in a "Pending" state. You must treat your XRDs as the API contract between your platform team and your developers.

Abstracting Cloud Complexity with Crossplane Compositions

If you give developers raw Crossplane resources, you've just traded Terraform for Kubernetes YAML, which does not reduce cognitive load. The real power of Crossplane lies in Compositions. A Composition allows you to bundle multiple low-level resources (like a VPC, a Subnet, and an RDS instance) into a single, high-level "Composite Resource" (XR) that developers can actually understand.

For example, instead of requiring a developer to specify db.aws.upbound.io/v1beta1 with 20 mandatory fields, you create a CompositeDatabase definition. The developer only provides a name and a size. Your platform team defines the "blueprint" that maps size: small to a t3.micro instance with 20GB of encrypted GP3 storage.

Here is an example of a simplified CompositeResourceDefinition (XRD) that defines the API your developers will use:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresdatabases.platform.example.org
spec:
  group: platform.example.org
  names:
    kind: XPostgresDatabase
    plural: xpostgresdatabases
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              storageGb:
                type: integer
              region:
                type: string

And here is how the developer's request (the "Claim") looks. This is the exact YAML that Backstage will generate:

apiVersion: platform.example.org/v1alpha1
kind: PostgresDatabase
metadata:
  name: order-service-db
  namespace: order-service-prod
spec:
  storageGb: 20
  region: us-east-1

By using this approach, you eliminate the need for developers to know AWS-specific jargon. You can change the underlying instance type or backup policy in the Composition without ever touching the developer's manifest.

Implementing the Backstage Scaffolder for Self-Service

The Backstage Scaffolder is the engine that turns a user's form input into a Git commit. To make this work with Crossplane, you create a template.yaml file. This template defines the UI form (the questions you ask the developer) and the "steps" required to process the answer.

In a production setup, your template should not just create a file; it should validate the input. For example, if a developer requests 10,000GB of storage, your template or a validating admission webhook in Kubernetes should catch it. The template uses "Nunjucks" templating to inject the form values into the Crossplane Claim YAML.

Below is a snippet of a Backstage software template designed to provision a Crossplane database:

apiVersion: backstage.io/template/scaffolder-entity/v1.0.0
metadata:
  name: provision-rds-postgres
  title: Provision RDS Postgres
  description: Creates a production-ready Postgres DB via Crossplane
spec:
  parameters:
    - title: Database Details
      properties:
        dbName:
          type: string
          title: Database Name
        storageGb:
          type: integer
          title: Storage Size (GB)
          default: 20
        environment:
          type: string
          title: Environment
          enum: [dev, staging, prod]
  steps:
    - id: fetch-base
      action: fetch:template
      input:
        templateRepo: templates/infrastructure/rds
        values:
          name: ${{ parameters.dbName }}
          storage: ${{ parameters.storageGb }}
          env: ${{ parameters.environment }}
    - id: publish
      action: publish:github
      input:
        allowedStatuses: [success]
        repoUrl: github.com?owner=my-org&repo=${{ parameters.dbName }}-infra

When the developer clicks "Create," Backstage creates a new repository (or updates an existing one) with the resulting YAML. The critical part is the fetch:template step. It takes the generic claim.yaml from your template repository and fills it with the user's specific requirements. This removes the possibility of syntax errors in the YAML, as the developer never actually writes the code.

The GitOps Feedback Loop and Production Gotchas

A major pain point in IDPs is the "black hole" effect: a developer clicks a button in Backstage, the commit happens, and then nothing. They have no idea if the database is actually ready or if the Crossplane provider is stuck in a back-off loop. To solve this, you must implement a feedback loop.

One effective method is using the Backstage Kubernetes plugin combined with the Crossplane status fields. Crossplane updates the status section of the Claim resource once the cloud provider confirms the resource is Ready: True. You can configure Backstage to surface these Kubernetes resource statuses directly on the service's catalog page. If a resource is failing, the developer sees a "Warning" status in the portal, which links them to the logs.

In clusters with >100 nodes, you'll notice that Crossplane's reconciliation loop can put significant pressure on the Kubernetes API server. I've seen cases where too many frequent updates to the status of 500+ cloud resources caused API latency. To mitigate this, tune the pollInterval in your Crossplane providers. Don't check every 60 seconds if a database is ready; 5 or 10 minutes is usually sufficient for infrastructure that takes 15 minutes to provision.

Another production gotcha is "orphaned resources." If a developer deletes the manifest from Git, ArgoCD deletes the Claim from Kubernetes, and Crossplane deletes the RDS instance. This is great for dev environments but catastrophic for production. You must implement a "deletion policy" in your Compositions. Set deletionPolicy: Orphan for production workloads. This ensures that if the YAML is accidentally deleted, the actual cloud resource remains intact.

Best Practices for Platform Engineering

Implementing an IDP is more of an organizational challenge than a technical one. If you build a perfect platform that no one uses, you've failed. Follow these principles to ensure adoption:

Start with the "Golden Path": Do not try to automate every possible cloud resource on day one. Identify the three most requested resources (for example, S3 buckets, Postgres DBs, and Redis caches) and build high-quality templates for those. This provides immediate value and builds trust.
Enforce Governance via Compositions: Use Crossplane Compositions to bake in security. Ensure every S3 bucket is encrypted and every RDS instance is in a private subnet by default. The developer shouldn't even see the "Encryption" checkbox; it should be mandatory and invisible.
Treat your IDP as a Product: Your developers are your customers. Conduct user interviews to find where the friction is. If they find the Backstage form too long, simplify it. If they need more visibility into costs, integrate a cost-tracking plugin.
Implement Strong RBAC: Use Kubernetes namespaces to isolate claims. Ensure that a developer in the team-a namespace cannot modify a PostgresDatabase claim in the team-b namespace. Use a tool like Kyverno to enforce these boundaries.
Version your Compositions: When you update a Composition (for example, upgrading the RDS instance class), don't just push it to production. Version your XRDs and Compositions so you can migrate services gradually rather than forcing a global update.

FAQ

How does this approach differ from using Terraform with a CI/CD pipeline?
Traditional Terraform requires a "push" model where a pipeline runs terraform apply. This often leads to state locking issues and configuration drift. The Backstage + Crossplane approach uses a "pull" model (Control Plane). Crossplane constantly monitors the state of the cloud and automatically corrects drift without needing a manual pipeline trigger.

Does this mean I have to migrate all my existing Terraform code to Crossplane?
No. You can run them side-by-side. Use Crossplane for new, self-service workloads while keeping your core networking and foundation (VPCs, IAM roles) in Terraform. You can even use the Terraform provider for Crossplane to manage existing Terraform modules through the Kubernetes API.

What happens if the cloud provider API is down during provisioning?
Crossplane employs an exponential back-off strategy. If the AWS API returns a 500 error, Crossplane will keep retrying the request. The Kubernetes resource will stay in a Synced: False state. Because you have a GitOps audit trail, you can easily see which resources are stuck.

Is Backstage overkill for small teams?
If you have fewer than five developers, a simple README and a set of shared Terraform modules might suffice. However, once you hit a scale where the platform team becomes a bottleneck for "simple" requests, the investment in Backstage pays off by eliminating the ticket queue.

Conclusion

Combining Backstage and Crossplane allows you to move from a culture of "ticket-based infrastructure" to true self-service. By using Backstage as the user interface and Crossplane as the control plane, you create a system where developers can provision production-ready resources in minutes, not days. This doesn't just speed up delivery; it allows your platform engineers to stop performing repetitive manual tasks and start focusing on high-value architectural improvements.

To get started, your first actionable step is to install Crossplane v1.14.x on a development cluster and create your first CompositeResourceDefinition for a simple resource, like an S3 bucket. Once the API is working, set up a basic Backstage instance and create a software template that outputs the YAML required by that XRD. Start small, validate the "Golden Path" with one team, and then scale the platform to the rest of your organization.

Essential kubectl Commands Cheat Sheet

DevOps Start — Tue, 14 Apr 2026 15:36:24 +0000

Stop memorizing every flag! I've put together a handy kubectl cheat sheet for managing pods, deployments, and debugging. Originally published on devopsstart.com.

Pod Management

Command	Description
`kubectl get pods`	List all pods in current namespace
`kubectl get pods -A`	List pods across all namespaces
`kubectl describe pod <name>`	Show detailed pod information
`kubectl delete pod <name>`	Delete a specific pod
`kubectl logs <pod>`	View pod logs
`kubectl logs <pod> -f`	Stream pod logs
`kubectl exec -it <pod> -- sh`	Open shell in pod

Deployments

Command	Description
`kubectl get deployments`	List all deployments
`kubectl scale deploy <name> --replicas=3`	Scale a deployment
`kubectl rollout status deploy/<name>`	Check rollout status
`kubectl rollout undo deploy/<name>`	Rollback a deployment
`kubectl set image deploy/<name> <container>=<image>`	Update container image

Services and Networking

Command	Description
`kubectl get svc`	List all services
`kubectl get ingress`	List all ingress resources
`kubectl port-forward svc/<name> 8080:80`	Forward local port to service
`kubectl get endpoints`	List service endpoints

Debugging

Command	Description
`kubectl get events --sort-by=.metadata.creationTimestamp`	View cluster events
`kubectl top pods`	Show pod resource usage
`kubectl top nodes`	Show node resource usage
`kubectl logs <pod> --previous`	View logs from crashed container
`kubectl describe node <name>`	Check node conditions

Context and Config

Command	Description
`kubectl config get-contexts`	List all contexts
`kubectl config use-context <name>`	Switch context
`kubectl config current-context`	Show current context
`kubectl get ns`	List namespaces
`kubectl config set-context --current --namespace=<ns>`	Set default namespace

Debug Kubernetes CrashLoopBackOff in 30 Seconds

DevOps Start — Tue, 14 Apr 2026 15:31:20 +0000

Struggling with a pod stuck in CrashLoopBackOff? This quick guide, originally published on devopsstart.com, shows you the exact commands to diagnose the root cause in seconds.

The Problem

Your pod is stuck in CrashLoopBackOff and you need to find out why — fast.

The Solution

kubectl logs <pod-name> --previous

The --previous flag shows logs from the last crashed container instance. This is the single most useful flag for debugging crash loops.

Combine with describe for the full picture

kubectl describe pod <pod-name> | grep -A 5 "Last State"

This shows the exit code and reason for the last termination:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

Common Exit Codes

Code	Meaning	Fix
1	Application error	Check app logs
137	OOMKilled	Increase memory limits
139	Segfault	Check binary compatibility
143	SIGTERM	Graceful shutdown issue

Why It Works

Kubernetes keeps logs from the previous container instance even after it crashes. Without --previous, you'd only see logs from the current (possibly empty) instance that hasn't had time to produce output before crashing again.

Rapid Rollback: `kubectl set image` for Urgent Fixes

DevOps Start — Tue, 14 Apr 2026 15:26:17 +0000

Originally published on devopsstart.com. When production breaks, every second counts—here is how to use kubectl set image for a precise and rapid rollback.

Introduction

You've just deployed a new container image to production, and almost immediately, monitoring alerts start screaming. Latency is spiking, error rates are through the roof, and your customers are experiencing service degradation. In these high-pressure moments, a fast, reliable rollback mechanism is critical. While Kubernetes offers robust rollout and rollback capabilities via kubectl rollout undo, there are specific scenarios where kubectl set image can provide a quicker, more direct path to recovery, especially when you know exactly which image version you need to revert to.

This tip focuses on leveraging kubectl set image for urgent rollbacks. You'll learn when this command is most effective, how to accurately identify the correct previous image tag, and how to execute the command to quickly stabilize your application.

Understanding `kubectl set image`

The kubectl set image command is primarily designed to atomically update the image of one or more specific containers within a Kubernetes resource. It typically targets Deployments, StatefulSets, DaemonSets, or ReplicationControllers. When executed, it modifies the resource's Pod template to point to the new image tag, which then triggers a new rolling update.

While kubectl set image is frequently used for forward deployments (e.g., updating v1.1.9 to v1.2.0), its direct nature makes it exceptionally well-suited for rapid rollbacks. When you specify a previous, stable image, Kubernetes initiates a new rollout toward that desired state. This behavior differentiates it from kubectl rollout undo, which inherently steps back through the deployment's recorded history, revision by revision.

Here’s a common example of how kubectl set image is used to update an image:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.2.0 -n production

Expected output:

deployment.apps/my-app image updated

This command updates my-container within the my-app deployment in the production namespace to use the v1.2.0 image from my-registry.

The Urgent Rollback Scenario

Consider this scenario: your my-app:v1.2.0 release introduced a critical bug that bypassed your staging environment checks. You pushed it to production an hour ago, and now, critical alerts are firing, indicating significant application failures. You need to revert to the last known good image, let's say my-app:v1.1.9, immediately.

Why might kubectl set image be preferred over kubectl rollout undo in such a situation?

Directness and Precision: If you know the exact, stable image tag to which you need to revert, kubectl set image offers an explicit and precise command. This avoids ambiguity and ensures you land on the intended stable state directly.
Bypassing Unhealthy Revisions: If multiple faulty deployments occurred after your last stable one (e.g., you tried v1.2.0, then v1.2.1-hotfix, both failed), kubectl rollout undo would sequentially step back through these potentially problematic revisions. kubectl set image allows you to jump directly to the known good v1.1.9 without traversing the unstable intermediate states.
Forced Redeploy (Edge Cases): In rare cases, even if an image tag is theoretically the same, you might want to force Kubernetes to re-pull container images and redeploy pods due to local caching issues or other inconsistencies. Re-setting the image explicitly with kubectl set image can achieve this, ensuring fresh pods are created. For more on debugging common Kubernetes issues, refer to our article on Troubleshooting CrashLoopBackOff in Kubernetes.

Identifying the Previous Image Tag

The critical first step for a kubectl set image rollback is accurately identifying the last known good image tag. You can achieve this by inspecting your deployment's revision history:

Check Rollout History:
This command provides a concise summary of your deployment's revision history, showing the changes made at each step.
```
kubectl rollout history deployment/my-app -n production
```


shell

    **Expected output:**


```bash
    deployment.apps/my-app 
    REVISION  CHANGE-CAUSE
    1         <none>
    2         my-container: my-registry/my-app:v1.1.8
    3         my-container: my-registry/my-app:v1.1.9
    4         my-container: my-registry/my-app:v1.2.0

From this output, if `v1.2.0` (revision 4) is currently causing issues, then `v1.1.9` (revision 3) is your immediate target for rollback. Note that `CHANGE-CAUSE` may also contain details if `--record` was used during deployment.

Describe a Specific Revision (Optional Verification):
To be absolutely certain about the container images used in a particular revision, you can describe it in detail. This is a good verification step before initiating a rollback.
```
kubectl rollout history deployment/my-app --revision=3 -n production
```


shell

    **Expected (truncated) output:**


```bash
    deployment.apps/my-app with revision 3
    Pod Template:
      Labels:       app=my-app
                    pod-template-hash=54c9c76...
      Containers:
        my-container:
          Image:        my-registry/my-app:v1.1.9
          Port:         8080/TCP
          Environment:  <none>
    ...

This confirms that `my-registry/my-app:v1.1.9` was indeed the image used for revision 3, making it a reliable rollback target.

Executing the `kubectl set image` Rollback

Once you have identified the precise desired image tag (e.g., my-registry/my-app:v1.1.9 in our example), executing the rollback is straightforward and immediate:

kubectl set image deployment/my-app my-container=my-registry/my-app:v1.1.9 -n production

Expected output:

deployment.apps/my-app image updated

Upon execution, Kubernetes will immediately initiate a new rolling update. It will begin replacing the currently failing v1.2.0 pods with new ones running the specified stable v1.1.9 image.

You can monitor the progress of this new rollout using the following command:

kubectl rollout status deployment/my-app -n production

Expected output during rollout:

Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out

Once the rollout is complete, your application should be consistently running the stable v1.1.9 image, and your monitoring alerts should ideally begin to subside as service is restored.

Important Considerations

Rollback Strategy Impact: This kubectl set image method performs a rolling update. It's crucial that your application is designed to handle a brief period where both the old (problematic) and new (stable) versions of pods are running concurrently. This typically means ensuring backward and forward compatibility for APIs and data schemas.
Image Immutability: Always strive to use immutable image tags (e.g., v1.1.9, v1.2.0, sha256:abcdef...) rather than mutable tags like latest. Immutable tags guarantee that a specific tag always refers to the exact same image content, which is fundamental for reliable and reproducible rollbacks.
Auditing and History: Using kubectl set image creates a new revision in the deployment's history. This automatically ensures that your rollback action is recorded, providing a clear audit trail of changes made to your deployment.
Stateful Workloads: For StatefulSets, exercising caution when changing image versions is paramount. If a new image version introduces changes that affect persistent storage or state, a simple image rollback might not fully resolve database schema migrations or data portability issues. Always understand the data implications.

Conclusion

When a problematic image release throws production into disarray, reaction time is paramount. While kubectl rollout undo is a valuable tool, kubectl set image provides a direct, efficient, and precise alternative for reverting to a specific, known-good image. This capability can significantly reduce Mean Time To Recovery (MTTR) by allowing you to bypass potentially multiple failing revisions and jump straight to stability. By understanding your deployment history and precisely targeting the last stable