Forem: Jatin Mehrotra

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

Jatin Mehrotra — Mon, 08 Dec 2025 11:56:45 +0000

Jatin Mehrotra for AWS Community Builders

Dec 6 '25

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

#ai #aws #agents #awsdevopsagent

Comments 12

8 min read

AWS DevOps Agent is not here to replace you! It's here to compliment you by acting like 24/7 on call engineers finding root cause for issues while you build new feature and innovate. More details in the blog on how to setup with 2 Demo.

Jatin Mehrotra — Mon, 08 Dec 2025 10:56:34 +0000

Jatin Mehrotra for AWS Community Builders

Dec 6 '25

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

#ai #aws #agents #awsdevopsagent

Comments 12

8 min read

AWS DevOps Agent Explained: Architecture, Setup, and Real Root-Cause Demo (CloudWatch + EKS)

Jatin Mehrotra — Sat, 06 Dec 2025 22:19:36 +0000

Amazon Web Services(AWS) launched Frontier agents that are autonoumous systems during re:Invent 2025 which achieve various use cases goals, scale massively to tackle concurrent tasks, and run persistently for hours or days without intervention.

In this blog we will talk about one of the frontier agents i.e AWS DevOps Agent !!!

This blog will explain what is AWS DevOps Agent, its architecture & components, security & demo for investigating Ec2 CPU spike (CW alarm) and EKS Pod error.

People can also jump straight in to investigating Cloudwatch alarms or EKS errors plus there is tf repo as well for EKS

In the end also shared DevOps Engineer Perspective on the fear of being replaced and its a must read.

Architecture
What is AWS DevOps Agent
How to maximize Agent's Effectiveness
AWS DevOps Agent working architecture
Resource Discovery by AWS DevOps Agent
Demo 1: Investigate Cloudwatch Alarm
Demo 2: Investigate EKS errors
DevOps Engineer perspective

Architecture

What is AWS DevOps Agent

Think of AWS DevOps Agent like an 24/7 continuously learning Autonomous on call Engineer which has all the tools(more about it later) primarily to investigate incidents, finds root causes, provides mitigation plan and provide you Prevention as well.

At the moment it cannot fix the incidents on its own. Of course a human will be needed to fix the root cause.

How it investigates and find the root cause

To investigate root cause and give recommendations it needs understanding and relationships of the infrastructure & applications called as topology inside AWS account.

The understanding of infrastructure holistically becomes the CONTEXT for this DevOps Agent.

How to maximize Agent's Effectiveness

While the topology provides important context during investigations, AWS DevOps Agent is not limited to investigating only the resources shown in the topology.

The agent may use additional data sources, such as AWS service APIs or connected observability tools, to investigate resources that are not in the application topology.

And that is why AWS has given option to add capabilities to maximize Agent's effectiveness by :

Connect multiple AWS accounts
Connect CI/CD pipelines through repo like Github/GitLab
MCP servers
Telemetry sources like Datadog, New Relic
Ticketing and chat like serviceNow and slack
Even EKS (demo over here)

Note: We can also provide runbooks as pre-loaded guidance/hints to enhance investigation performance to provide investigation hints and guidance.

IN THE END IT ALL ABOUT UNDERSTANDING RELATIONSHIPS ABOUT YOUR RESOURCES

AWS DevOps Agent working architecture [IMP]

Operates through a dual-console architecture.
Admins uses management Console to create and manage Agent Spaces,configure capabilities & set up access controls.
Operations teams uses AWS Agent web app to interact with agent and start investigation.

DevOps Agent Spaces

DevOps Agent spaces is logical container/boundary that defines what tools and infrastructure AWS DevOps Agent has access to.

When you create an Agent Space, you define which AWS accounts the agent can access, which external tools it can connect to, and which users in your organization can interact with the agent.

Admins configure the Agent Space through the AWS Management Console

Security Aspect of Agent Spaces:

Each Agent Space uses dedicated IAM roles that grant access only to specific AWS accounts and resources
You control which users or groups can access each Agent Space.
Information from one Agent Space is not visible or accessible from another Agent Space

DevOps Agent Web App

Operations team uses web apps for daily incident response activities.

Security Aspect of Web Apps:

IAM identity Center (user Access): centrally manage user access to the DevOps Agent Space web apps even federate with external identity providers. MFA support is included
IAM authentication link (Admin access): direct access to the web app from the AWS Management Console using your existing console session.

Resource Discovery by AWS DevOps Agent

Until this point we understand Agent's context starts with Resource discovery(topology) and it does by 2 ways

CloudFormation stacks: By default Agent will list all of the CloudFormation stacks and their resources. Resources created by CDK is also supported.
Resource Tags: Resources not deployed from CloudFormation (like Console or TF), will be discovered by AWS Tag Key and value pairs to include in topology.

Demo 1: Investigate Cloudwatch Alarm

Pre-requisite

For the brevity of the blog I won't cover how to create agent space. Pretty straightforward and you don't need aws organizations to do this.
Cloudformation basics
Access to us-east-1 region. DevOps agent is available only in this region.
I used single standalone account.

1) Investigate Cloudwatch Alarm

Deploy the cloudformation stack.

This CFN template is creating an security group, key-pair for ssh ec2 instance with startup script to do CPU stress test, CW alarm for CPU utilization, auto shutdown ec2 instances after 2 hours.
After ec2 instance is created wait for 5-10 minutes to trigger cloudwatch alarm
You can also SSH into instance and run the stress test manually as well ./cpu-stress-test.sh

Once stack is deployed DevOps Agent automatically identifies the new resources.

Access Web App

After creating agent spaces click on View Details

Click on Web app or directly Operator Access link

Investigation Root Cause

Once your Cloudwatch Alarm is triggered go to Web app and under incident Response click on latest alarm and hit on Start Investigation

_DevOps Agent is smart enough to fill the completed prompt. Agent will figure out the steps on its own and give you the final ROOT CAUSE
_

Well I also found an interesting finding when the time between 2 alarms was around 40 minutes agent was unable to find the root cause and I had to rerun the investigation.

Mitigation plan

Well since it was user initiated agent was smart enough to give no mitigation plan

Prevention

Since it was a very straightforward demo with user initiated errors so it wasn't enough to generate prevention recommendations

Investigation gaps

One of my favorite feature which even docs does not cover is Investigation gaps.

No investigation can be perfect and that's what investigation gaps fill in to tell when it cannot cover extreme details due to absence of resource at infrastructure level for example in this absence of ssh agent, CloudWatch log groups then it tells those details.

Using Chat

You can ask more detailed questions using chat in natural language

Demo 2: Investigate EKS errors

Adding Capability;Give DevOps Agent access to EKS clusters

Go to Capabilities and click Edit
As we learned earlier Agent space IAM role control the access of AWS resources of Agent, Click view role permissions

Cope the IAM Role ARN. You will need to add an access entry in the EKS cluster with AmazonEKSAdminViewPolicy ( we have tf code for it)

Go to terraform code and replace the arn in terraform.vars
After the terraform code is finished you can see nginx has imagePullBack Error which is intentional

As we learn earlier we are deploying resources by terraform so we need to add tags in the Agent space so that DevOps agent can find the resources.

I am using the same tags in my terraform code as well. Click Save. You will see Agent Space automatically finds the Newly created resources.

Go to Web App

Now we will ask the AWS DevOps Agent what is the cause of this error.

Agent successfully investigated the EKS cluster and find out the root cause

Not only it found out the root cause it also gave the mitigation steps also rollback steps if mitigation cause issues.

DevOps Engineer perspective

We explored how the AWS DevOps Agent reduces MTTR, prevents future incidents through recommendations, and pinpoints root causes.

The Agent’s effectiveness comes from its deep understanding of your infrastructure—both inside AWS accounts and across external systems. It would be interesting to connect MCP server to enhance context even further.

Though Devops Agent is in preview and its free but has some limits

Is it secure? if configured Absolutely yes! because administrators control what DevOps Agent can access in AWS account, Agent IAM permissions control access to its feature and capabilities.

Will it replace you? Absolutely not! because still need an engineer to fix the issues, an engineer to build new features for the infrastructure & need that understanding of infrastructure how to rollback if things goes wrong.

Let me know what do you think?

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

AWS launched EKS Capabilities to simplify Kubernetes with a quick on/off model. It runs Argo CD, ACK, and kro for you—no installs or upgrades. I tested it by deploying an S3 bucket using ACK + Argo CD using EKS Capabilities. Terraform Repo link inside.

Jatin Mehrotra — Wed, 03 Dec 2025 16:40:44 +0000

Jatin Mehrotra for AWS Community Builders

Dec 1 '25

I Created S3 Buckets Using ArgoCD , ACK with EKS Capabilities—No Controllers Installed.

#aws #kubernetes #eks #reinvent2025

Comments

9 min read

I Created S3 Buckets Using ArgoCD , ACK with EKS Capabilities—No Controllers Installed.

Jatin Mehrotra — Mon, 01 Dec 2025 15:08:10 +0000

Amazon Web Services (AWS) EKS team has introduced EKS capabilities on Dec 1, 2025 to help developers for writing their code by offloading Kubernetes management of resources like Argo CD, ACK, kro by a simple turn/off switch.

Amazon EKS Capabilities is a new set of fully managed features designed to make running Kubernetes easier and faster for developers. Think of it as a on-off button on top of your EKS cluster that removes a lot of the heavy lifting for you.
These capabilities give you Kubernetes-native tools for things like:

deploying your apps continuously (Argo CD)

managing AWS resources directly from Kubernetes (ACK)

creating and organizing your Kubernetes objects (kro) And the best part? AWS manages all of this for you.

Instead of installing, updating, and scaling these tools on your worker nodes, EKS now runs them for you. That means less time fighting with cluster operations and more time actually building and scaling the applications.

For the people who wants to jump straight in to trying the solution can follow my repo here

I tested this setup today and here’s how I deployed an S3 bucket with ACK and GitOps using Argo CD and EKS Capabilities.

Bonus
Architecture
Pricing
What is EKS capability
Prerequisites
Terraform Code Walkthrough
DevOps and Platform Engineering perspective

Bonus

I have found a bug which does not allow ArgoCD to get in healthy state, follow along to find out what that bug is and how to fix that bug.

Note: This blog assumes that you have fundamental knowledge on how Argo CD, ACK and kro works conceptually.

Architecture

Available capabilities

ACK: ACK enables the management of AWS resources using Kubernetes APIs, allowing you to create and manage S3 buckets, RDS databases, IAM roles, and other AWS resources using Kubernetes custom resources.
Argo CD: GitOps-based continuous deployment for your applications, using Git repositories as the source of truth for your workloads and system state.
kro: Create custom Kubernetes APIs that compose multiple resources into higher-level abstractions, allowing platform teams to define reusable patterns for common resource combinations-cloud building blocks

Pricing [IMP]

Well the pricing is tricky. You pay for EKS Capabilities based on two components that are both billed hourly:

a base hourly rate for each enabled capability
hourly usage charges based on the quantity of resources managed by each capability.

In short base charge + usage charge.

For Argo CD, you pay hourly for each Argo CD Application managed. For AWS Controllers for Kubernetes (ACK), you pay hourly for each ACK resource managed. For Kubernetes Resource Orchestrator (KRO), you pay hourly for each KRO Resource Graph Definition (RGD) instance managed.

What is EKS Capability?

It is basically an AWS resource managed by AWS including its scaling, lifecycle, security but runs inside your EKS cluster but not on your worker nodes.

Basically Capabilities run in EKS, eliminating the need to install and maintain controllers and other operational components on your worker nodes.

Resources That can be created by Capability

For ArgoCD
- Application
- ApplicationSet
- AppProject
For kro
- ResourceGraphDefinition (RGD)
- Custom resource instances
For ACK
- When you enable the ACK capability, you can create and manage AWS resources using Kubernetes custom resources. ACK provides over 200 CRDs for more than 50 AWS services,

Capability IAM Role [IMP]

Each EKS capability resource has a configured capability IAM role.
The capability role is used to grant AWS service permissions for EKS capabilities to act on your behalf.
For example, to use the EKS Capability for ACK to manage Amazon S3 Buckets, you will grant S3 Bucket administrative permissions to the capability, enabling it to create and manage buckets.

Prerequisites

In order to follow along make sure you have following prerequisites met.

Common to all 3 capabilities

An EKS cluster
An IAM Capability Role with permissions for ACK, ArgoCD, kro
Sufficient IAM permissions to create capability resources on EKS clusters
(For CLI/eksctl) The appropriate CLI tool installed and configured

Argo CD capability specific

AWS Identity Center configured - Required for Argo CD authentication (local users are not supported)
kubectl configured to communicate with your cluster

Terraform Code Walkthrough

Terraform code for this repo lies in this repo.

Just run terraform apply

Demo: Creating S3 buckets with ACK and ArgoCD using EKS Capabilities

The whole process of giving this managed platform capabilities is simple.

Create IAM Capability Role -> Enable Capabilities (ACK and ArgoCD for this Demo) -> Register cluster to ArgoCD -> add ArgoCD applications to track Git repo

We create an capability IAM role with trust policies as mentioned in the docs
We add permissions for the role like if we want to create s3 bucket using ACK we add s3 policy to the role so that ack can talk to AWS Services.

For kro and ArgoCD the role does not need any permissions because when you enable capability adds access entries policies which add permissions for these capabilities to interact with EKS cluster.

The Capability Role

For this we create a trust relationship which allows capabilities to assume role.

{
  "Version": "2012-10-17",               
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "capabilities.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Now based on the capability you add permissions to it, for the brevity of this blog ACK and ArgoCD capability will use the same role and in the tf code I have added full s3 permissions for ACK to create S3 bucket.

ArgoCD IAM Identity Center Integration

The Argo CD managed capability integrates with AWS Identity Center for authentication and uses built-in RBAC roles for authorization

That is why IAM identity Center is a prerequisite.

This is how permissions work with ArgoCD. When a user accesses Argo CD UI:

They authenticate using AWS Identity Center (which can federate to your corporate identity provider)
AWS Identity Center provides user and group information to Argo CD
Argo CD maps users and groups to RBAC roles based on your configuration
Users see only the applications and resources they have permission to access

This is exactly what I have done in my terraform code, accessing IAM identity center instance and user details and passing to the argoCD capability and configuring RBAC role for my user as ADMIN role.

data "aws_ssoadmin_instances" "main" {}

data "aws_identitystore_user" "admin" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.main.identity_store_ids)[0]

  alternate_identifier {
    unique_attribute {
      attribute_path  = "UserName"
      attribute_value = var.idc_username
    }
  }
}

resource "null_resource" "eks_capability_argocd" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-argocd \
        --type ARGOCD \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --configuration '{
          "argoCd": {
            "awsIdc": {
              "idcInstanceArn": "${tolist(data.aws_ssoadmin_instances.main.arns)[0]}",
              "idcRegion": "${local.region}"
            },
            "rbacRoleMappings": [{
              "role": "ADMIN",
              "identities": [{
                "id": "${data.aws_identitystore_user.admin.user_id}",
                "type": "SSO_USER"
              }]
            }]
          }
        }' \
        --profile jj
    EOT
  }
}

To restrict for your use- case see built-in RBAC roles

EKS cluster and installing capabilities]

I am using EKS auto mode for this blog. As this is fresh update I am using aws cli commands to enable capabilities.

Note:

1) Capabilities are not ACTIVE instantly so we need to wait until they become ACTIVE for next operation.

2) Behind the scenes after enabling capabilities, an access entry is created for the capability role with access entries like AmazonEKSArgoCDPolicy,AmazonEKSACKPolicy , AmazonEKSArgoCDClusterPolicy

resource "null_resource" "eks_capability_ack" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-ack \
        --type ACK \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --profile jj
    EOT
  }
}

resource "null_resource" "eks_capability_argocd" {
  depends_on = [module.eks]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks create-capability \
        --region ${local.region} \
        --cluster-name ${local.cluster_name} \
        --capability-name my-argocd \
        --type ARGOCD \
        --role-arn ${aws_iam_role.eks_capability_role.arn} \
        --delete-propagation-policy RETAIN \
        --configuration '{
          "argoCd": {
            "awsIdc": {
              "idcInstanceArn": "${tolist(data.aws_ssoadmin_instances.main.arns)[0]}",
              "idcRegion": "${local.region}"
            },
            "rbacRoleMappings": [{
              "role": "ADMIN",
              "identities": [{
                "id": "${data.aws_identitystore_user.admin.user_id}",
                "type": "SSO_USER"
              }]
            }]
          }
        }' \
        --profile jj
    EOT
  }
}

Registering the eks cluster to ArgoCD to deploy applications

Docs suggest to use argocd cli but it never succeeded as it always timed out so I register the secret using K8 Secret. This is possible because of the new access entry(AmazonEKSArgoCDPolicy) policy which ArgoCD capability to read the K8 secrets

resource "null_resource" "argocd_add_cluster" {
  depends_on = [null_resource.wait_for_argocd]

  provisioner "local-exec" {
    command = <<-EOT
      aws eks update-kubeconfig \
        --name ${local.cluster_name} \
        --region ${local.region} \
        --profile jj

      kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${local.cluster_name}-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  name: ${local.cluster_name}
  server: ${module.eks.cluster_arn}
  project: default
EOF
    EOT
  }
}

Create an ArgoCD Application

This demo uses a public GitHub repository, so no repository configuration is required. For private repositories, configure access using AWS Secrets Manager, CodeConnections, or Kubernetes Secrets

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: eks-capability
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/jatinmehrotra/aws-reinvent-2025
    targetRevision: HEAD
    path: eks-capabilities/ack_yaml
  destination:
    name: reinvent-2025
    namespace: ack
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Over this path there is yaml file to create S3 bucket using ACK which will be synced and applied to cluster and then ACK will create this bucket in our AWS Account

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: my-test-bucket
  namespace: default
spec:
  name: jj-bucket-name-12345

Fixing the Bug

Even after doing all this or following AWS docs this won't run because the access entries(AmazonEKSArgoCDClusterPolicy) policies created by AWS for ArgoCD Capability lacks permissions to list the cluster resources.

SO added cluster admin permission for the blog

horizontalpodautoscalers.autoscaling is forbidden: 
User "arn:aws:sts::xxxxxxx:assumed-role/eks-capability-role/aws-go-sdk-1764587162523491382" 
cannot list resource "horizontalpodautoscalers" in API group "autoscaling" at the cluster scope

So to fix that we need to import the access entry created by AWS and add the Cluster Admin Access entry.

# Import the access entry created by EKS capability
resource "null_resource" "import_access_entry" {
  depends_on = [null_resource.wait_for_argocd]

  provisioner "local-exec" {
    command = <<-EOT
      terraform import -input=false \
        aws_eks_access_entry.capability_role \
        "${local.cluster_name}:${aws_iam_role.eks_capability_role.arn}" || true
    EOT
  }
}

resource "aws_eks_access_entry" "capability_role" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.eks_capability_role.arn
  type          = "STANDARD"

  lifecycle {
    ignore_changes = [kubernetes_groups]
  }

  depends_on = [null_resource.import_access_entry]
}

# Add ClusterAdmin policy to fix insufficient ArgoCD permissions
resource "aws_eks_access_policy_association" "capability_role_admin" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.eks_capability_role.arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.capability_role]
}

You can access ArgoCD UI from console -> Capabilities and Under Argo CD

You can login using your IAM identity Center

Argo CD successfully sync the application for Git repository and then ACK creates the S3 bucket

Cleanup

For cleanup before running terraform destroy. Read the delete docs

Delete the s3 bucket first
run the cleanup script

Then run terraform destroy.Because cluster won't be deleted if capabilities are not deleted.

DevOps and Platform Engineering perspective

In this blog we saw the Use case of GitOps for Applications and Infrastructure where Use Argo CD to deploy applications and ACK to provision infrastructure, both from Git repositories. Your entire stack—applications, databases, storage, and networking—is defined as code and automatically deployed

We just push the AWS infra yaml files to Git and ArgoCD deploys the updated application, and ACK provisions a new S3 bucket with the correct configuration.
All changes are auditable, reversible, and consistent across environments.

There can be many use cases for these capabilities like Account and regional Bootstrapping, Modernization of EKS resources

Its very interesting to see how they are compared to self managed versions ack,argocd,kro. Ofcourse there are limitations like ArgoCD right now supper single namespace deployment or Notifications Controller in ArgoCD isn't supported

But overall a great update for Platform engineering which offloads the setup, complexity and management of Kubernetes resources to AWS and lets developers focus on their productivity and applications.

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

Cut CI/CD Costs by 77% & 2x Deployment Speed with GitHub Actions on EKS Auto Mode

Jatin Mehrotra — Mon, 21 Apr 2025 00:18:08 +0000

On April 5, 2025 I did a live stream on how to run Github Actions Self Hosted Runners on EKS Auto with AWS Heroes Arshad Zackeriya and Jones Zachariah Noel.

Disclaimer: 🐶 No Beagles were harmed. Slightly annoyed, maybe — but unharmed.

The results {Performance, Speed, Cost} were not only astonishing but impeccable and promising enough to adopt this solution at the enterprise level. This solution isn't just AWS agnostic, with the knowledge gained in this blog can be extended to Azure(AKS), google(GKE) or if you are running K8 on your own bare metal servers.

For the people who wants to jump straight in to trying the solution can follow my repo here

This is going to be a bit of a long one, so grab a coffee, get comfy, and make sure you're sitting in your optimal developer position™ — you know the one.

Architecture
Why this solution
Self Hosted Runner Concept on K8
Terraform Code Walkthrough
How to test this solution
Github Large Hosted Runner vs Running Runners on EKS Auto
From a solutions architect perspective

Architecture

Why this solution

I work as a senior DevOps Engineer at Colorkrew where we have a lot of products and to support the development workflow we have lot of GitHub repositories.

As we started to increase our product portfolio, our CI/CD pipelines also started to become more complex, concurrent and frequent leading to the need of more computation power and eventually more robust infrastructure layer which supports our growing needs.

Running Github Actions on default free machines (called as runners) started to become slow and the initial solution was to either use Large hosted runners by GitHub which are paid or run the runners on our Infrastructure like Kubernetes.

So I started to compare both solutions on performance, speed and cost and this lead to the inception of this running GitHub's self hosted runners on EKS Auto.

Self Hosted Runner Concept on K8

Runners: The machines (servers or virtual environments) that actually execute the jobs defined in your workflows. When you manage called as Self Hosted runners otherwise GitHub Hosted runners. Runners are ephemeral in nature.

Runner Scale Sets: Think of it as a logical grouping of runners that are homogeneous in nature which means all runners under a particular group will have same configuration. Can be installed at repository, organization or enterprise level.

If you want heterogeneous setup which means different configuration for runners for different ci/cd jobs then you need a multiple runner scale sets.

Another important thing to know about Runner scale sets are that you configure how many minimum and maximum runners you want all the time.

Name of Runner Scale Sets: Runner scale sets are addressable by their name so remember it because when you need to specify that name of runner scale set in the runs on: property of GHA to assign that workflow on on a particular runner scale set.

Endpoints: ARC talks to two endpoints api.github.com and pipelines.actions.githubusercontent.com. Make sure your organization's firewall, proxies, nat gateway whatever being used to access internet should be configured to allow the above endpoints for ARC controller.
ARC controller: contains of 2 elements/pods.
- controller-manager: First pod that comes online. This has different controllers managing different resources in the cluster. Important one to understand is AutoScalingListener Controller manages the listener pod.
  - Responsible for creating the resources and making sure that match the desired count and state.
- Runner ScaleSet Listener: Manages the decision making about scaling needs. Responsible to decide how many runners to create. Each listener has its own pod which means 1 listener pod per runner scale set. If you have 2 runner scale set then 2 listener pods either on same namespace like controller manager or different namespace ( configurable)

I thought should give easier explanation in my own analogies 😋🫣

Actions Runner Controller (ARC is like the manager of a smart, automated coffee shop — it watches how many customers are coming in (workflows) and instantly hires or releases baristas (runners) as needed.
Instead of having baristas standing around all day just in case, ARC spins up temporary baristas (containers in Kubernetes) only when customers arrive, and lets them go when the work is done. This keeps the system fast, efficient, and cost-effective.

With Runner Scale Sets, you can define the rules for how many baristas you want at any given time, based on how busy your shop is — and ARC handles the rest.

You can read more about detailed end to end workflow over here

Terraform Code Walkthrough

jatin-mehrotra-personal / eks-auto-self-hosted-runners

Run GitHub Actions Self Hosted Runners on EKS Auto Mode

This repository provides infrastructure as code (IaC) to deploy auto-scaling GitHub Actions self-hosted runners on Amazon EKS using GitHub's Actions Runner Controller (ARC).

detailed blog: https://dev.to/aws-builders/cut-cicd-costs-by-77-2x-deployment-speed-with-github-actions-on-eks-auto-2ob2

See it in action: https://www.youtube.com/live/s23QvNz2WuY?si=08n0qCVpMYC1qTKc

Overview

This solution allows you to:

Deploy a fully managed EKS cluster with auto-scaling capabilities
Set up GitHub Actions Runner Controller (ARC) for managing self-hosted runners
Configure auto-scaling runner sets that scale based on workflow demand
Support Docker-in-Docker (DinD) runners for container-based workflows

Architecture

The infrastructure consists of:

Amazon EKS cluster running in a custom VPC
GitHub Actions Runner Controller deployed via Helm
Auto-scaling runner sets configured to scale from 0 to meet demand
Optional Docker-in-Docker (DinD) runner support
Karpenter for node auto-scaling (configured but optional)

Prerequisites

AWS CLI configured with appropriate permissions
Terraform v1.0.0+
kubectl
Helm v3+
A GitHub repository or organization where you want to deploy runners
…

View on GitHub

Setting Up Auto-Scaling GitHub Actions Self-Hosted Runners on Amazon EKS

Introduction

In this walkthrough, we'll set up GitHub Actions Runner Controller (ARC) on Amazon EKS to automatically scale self-hosted runners based on workflow demand.

Project Structure

Here's the structure of our implementation:

eks-auto-self-hosted-runners/
├── README.md
├── architecture/
├── commit_log.txt
├── scripts/
└── terraform/
    ├── base/
    └── modules/
        ├── arc/
        ├── eks/
        ├── karpenter_config/
        └── vpc/

1. Root directory - Contains the main README and architecture diagrams
2. scripts/ - Contains utility scripts for cleanup and performance testing
3. terraform/ - The main infrastructure code
   base/ - The entry point for Terraform deployment
   modules/ - Reusable Terraform modules:
   arc/ - Actions Runner Controller configuration
   eks/ - EKS cluster configuration
   karpenter_config/ - Node auto-scaling configuration
   vpc/ - Network infrastructure configuration

Architecture Overview

Our solution uses the following components:

Amazon EKS Auto: Managed Kubernetes service to host our runner infrastructure with Karpenter for provisioning nodes on-demand for runners compute
GitHub Actions Runner Controller (ARC): Kubernetes controller that manages self-hosted runners
Terraform: Infrastructure as Code tool to deploy and manage all components

The architecture allows GitHub Actions workflows to dynamically request runners, which are provisioned on-demand in our EKS cluster and automatically scaled down when not needed.

Step 1: Setting Up the Infrastructure

VPC Configuration

We start by creating a VPC with public and private subnets. Our VPC configuration uses a CIDR block of 10.0.0.0/16 with subnets spread across two availability zones. The
private subnets host our EKS nodes, while public subnets are used for NAT gateways and load balancers.

The configuration in terraform/base/vpc.tf references the VPC module and sets up all necessary networking components with appropriate tagging for Kubernetes integration.

EKS Cluster Setup

Next, we create an EKS cluster using the EKS module defined in terraform/modules/eks. Our cluster runs Kubernetes version 1.31 and includes a system node group for running
essential cluster services.

The terraform/base/eks.tf file configures the cluster with public endpoint access and places the worker nodes in the private subnets for enhanced security.

Step 2: Configuring EKS Auto's pre-installed Karpenter for Auto-Scaling

EKS Auto comes with Karpenter pre-installed. We leverage Karpenter to provision and auto scale Ec2 spot instances from our desired Ec2 type, capacity and configuration for our GHA runner's compute.

Our Karpenter configuration in terraform/base/karpenter_config.tf uses m7a instance types with spot pricing for cost efficiency. The consolidation policy is set to "WhenEmpty" with a 5-minute timeout, which means nodes will be removed when they're no longer needed.

Key configuration parameters include:
• Instance types: m7a family with 8 CPUs
• Capacity type: Spot instances for cost savings
• Storage: 300GB with 5000 IOPS
• Availability zones: us-east-1a and us-east-1b

Step 3: Deploying Actions Runner Controller (ARC)

Now we deploy the GitHub Actions Runner Controller using the ARC module defined in terraform/modules/arc. The module is referenced in terraform/base/arc.tf with configuration parameters from locals.tf.

The ARC deployment consists of two main components:

Controller: Manages the lifecycle of runner pods
Runner Sets: Define the configuration for the runners

Controller Deployment

The controller is deployed using a Helm chart from the official GitHub Actions Runner Controller repository. The configuration is defined in terraform/modules/arc/controller.tf and uses values from helm/controller_values.yaml.

Runner Sets Configuration

We deploy two types of runner sets:

Standard Runners: For general workflow jobs
Docker-in-Docker (DinD) Runners: For jobs that need to build Docker images
- There is also another type of runner set called Kubernetes mode which should be used when organizations cannot afford to run docker with superuser context as in case DinD runners then use Kubernetes mode ( not covered in this blog)

The runner sets are defined in terraform/modules/arc/runner_sets.tf and use values from helm/arc_listener_values.yaml and helm/arc_listener_values_dind.yaml respectively.

Note: This is very important that Controller pod and runner scale set pod should be configured to be deployed on on-demand ec2 instance for reliability and ephermal runner pods should be configured to run spot instances.

# controller config for on demand node

nodeSelector:
  karpenter.sh/nodepool: system

tolerations:
- key: "CriticalAddonsOnly"
  operator: "Exists"

# listener config for on demand node

listenerTemplate:
  spec:
    nodeSelector:
      karpenter.sh/nodepool: system
    tolerations:
    - key: "CriticalAddonsOnly"
      operator: "Exists"

Step 4: GitHub Authentication Setup

ARC authenticates with GitHub using a GitHub App. The credentials are stored in Kubernetes secrets and used by the runner controller to authenticate with GitHub.

To set up authentication:

Create a GitHub App with the necessary permissions
Generate a private key for the app
Store the app ID, installation ID, and private key in the terraform/modules/arc/secrets (Don't forget to create it, gitignored) the directory
The terraform/modules/arc/secrets.tf file creates a Kubernetes secret with these credentials

Step 5: Namespace Management

We create dedicated namespaces for ARC components:

arc-systems: For the controller components
arc-runners: For the runner pods

These namespaces are defined in terraform/modules/arc/namespaces.tf.

Step 6: Cleanup Handling

To ensure proper cleanup of resources, we've included a cleanup script in scripts/cleanup-finalizers.sh that removes finalizers from Kubernetes resources. This script is called during the Terraform destroy process to ensure resources are properly cleaned up.

The script handles various resource types including:
• AutoscalingRunnerSet
• EphemeralRunnerSet
• AutoscalingListener
• ServiceAccounts
• RoleBindings
• Roles

How It Works

When a GitHub Actions workflow runs, it requests a runner with specific labels
The ARC controller detects this request and creates a runner pod in the EKS cluster
If needed, Karpenter provisions a new EC2 instance to host the runner pod
The runner registers with GitHub and runs the workflow job
After the job completes, the runner pod is terminated
When no more runners are needed, Karpenter consolidates and removes unused nodes

Benefits of This Approach

Cost Efficiency: Runners are only provisioned when needed and automatically scaled down when idle
Flexibility: Custom runner environments can be defined to meet specific workflow requirements
Scalability: The system can handle large numbers of concurrent workflows
Security: Runners run in isolated Kubernetes pods with defined security contexts
Reliability: Failed runners are automatically replaced, ensuring workflow reliability

How to test this solution

I have created 3 GHA of different types to test different scenarios.

Simple test

The first type is the simple GHA but important point to pay attention is runs-on parameter which uses arc-runner-set label.

The moment you run the first workflow it takes around 1 minute because Karpenter is provisioning an Ec2 instance.

Once it is provisioned and if you run the workflow again because Ec2 instance is already present the action completes its execution immediately.

Karpenter will delete the node if it's unused for at least 5 minutes.

Now if you have latency sensitive requirement of running your actions immediately then set minRunners>0 for the runner scale set configuration

Concurrent job test

In this GHA I am running 5 concurrent jobs running for exactly 1 minute each triggered either through push or manually.

DinD job test

There are times when we want to run microservice in container like redis and do e2e testing.

That's where we need Docker Inside Docker kind of GHA.

Important point to pay attention here is runs-on parameter which uses different runner set configured especially for dind workflows

In this workflow busy box container runs the main job which can talk to redis service container.

Github Large Hosted Runner vs Running Runners on EKS Auto

Performance & Speed

I tried to do the performance and speed comparison of 8 Core CPU vs our solution.

I tried to run the sleep-matrix GHA using auto-commit script for 10 minutes and every commit happening in every 45 seconds effectively simulating a concurrent job situations for both EKS Auto solution and Github large Runners and results were hands down in favor of our solution

Our solutions has stable performance with constant 1 minute 8 seconds Execution time throughout.

Price

Update [21st April 2025] : I did a mistake in EKS Auto node management price earlier but after correction our solution makes it even cheaper

This is little tricky and may not be 100% accurate comparison but still can be considered 70-80% accurate.

Assume over a 30‑day month with 1 hour of concurrent CI workloads per day (12 jobs running in parallel). Lets calculate the cost on our solution and as well on Github's self hosted runners

For our solution

For Github's large runner

There is also a cost Github Teams/Enterprise subscription. I am assuming it's on cheaper side i.e Team's plan for a team of 10 developer

self‑hosting on EKS with m7a.2xlarge Spot nodes costs approximately $170.38/month, while same GitHub’s 8‑core large hosted runners cost $731.20/month, so my solution on EKS Auto is YIELDING A WHOOPING SAVINGS OF 76.7%.

Note: Even if we add Github Team's for EKS solution i.e $40 more which is 170+40 =$210/month which is still a whooping savings of 72%

Lets assume it's not 1 hour, it's 2 hour of concurrent CI workloads per day (12 jobs running in parallel)

yielding a savings of 84%

Note: I know I haven't added the networking components cost like nat gateway even if we add up it will still be cheaper GitHub's large runners :D

From a solutions architect perspective

By combining Amazon EKS Auto and GitHub Actions Runner Controller, we've created a solution that aligns with the AWS Well-Architected Framework:

Operational Excellence:
• Infrastructure as code through Terraform ensures consistent deployments
• Auto-scaling runners eliminate manual capacity management
• Clean separation of components improves maintainability

Security:
• GitHub App authentication with scoped access
• EKS security groups and IAM roles enforce least-privilege
• Private subnets reduce attack surface

Reliability:
• Multi-AZ deployment ensures high availability
• Auto-scaling from zero accommodates varying workloads without manual intervention
• EKS Auto's Karpenter provides intelligent node provisioning, ensuring resources are available when needed

Performance Efficiency:
• On-demand scaling ensures resources are used only when needed
• Spot instances reduce costs while maintaining performance
• Customizable instance types allow optimization for specific workload requirements
• Docker-in-Docker support enables container-based workflows with minimal overhead

Cost Optimization:
• Scale-to-zero capability eliminates costs when no workflows are running
• Spot instances provide significant cost savings (up to 90%) compared to on-demand instances
• Resource consolidation through Karpenter's WhenEmpty policy reduces idle resources

Sustainability:
• Efficient resource utilization through auto-scaling minimizes environmental impact
• Scale-to-zero capability reduces energy consumption during idle periods
• Reduced idle resources lower energy consumption

This solution provides organizations with a scalable, cost-effective platform for running GitHub Actions workflows while maintaining full control over their infrastructure. The architecture can easily scale from small development teams to enterprise-level deployments, adapting to changing workflow requirements without compromising on security or performance.

By leveraging AWS managed services like EKS and implementing infrastructure as code through Terraform, this solution reduces operational overhead while providing the flexibility needed
for modern CI/CD pipelines. The result is a robust, efficient platform that enables teams to focus on delivering value rather than managing infrastructure.

Follow The Zacs' Show Talking AWS on YouTube or reach out to them if you want to share your knowledge on AWS.

Have some questions on this solution reach out to me over Linkedin, X.

AWS CodePipeline + EKS Action: The Easiest Way to Deploy Containers Yet!

Jatin Mehrotra — Tue, 25 Feb 2025 06:26:06 +0000

Amazon Web Services (AWS) CodePipeline Team has simplified Developer, DevOps engineers operational overhead and streamlined the Deployment process to EKS by introducing an CodePipeline action to deploy directly to your EKS cluster.

Why does it matter to you?

Previously if I had to deploy resources to EKS using DevOps approach, I had to manage codebuild project, permissions to access eks, kubectl, helm commands, other horrific shell commands and still it wont be perfect in one shot.

I tried this out today so let me show you how you can simplify your deployment pipeline for EKS by 100%, say goodbye to codebuild and remove all the complex process, scripts and command!!!

Architecture

Prerequisites

EKS cluster with public endpoint.
Kubernetes resource like deployment.yaml file
You can also helm chart if you wish.

Note: If you want to try this update with EKS private Endpoint then checkout this blog by AWS Hero Arshad Zackeriya.

Whichever you prefer I have provided both (helm chart and deployment.yaml) in this repository.

Behind the Scenes

In case of kubernetes manifest file

This action logs into eks cluster, set kubeconfig context
Installs kubectl
Applies Kubernetes manifests
Rollout resources

In case of helm chart

This action logs into eks cluster, set kubeconfig context
Instals helm
Instal Helm charts

Create an EKS cluster

Your cluster can be public or private (including those in private VPCs).The pipeline will automatically establish a connection into your private network to deploy your container application, without additional infrastructure needed.

I have used eks auto using terraform eks module which is quick and easy with EKS endpoint as public.

eks.tf

# eks cluster

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name                   = local.cluster_name
  cluster_version                = "1.32"
  cluster_endpoint_public_access = true


  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }

  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true


  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}

vpc.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.19.0"

  name = "codepipeline-eks-action"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  one_nat_gateway_per_az = false

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }
  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = {
    Environment = "terraform-playground"
  }
}

pod_identity.tf

data "aws_iam_policy_document" "allow_pod_identity" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
  }
}

resource "aws_iam_role" "read_ecr" {
  name               = "read-ecr-role"
  assume_role_policy = data.aws_iam_policy_document.allow_pod_identity.json
}

resource "aws_iam_role_policy_attachment" "read_ecr" {
  role       = aws_iam_role.read_ecr.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_eks_pod_identity_association" "read_ecr" {
  cluster_name    = local.cluster_name
  namespace       = "default"
  service_account = "ecr-sa"
  role_arn        = aws_iam_role.read_ecr.arn
}

Note: I am using pod identity with ecr repository permission to obtain ecr image for deployment.yaml.

If you want complete code for cluster, you can follow this repo

Create pipeline with EKS deployment action

Case 1: when kubectl configuration is used.

I am using the source as GitHub using code start connection and code resides in this repo

Select the cluster
Provide the path for your deployment.yaml file (in my case its deployment.yaml file)

Case 2: When Helm chart is used

Enter the release name
Enter the helm chart (in my case its the test-chart)

Important step

Note: Once pipeline is created you need to edit the pipeline service role or update the existing one and add the following permissions to avoid error.

{
    "Statement": [
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Condition": {
                "StringEqualsIfExists": {
                    "iam:PassedToService": [
                        "cloudformation.amazonaws.com",
                        "elasticbeanstalk.amazonaws.com",
                        "ec2.amazonaws.com",
                        "ecs-tasks.amazonaws.com"
                    ]
                }
            }
        },
        {
            "Action": [
                "codecommit:CancelUploadArchive",
                "codecommit:GetBranch",
                "codecommit:GetCommit",
                "codecommit:GetRepository",
                "codecommit:GetUploadArchiveStatus",
                "codecommit:UploadArchive"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codedeploy:CreateDeployment",
                "codedeploy:GetApplication",
                "codedeploy:GetApplicationRevision",
                "codedeploy:GetDeployment",
                "codedeploy:GetDeploymentConfig",
                "codedeploy:RegisterApplicationRevision"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codestar-connections:UseConnection"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticbeanstalk:*",
                "ec2:*",
                "elasticloadbalancing:*",
                "autoscaling:*",
                "cloudwatch:*",
                "s3:*",
                "sns:*",
                "cloudformation:*",
                "rds:*",
                "sqs:*",
                "ecs:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "lambda:InvokeFunction",
                "lambda:ListFunctions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "opsworks:CreateDeployment",
                "opsworks:DescribeApps",
                "opsworks:DescribeCommands",
                "opsworks:DescribeDeployments",
                "opsworks:DescribeInstances",
                "opsworks:DescribeStacks",
                "opsworks:UpdateApp",
                "opsworks:UpdateStack"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStacks",
                "cloudformation:UpdateStack",
                "cloudformation:CreateChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:SetStackPolicy",
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild",
                "codebuild:BatchGetBuildBatches",
                "codebuild:StartBuildBatch"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Effect": "Allow",
            "Action": [
                "devicefarm:ListProjects",
                "devicefarm:ListDevicePools",
                "devicefarm:GetRun",
                "devicefarm:GetUpload",
                "devicefarm:CreateUpload",
                "devicefarm:ScheduleRun"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "servicecatalog:ListProvisioningArtifacts",
                "servicecatalog:CreateProvisioningArtifact",
                "servicecatalog:DescribeProvisioningArtifact",
                "servicecatalog:DeleteProvisioningArtifact",
                "servicecatalog:UpdateProduct"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:ValidateTemplate"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:DescribeImages"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "states:DescribeExecution",
                "states:DescribeStateMachine",
                "states:StartExecution"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "appconfig:StartDeployment",
                "appconfig:StopDeployment",
                "appconfig:GetDeployment"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:xxxx:log-group:/aws/codepipeline/eks-deploy-codepipeline",
                "arn:aws:logs:us-east-1:xxxxx:log-group:/aws/codepipeline/eks-deploy-codepipeline:log-stream:*"
            ]
        },
        {
            "Sid": "EksClusterPolicy",
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "EksVpcClusterPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeVpcs"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:292170836962:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:292170836962:subnet/subnet-example2"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterfacePermission",
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example2"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DeleteNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-east-1:xxxxx:subnet/subnet-example1",
                        "arn:aws:ec2:us-east-1:xxxx:subnet/subnet-example2"
                    ]
                }
            }
        }
    ],
    "Version": "2012-10-17"
}

Create a access entry in eks cluster with AmazonEKSClusterAdminPolicy for the above codepipeline service role.

Run the pipeline

Case when kubectl manifest is updated

(base) jatin.mehrotra@CK0662-001 codepipeline-eks-deployment-no-github % kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
hello-k8s-74fd98b69b-2pwrv   1/1     Running   0          2m37s
hello-k8s-74fd98b69b-g876f   1/1     Running   0          2m37s
hello-k8s-74fd98b69b-phs77   1/1     Running   0          2m37s

Case when helm chart manifest is updated

(base) jatin.mehrotra@CK0662-001 codepipeline-eks-deployment-no-github % kubectl get pods                             
NAME                    READY   STATUS    RESTARTS   AGE
test-67cbfddc66-2gcvj   1/1     Running   0          10m

From Developer, DevOps perspective

With this update I don't have to manage codebuild projects or any kind of compute environment and manage complex scripts, permissions and tool installations.
Following is the image of the setup which I used to have with codebuild before this update. Such a complex mess.

This is surely a game changer for developers, devops engineers and infra engineers who wants to focus on business problem, their applications and other kubernetes issue like monitoring and scaling.

This is not GitOps approach which is followed by Flux/ArgoCD but the best DevOps approach for EKS cluster in my view.

Will you use this action for your eks clusters? Let me know in the comments!!!

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

CodePipeline EC2 Action: Simplify your EC2 application deployment pipeline by 90%!

Jatin Mehrotra — Mon, 24 Feb 2025 03:48:57 +0000

Amazon Web Services (AWS) CodePipeline Team has simplified Developer, DevOps engineers operational overhead and streamlined the Deployment process to EC2 by introducing an CodePipeline action to deploy directly to your EC2 instances.

Why does it matter to you?

Previously, if you wanted to deploy to EC2 instances, you had to use CodeDeploy with an AppSpec file to configure the deployment. After this update NO NEED TO MANAGE CODEDEPLOY RESOURCE AND APPSPEC FILE.

I tried this out today so let me show you how you can simplify your deployment pipeline by 90% and remove all the complex process and scripts!!!

Architecture

Prerequisites

supports only Linux instance types
maximum fleet size supported is 500 instances.
Pipeline V2 only supported
SSM Agent must be installed

Behind the Scenes

This action performs a send-command using SSM to execute the script in the instance.

Create an EC2 instance with Apache

Note: If you are creating an instance then you need to create a role with AmazonSSMManagedInstanceCore, AmazonS3ReadOnlyAccess and attach to ec2 instance.

Add ssm agent to the instance manually either using user data or logging into existing instance and then running the commands.
If you are using existing instance then you need to reboot the instance after adding the ssm agent.

I have created EC2 with Apache web server for the brevity of the blog.

sudo su
yum update -y
yum install httpd -y
service httpd start
chkconfig httpd on
echo "codepipeline ec2 deployment action" > /var/www/html/index.html

Create pipeline with Ec2 deployment action

Here we need to select instance using tags; I have used name tag to identify my instance.
Then you need to provide the target directory in ec2 instance where you want to deploy.
Finally path to the executable script file that runs AFTER the Deploy phase.

Note: Once pipeline is created you need to edit the pipeline service role and add the following permissions to avoid error.

        {
            "Effect": "Allow",
            "Action": [
                "ssm:CancelCommand",
                "ssm:DescribeInstanceInformation",
                "ssm:ListCommandInvocations",
                "ssm:SendCommand"
            ],
            "Resource": "*"
        }

Run the pipeline

I am using the source as GitHub using code start connection and code resides in this repo.

Note: The aws guide mentioned about adding AWSSystemsManagerDefaultEC2InstanceManagementRoleeployAction to the ec2 instance role but in my environment I didn't needed that permission

Some advanced options for this action

You can specify number of instances in number or percentages to deploy to in parallel.
You can specify number of instances in number or percentages to stop the task after the task fails.
You can specify load balancer which will block the traffic Toi the instance when deployment is taking for that instance.

From Developer, DevOps perspective

With this update I can have an END-to-END AWS native Continuous deployment on EC2 without managing CodeDeploy resources.
This is surely a game changer for developers and devops engineers who wants to focus on business problem, their applications and not on their complex deployment process.
Of course this experience of simplifying can be augmented even further to 100% if AWS can add necessary permissions for the role.

After reading this, do you wish to migrate to this action and simplifying your deployment process? Let me know in the comments!!!

I share such amazing AWS updates on DevOps, Kubernetes and GenAI daily over Linkedin, X. Follow me over there so that I can make your life more easy.

EKS Auto Mode Unlocked for Existing Clusters with Terraform

Jatin Mehrotra — Fri, 13 Dec 2024 05:59:47 +0000

In the previous blog, I explained that EKS Auto mode is now supported by terraform-eks-module and illustrated how we can create new cluster with EKS Auto Mode.

In this blog, we’ll learn how to enable EKS Auto Mode on existing clusters and migrate workloads from EKS Managed Node Groups to EKS Auto nodes with ZERO DOWNTIME and continued application availability using my terraform code.

I have also added a BONUS section which explains how we can control our pod's deployments on EKS Auto Mode nodes or other compute types.

Motivation

Terraform-provider-aws released a new version v5.8.1 which allows to enabled EKS Auto with built-in NodePools on existing cluster.

Terraform-aws-eks release a new version v20.31.1 which allows to use custom NodeClass/NodePools when EKS Auto is enabled without built-in NodePools.

I want this blog to be really short, crisp and efficient so lets jump into actual steps!

Deploy Terraform cluster without EKS Auto Mode

We want to create the use case where we have an existing cluster WITHOUT EKS Auto Mode using EKS MNG.
Use this repository code to deploy EKS cluster with Managed node group.

Note: I am attaching policies to the node IAM role for EKS MNG - this is too permissive, better to use EKS Pod Identity (or IRSA, but EKS Pod Identity is preferred). Feel free to send a PR to the repo :)

Deploy workload or pods

We will automate this as well using terraform's kubectl_manifest resource, we will deploy workload yaml code using terraform

Note: During cluster creation, test workload(pods) were not deployed because kubectl context was not set locally. So run the following command to set the kubectl context and run terraform apply again once cluster is created.

aws eks --region us-east-1 update-kubeconfig --name eks-existing-cluster-tf-test --profile <your-profile-name> ; terraform apply

Current state of EKS cluster before EKS Auto Mode

Let's verify the current state of eks cluster when EKS Auto mode is not enabled.
EKS Auto mode is disabled.

EKS Auto Managed Node group created by me is running.

Pods are running on EKS managed node group

Enable EKS Auto Mode on Existing Cluster

Uncomment the following code to the eks.tf and terraform apply to enable EKS Auto Mode

bootstrap_self_managed_addons = true

cluster_compute_config = {
   enabled = true
}

bootstrap_self_managed_addons = true is very important otherwise you will face error where terraform tries to recreate the cluster again. I literally cried over this

Current state of EKS cluster after EKS Auto Mode

As expected built-in NodePools are empty

Migrate workload(pods) from EKS MNG to EKS Auto Node

There are couple of ways to smoothly migrate existing workloads from MNG to EKS Auto with minimal disruption while maintaining application’s availability throughout the migration.

Note: Copy the EKS MNG node group name.

Using eksctl tool

The following command will cordon all nodes and all pods are evicted from a nodegroup and EKS will provision pods to node managed by EKS Auto.

eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName>  --region us-east-1 --profile=<profile>

eksctl command evicts pod one at a time which I have tested so application availability is maintained.
But if you still want to be 100% sure, you can use the best practice of using pod Disruption budget. We will automate this using terraform so run terraform apply

resource "kubectl_manifest" "test_pdb" {
  yaml_body = <<YAML
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: test-pdb
  labels:
    environment: test
spec:
  minAvailable: 1
  selector:
    matchLabels:
      environment: test
YAML
}

After migrating; If we want to allow scheduling pods to EKS MNG we need to uncordon the EKS MNG or you can delete the Node group

eksctl drain nodegroup --cluster=<clusterName> --name=<copiedNodegroupName>  --region us-east-1 --profile=<profile> --undo

Using kubectl

we can use the following command to drain the nodes using kubectl

kubectl drain --ignore-daemonsets <node name>

Once it returns (without giving an error), you can delete the node or you want to tell Kubernetes that it can resume scheduling new pods onto the node

kubectl uncordon <node name>

[ BONUS ] How to schedule Pods always on EKS Auto Nodes?

There are 2 options to achieve this :

Either delete The NodeGroup and let EKS Auto handle the scheduling on EKS Auto Nodes
Using labels and NodeAffinity

Control if a workload is deployed on EKS Auto Mode nodes

There is concept called mix-mode cluster where you’re running both EKS Auto Mode and other compute types, such as self-managed Karpenter provisioners or EKS Managed Node Groups.
In mix mode clusters by default deployment is deployed to EKS MNG nodes and not EKS Auto Nodes
In such case we can use labels and nodeAffinity.

Using NodeSelector label

Use the label eks.amazonaws.com/compute-type: auto when you want a workload is deployed to EKS Auto Node.
This nodeSelector value is only relevant if you are running a cluster in a mixed mode, node types not managed by EKS Auto Mode

apiVersion: apps/v1
kind: Deployment
spec:
      nodeSelector:
        eks.amazonaws.com/compute-type: auto

I have an added the above configuration in sample_app_on_eks_auto_nodes.tf file. We are automating using Terraform so uncomment and run `terraform apply.

Using nodeAffinity

You can add this nodeAffinity to Deployments or other workloads to require Kubernetes to not schedule them onto EKS Auto Mode nodes

I have added the workload with nodeAffinity in sample_app_not_on_eks_auto_nodes.tf . We are automating using Terraform so uncomment and run terraform apply.

From DevOps, IaC Perspective

We saw how we can enable EKS Auto mode for Existing clusters with built-in NodePools using terraform-eks-module
We also saw how we can migrate our existing workload from EKS Managed Group to EKS Auto Nodes without any down time as EKS Auto node respect PodDisruptionBudget.
We also saw how we can use nodeSelector Labels and nodeAffinity to control deployment of workload in case of mixed-mode EKS clusters.

Currently EKS Auto deploys EC2 of instance type c6a.large which can be also customized using nodeClass and NodePool which we will see in the next blog. Follow me on Linkedin or on dev.to so that you get timely updates of what I share.

Feel free to reach out to me on Linkedin, X if you face any error migrating your Existing workloads to EKS Auto Mode Nodes using terraform.

EKS Auto Mode Arrives in Terraform – Simplify Kubernetes Today

Jatin Mehrotra — Thu, 05 Dec 2024 10:14:13 +0000

During AWS re:Invent 2024 AWS released a new feature to EKS i.e EKS Auto Mode, which i have already covered in detail in my previous blog

In this blog we will see how we can create cluster with EKS Auto mode using the OG "terraform-eks-module" and how it simplified my eks.tf code.

I will also talk about the differences in the terraform code which we used for eks cluster before Auto mode feature v/s the terraform code after using using Auto mode feature and how it save a beginner who does not know anything EKS.

Motivation

Terraform provider for AWS released a new version v5.79.0 which adds the resources (compute_config, storage_config, storage_config and kubernetes_network_config.elastic_load_balancing) for EKS Auto Mode.

Terraform eks module released a new version v20.31.0 which enables the support of EKS Auto mode and EKS Hybrid Nodes.

Let's use Terraform AWS EKS module for EKS Auto Mode

If you want to follow along use this repository for the working code.

Enable EKS Auto mode for new cluster

One node pool (general purpose) created by EKS Auto mode

There are no nodes or pods in the cluster( no workload is running) or can also say I did not provisioned any nodes so far because that the job of EKS Auto mode now.
The moment I install a sample using this code, EKS Auto Mode provisions the Ec2 nodes by itself. What a magic zero management for node provisioning from my side.

Note: You need to run kubectl set context command and run terraform apply again to deploy the sample app. Sample app wasn't deployed when you created cluster because context wasn't set.

aws eks --region us-east-1 update-kubeconfig --name tf-module-support --profile ck-test

terraform apply

 kubectl get po
NAME                    READY   STATUS    RESTARTS   AGE
test-65b7dbddd4-j6mbt   1/1     Running   0          104s
test-65b7dbddd4-wdz56   1/1     Running   0          104s

What is making the difference

cluster_compute_config resource is the difference or resource use to enable or disable EKS auto Mode in the module side.
At the terraform aws provider side compute_config is used to enable or disable to the EKS Auto Mode

  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }

Due to cluster_compute_config option, now I don't have to mention eks_managed_node_group_defaults, eks_managed_node_groups, node_security_group_additional_rules or even know what are those concepts.

The following code might look small but if someone who does know about EKS and node provisioning has to understand the concepts as well figure out how to write this code. But now due to EKS Auto mode no more management. WOW just so sleek.

# code which is not needed anymore

eks_managed_node_group_defaults = {
    ami_type       = "AL2_x86_64"
    instance_types = ["m5.large"]
    # instance_types = ["t3.small"]
    # vpc_security_group_ids = [aws_security_group.all_worker_mgmt.id]
    iam_role_additional_policies = {
      ebs_policy                                 = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy" #IAM rights needed by CSI driver
      auto_scaling_policy                        = "arn:aws:iam::aws:policy/AutoScalingFullAccess"
      cloudwatch_container_insights_agent_policy = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      xray_policy                                = "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess"
    }
  }

  eks_managed_node_groups = {

    node_group = {
      min_size     = 2
      max_size     = 5
      desired_size = local.node_group_desired_size
    }
  }

  node_security_group_additional_rules = {
    http_traffic_node_to_node = {
      description = "Allow inbound HTTP from self"
      from_port   = 80
      to_port     = 80
      protocol    = "tcp"
      self        = true
      type        = "ingress"
    }
  }

# For triggering managed node group desired size

resource "null_resource" "update_desired_size" {
  triggers = {
    desired_size = local.node_group_desired_size
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    command = <<-EOT
      aws eks update-nodegroup-config \
        --cluster-name ${module.eks.cluster_name} \
        --nodegroup-name ${element(split(":", module.eks.eks_managed_node_groups["node_group"].node_group_id), 1)} \
        --scaling-config desiredSize=${local.node_group_desired_size} \
        --region us-east-1 \
        --profile ck-test
    EOT
  }
}

What isn't supported

For existing EKS cluster where you want to enable eks auto mode utilizing built-in node pool is not possible at the moment using terraform due to bug in terraform aws provider side.

From DevOps, IaC perspective

We saw how we can use EKS auto mode; a game changer feature for container workloads on a new cluster where we did not had to plan, provision infrastructure (compute) for running our workloads.
Still some bug fixes are needed on terraform aws provider side as well EKS side to enable eks auto mode for existing cluster till then use console.
Not only EKS auto mode takes away operations from user but also managed to simplify the IaC(terraform code)
As a consumer of terraform, terraform-eks-module it is mind blowing to see the speed at which this feature was supported. All thanks to Bryant Biggs for supporting this community.

Unlocking Aurora DSQL with AWS Lambda: A Seamless Solution for Serverless, Scalable, and Event-Driven Architectures

Jatin Mehrotra — Wed, 04 Dec 2024 02:09:39 +0000

AWS re:Invent has begun and there are tons of new service and feature announcement from the AWS CEO Matt Garman's Keynote ( check my x/bluesky thread from the keynote updates)

In this blog we will investigate about Amazon Aurora DSQL which is serverless, distributed SQL database with virtually unlimited scale, high availability, and zero infrastructure management claiming 99.99% single-Region and 99.999% multi-Region availability.

My intentions for this blog is to make you understand the architecture, innovation and core components and provide you a completely serverless solution to manage Aurora DSQL using AWS Lambda function with working code 🔥

Motivation

December 4, 2024 4AM (JST) In the Day 2 re:Invent 2024, AWS released an exciting update introducing Amazon Aurora DSQL(Preview)!!!

What does this Update Mean for You?

Here are the key reasons why Amazon Aurora DSQL is a game-changer:

Unlimited Scalability: Effortlessly scale reads, writes, compute, and storage to handle any workload without sharding or instance upgrades.
High Availability: Aurora DSQL's active-active serverless design automates failure recovery, ensuring seamless Multi-AZ and multi-Region availability with strong consistency, eliminating concerns about failovers or data loss.
Fastest Performance: Offers the fastest distributed SQL reads and writes, making it ideal for high-performance applications.
Zero Infrastructure Management: Fully serverless design eliminates the need for patching, upgrades, and maintenance downtime, saving time and resources.
Strong Data Consistency: Aurora DSQL is optimized for transactional workloads that benefit from ACID transactions and a relational data model.
PostgreSQL Compatibility: Simplifies development with a familiar and widely-used SQL interface, reducing learning curves.
Developer-Friendly: An intuitive experience that enables rapid application development without operational complexities.

Cost of Aurora DSQL

Aurora DSQL is currently available in preview at no charge.

Core Components in Aurora DSQL

Distributed Architecture

Aurora DSQL is designed as a distributed database, meaning its parts work together across multiple locations (Availability Zones) to ensure high availability and fault tolerance. It has four main components:

Relay and Connectivity: Handles how data moves within the system and connects users to the database.
Compute and Databases: Manages the actual processing of queries and database logic.
Transaction Log and Isolation: Ensures safe, consistent handling of multiple simultaneous operations (e.g., no data conflicts).
User Storage: Stores the actual data securely.

A control plane oversees and coordinates these components, which are designed to self-heal and scale automatically if something fails.

Aurora DSQL Clusters

Single-Region Clusters:
- Your data is synchronized across multiple data centers (AZs) within a single Region.
- This setup avoids issues like replication delays or database failovers.
- Strong consistency ensures all users see the same data no matter where they connect.
- If part of the system fails, requests automatically shift to healthy infrastructure without your intervention.
- Supports ACID transactions (ensuring reliability, consistency, and durability).
Multi-Region Linked Clusters:
- These extend the above features across multiple Regions, offering two endpoints (one in each Region) that act as a single database.
- Both Regions can handle reads and writes simultaneously while ensuring strong consistency.
- Ideal for global applications where performance and resilience are crucial.

PostgreSQL Compatibility

Aurora DSQL is built on PostgreSQL 16, a popular open-source

How Data Resiliency and Backup is supported

Backup and Restore

Currently, backup and restore is not supported during the preview phase.
Aurora DSQL plans to integrate with the AWS Backup console, enabling full backup and restore capabilities for both single-Region and multi-Region clusters.

Replication

Transaction Logs:
- Aurora DSQL commits all writes to a distributed transaction log and replicates data synchronously across three AZs.
Multi-Region Replication:
- Provides cross-Region replication for both read and write Regions.
- Uses a witness Region for encrypted transaction log storage, requiring no manual configuration or storage overhead.
Data Management:
- Automatically splits, merges, and replicates data based on access patterns and primary key ranges.
- Dynamically scales read replicas based on read demand.
Self-Healing:
- Redirects access during AZ impairments and repairs missing data asynchronously.
- Repaired replicas are added back to the storage quorum automatically.

High Availability

Active-Active Design:
- Both single-Region and multi-Region clusters are active-active, with fully automated recovery.
- Eliminates the need for traditional primary-secondary failover processes.
Multi-AZ Replication:
- Ensures synchronous replication across three AZs, avoiding risks of data loss or lag.
Regional Endpoints:
- Single-Region clusters offer a redundant endpoint for consistent reads and writes across three AZs.
- Multi-Region clusters provide two Regional endpoints for zero-lag, strongly consistent access across Regions.
- Use Amazon Route 53 for managed global endpoints if needed.

Using AWS Lambda with Aurora DSQL

Ofcourse its re:Invent and I would explain my readers a innovative way to connect to Aurora DSQL Database and perform Database operations like creating table and inserting some data using AWS Lambda function

During preview, you can interact with clusters in us-east-1 – US East (N. Virginia) and us-east-2 – US East (Ohio).

Create Aurora DSQL database

Search Aurora DSQL in console

For this blog I am creating single region database and keeping other confgiraution as default. In the preview, name of the cluster is configured using Name Key-Value Tags.
Select create cluster and after creation copy the endpoint url of DB.

That's it nothing else to manage, provision. That's 1 click deployment of Aurora DSQL for you!!!

Create Lambda function

This lambda function will connect to Aurora DSQL database, create table, insert some data and then verify that data by reading it back. Yes a truly server less operation using Lambda and Aurora DSQL

Authorize your Lambda execution role to connect to your cluster by adding inline policy Admin role as inline Policy to all the resources.
Lambda -> Configuration -> Permissions -> add inline policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": ["dsql:DbConnectAdmin"],
            "Resource": ["*"]
        }
    ]
}

Note: You shouldn't use a admin database role for your production applications, doing this for the blog

If you have worked with Lambda then you know we need to upload a zip package. I have shared the lambda code in my repository
Use the following commands after forking and pulling the repository to your local machine

npm install. //generates package-lock.json
cd ~/path-to-code
zip -r pkg.zip .

Upload the package

In your Lambda function’s Test tab, use the following Event JSON modified to specify your cluster’s endpoint.

{"endpoint": "replace_with_your_cluster_endpoint"}

You can also use Environment Variables then you need to change code and refer endpoint as process.env.ENDPOINT

Test the truly serverless operations (AWS LAMBDA + Aurora DSQL)

Implementation worked but it failed
Lets use Amazon Q to diagnose it and fix it for us

Increasing Lambda timeout to 10 seconds from default 3 seconds
Voila it worked !!!

Note:TO verify Data we are using the following code within our Lambda Function

        assert.strictEqual(result.rows[0].city, "Anytown");
        assert.notStrictEqual(result.rows[0].id, null);

Now lets try to fail the lambda by giving wrong endpoint

Understanding authentication and authorization for Aurora DSQL

Aurora DSQL uses IAM roles and policies for cluster authorization and authentication. You associate IAM roles with PostgreSQL database roles for database authorization.
When you connect, instead of providing a credential, you use a temporary authentication token (valid for 1 hour).
For authentication:
- *If you're using the admin role:* IAM identity should have the policy action of dsql:DbConnectAdmin
- *If you're using the custom database role:* IAM identity should have the policy action of dsql:DbConnect

Interact with your database using PostgreSQL database roles and IAM roles

For Database Authorization:
- Use PostgreSQL database roles for database-level authorization.
- Aurora DSQL provides two types of roles:
- Admin Role: Pre-created by Aurora DSQL, unmodifiable, and used for administrative tasks like creating custom roles.
- Custom Roles: Created by you and assigned PostgreSQL permissions as needed.
Role Association:
- Link custom database roles with IAM roles to allow IAM identities to connect to the database.
Authentication and Authorization:
- Use the admin role to connect to clusters and manage custom roles.
- Use the AWS IAM GRANT command to associate IAM identities with custom roles for database access.

For detailed steps, refer to:

Exploring Aurora DSQL: Access, Development, and Best Practices

I am very happy and satisfied with the documentation for the Aurora DSQL. It's to the points and really well written.

When working with aurora there will certain topics which would be of interest like:

Accessing Aurora DSQL
Working with Amazon Aurora DSQL
Programming with Aurora DSQL
Utilities, tutorials, and sample code in Amazon Aurora DSQL
Security and its best practices : It is important to understand both Detective Best practices as well as Preventive security Best Practices
At the writing of this blog there are unsupported PostgreSQL features in Aurora DSQL which should be considered too

From Solutions Architect Perspective

We saw in this blog how we can create Aurora DSQL cluster with just one click. It is truly serverless.
We also saw how an example of a truly serverless solution for managing Aurora DSQL database using AWS Lambda functions making Aurora DSQL is ideal for application patterns of microservice, serverless, and event-driven architectures
Amazon Aurora is a game changer for developers, offering serverless architecture with automatic scaling, high availability, and resilience. It eliminates the need for manual database management, allowing developers to focus on building applications. With Aurora, developers can innovate faster while ensuring reliability and efficiency.
Aurora DSQL is PostgreSQL compatible, so you can use familiar drivers, object-relational mappings (ORMs), frameworks, and SQL features.

Did you tried the Lambda solution presented in the blog and do you think this will be game changer for serverless event driven and micrservice architectures?

Drop your thoughts in the comments below or connect with me on Linkedin, X! 🚀

Run Kubernetes Like a Pro—Without the Expertise! Introducing EKS Auto Mode

Jatin Mehrotra — Mon, 02 Dec 2024 04:17:24 +0000

AWS re:Invent hasn't officially begun, yet there is a game changing new feature to EKS to make you run Kubernetes like a pro!!!

In this blog we will investigate what EKS Auto Mode is all about and illustrates how to enables EKS Auto Mode on existing cluster and try to migrate to our managed node groups workloads to EKS auto mode of an existing cluster but we will be not able to do that, find out why in the blog!

I have also analyzed the price you will pay for EKS Auto v/s the simplicity of operations it provides to help you make an informed decision whether to enable it or not.

I have created a terraform repository to quickly reproduce this blog with just 2 commands, let's conquer EKS Auto Mode.

Motivation

December 2, 2024 (JST) – Just ahead of re:Invent 2024, AWS released an exciting update introducing EKS Auto Mode.

What Does This Update Mean for You?

It automates all kinds of management for your EKS cluster which allow AWS to also set up and manage the infrastructure that enables the smooth operation of your workloads.
Cluster infrastructure managed by AWS includes many Kubernetes capabilities as core components, as opposed to add-ons, such as compute autoscaling, pod and service networking, application load balancing, cluster DNS, block storage, and GPU support
You can deploy a new EKS Auto Mode cluster or enable EKS Auto Mode on an existing cluster

Note: Terraform support(still ongoing) not yet there so for this blog we will use AWS console.

Cost considerations of this Update

According to https://aws.amazon.com/eks/pricing/
You will stay pay for 2 components :
- Amazon EKS cluster pricing: *$0.10 per cluster/hour *($0.60/cluster for extended)
- But if you use auto mode will have to pay extra!!!

You pay for Amazon EKS Auto Mode based on the duration and type of Amazon EC2 instances launched and managed by EKS Auto Mode. The Amazon EKS Auto Mode prices below are in addition to the Amazon EC2 instance price, which covers the EC2 instances themselves.

In simple words EKS Auto Mode charges a management fee that varies based on the EC2 instance type launched, in addition to your regular EC2 instance costs.

What does it actually automate

EKS auto mode is designed to automate data plane components so here are the following components being automated along with the features:

Automated Data Plane Components in EKS Auto Mode

Compute Management: Automates key aspects of cluster compute, including node provisioning, scaling, upgrades, and load balancing, for hassle-free Kubernetes operations.
- Nodes:
  - Automatically selects optimized AMIs with essential services.
  - Secures nodes using SELinux, read-only root files, and disallowed SSH/SSM access.
  - Includes GPU support for NVIDIA and Neuron GPUs.
- Auto Scaling:
  - Uses Karpenter to monitor and deploy new nodes for unschedulable Pods.
  - Dynamically terminates unused nodes to optimize resources.
Upgrades:
- Applies OS and component updates with minimal workload disruption.
- Enforces a 21-day maximum node lifetime for security and stability.
Load Balancing:
- Integrates with Elastic Load Balancing (ALB & NLB) for automated provisioning and scaling.
- Provides production-ready load balancing aligned with AWS best practices.
Storage Automation
- Configures ephemeral storage with volume settings, encryption, and deletion policies automatically managed for optimal use.
Networking Automation
- Manages Pod and service connectivity with IPv4/IPv6 support and extends IP spaces using secondary CIDR blocks.

Enabling auto mode on Existing Cluster

In this blog we will see how to enable on an existing cluster as AWS blog already covered EKS auto mode on new cluster
To mimic existing cluster, you can deploy EKS cluster with EKS managed node groups using Terraform in my repo

Migrations Supported

At the writing of this blog EKS support migrating from Karpenter to EKS Auto Mode, Migrate from EKS Managed Node Groups to EKS Auto Mode, Migrating from EKS Fargate to EKS Auto Mode Nodes
To keep this blog simple I will show how to migrate from EKS managed Node Groups to EKS Auto Mode (hopefully :))

Note: Also remember to take a look on how to configure Kubernetes based resource to be owned by either self-managed controllers or EKS Auto Mode.

Prerequisites

Updating IAM permissions and configuring core EKS Auto Mode settings
Minimum required version of certain Amazon EKS Add-ons Perspective.

Updating IAM permissions and configuring core EKS Auto Mode settings

Cluster IAM role of an EKS Cluster cannot be changed after the cluster is created. EKS Auto Mode requires additional permissions on this role. You must attach additional policies to the current role.

Navigate to Cluster IAM role and adding the following policies
- AmazonEKSComputePolicy
- AmazonEKSBlockStoragePolicy
- AmazonEKSLoadBalancingPolicy
- AmazonEKSNetworkingPolicy
- AmazonEKSClusterPolicy ( already exist for me )

Edit the trust Policy and add sts:TagSession to allow Action.

Minimum required version of certain Amazon EKS Add-ons

Perspective.

I have set the terraform code to fetch latest versions of add-ons so you don't have to do anything.

Create a Karpenter Node Pool

EKS auto uses Karpenter behind the scenes for scaling.

Karpenter, a node provisioning tool that helps optimize cluster scaling and resource utilization. With Karpenter’s NodePool resource, you can define specific requirements for your compute resources, including instance types, availability zones, architectures, and capacity types.

Select Create recommended roles for Node IAM Role
I am using preconfigured node pool, but you can configure your own using your own yaml file like this example

Select default permissions and create role

Once the role is created and you select next you can see notifications for auto mode being enabled

Once enabled you can confirm the auto mode option using Overview Tab in the console

As enabled during enable step EKS created node pools too.

Disable EKS Auto Modę

You can disable EKS Auto Mode on an existing EKS Cluster. This is a destructive operation.

EKS will terminate all EC2 instances operated by EKS Auto Mode.
EKS will delete all Load Balancers operated by EKS Auto Mode.

Migrate from EKS Managed Node Groups to EKS Auto Mode

When transitioning your Amazon EKS cluster to use EKS auto mode, you can smoothly migrate your existing workloads from managed node groups using the eksctl CLI tool.

This process ensures continuous application availability while EKS auto mode optimizes your compute resources. The migration can be performed with minimal disruption to your running applications.

Prerequisites

Cluster with EKS Auto Mode enabled
eksctl CLI installed and connected to your cluster. For more information, see Set up to use Amazon EKS.
Karpenter is not installed on the cluster.

Current state of Pods

2 pods are running on managed node groups.

jatin.mehrotra@CK0662-001 eks-auto-mode % kubectl get po -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP           NODE                         NOMINATED NODE   READINESS GATES
test-65b7dbddd4-jdxhw   1/1     Running   0          2m29s   10.0.2.52    ip-10-0-2-85.ec2.internal    <none>           <none>
test-65b7dbddd4-lzd4r   1/1     Running   0          2m29s   10.0.1.118   ip-10-0-1-128.ec2.internal   <none>           <none>

As per docs, Use the following eksctl CLI command to initiate draining pods from the existing managed node group instances. EKS Auto Mode will create new nodes to back the displaced pods.

eksctl update auto-mode-config --cluster eks-auto-test --drain-nodegroup  --region us-east-1 --profile ck-test

I thought it's my eksctl version but looks like my eksctl hasn't got update for this flag so far. There is open issue too for the official eksctl repository

my eksctl version output (its the latest version)

eksctl info 
eksctl version: 0.194.0

The other commands for eksctl are running for my cluster so definitely not the connection problem

jatin.mehrotra@CK0662-001 eks-auto-mode % eksctl get addons  --cluster eks-auto-test  --region us-east-1 --profile ck-test 
2024-12-02 12:29:41 [ℹ]  Kubernetes version "1.30" in use by cluster "eks-auto-test"
2024-12-02 12:29:41 [ℹ]  getting all addons
2024-12-02 12:29:44 [ℹ]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME                            VERSION                 STATUS  ISSUES  IAMROLE UPDATE AVAILABLE                                                                     CONFIGURATION VALUES     POD IDENTITY ASSOCIATION ROLES
amazon-cloudwatch-observability v2.5.0-eksbuild.1       ACTIVE  0
aws-ebs-csi-driver              v1.37.0-eksbuild.1      ACTIVE  0
coredns                         v1.11.1-eksbuild.8      ACTIVE  0               v1.11.3-eksbuild.2,v1.11.3-eksbuild.1,v1.11.1-eksbuild.13,v1.11.1-eksbuild.11,v1.11.1-eksbuild.9
kube-proxy                      v1.30.6-eksbuild.3      ACTIVE  0
vpc-cni                         v1.19.0-eksbuild.1      ACTIVE  0

Update: Dec 3, 12:20 AM JST

Even though AWS team has released a new version of eksctl, issue still persist as well as documentation is also incorrect.

It needs --drain-all-nodegroups as well as cluster-config file which is very difficult to generate for existing cluster created by console or terraform.

eksctl update auto-mode-config --drain-all-nodegroups -f eks-config.yaml --profile ck-test

We need to wait more so that AWS can fix this bug. There is new issue for this bug.

Investigating the Update the Kubernetes Version of an EKS Auto Mode cluster

The compute capability of Amazon EKS Auto Mode controls the Kubernetes version of nodes. After you upgrade the control plane, EKS Auto Mode will begin incrementally updating managed nodes. EKS Auto Mode respects pod disruption budgets.
You do not have to manually upgrade the capabilities of Amazon EKS Auto Mode, including the compute autoscaling, block storage, and load balancing capabilities.
Auto Mode simplifies the version update process by handling the coordination of control plane updates with node replacements, while maintaining workload availability through pod disruption budgets.
When upgrading an Auto Mode cluster, many components that traditionally required manual updates are now managed as part of the service.
After you initiate a control plane upgrade, EKS Auto Mode begins replacing nodes in your cluster. The new nodes have the corresponding new Kubernetes version. EKS Auto Mode observes pod disruption budgets when upgrading nodes.
- CoreDNS
- KubeProxy
- AWS Load Balancer Controller
- Karpenter
- AWS EBS CSI Driver
However You are still responsible for updating:
- Apps and workloads deployed to your cluster
- Self-managed add-ons and controllers
- Amazon EKS Add-ons

From Solutions Architect Perspective

In this blog we saw how EKS Auto mode simplifies Kubernetes operations by offloading infrastructure management to AWS.
It automatically selects the best EC2 instances, optimizes compute costs, and dynamically scales resources based on demand.
This feature enhances security, performance, and availability while reducing the need for deep expertise, capacity planning, and manual management.
But as we saw in the blog its still new feature there might be a need for updates and bug fixes.
Yes, its true the its simplifies operations and allows users to run Kubernetes like a PRO in secure manner but you have to pay a price for it.

Initially I think this will increase the cost but the real benefits would kick in after a sustained usage to offset the extra price with ease of operations.

Drop your thoughts in the comments below or connect with me on Linkedin, X! 🚀