Forem: Phu Hoang

Amazon EKS From The Ground Up - Part 2: Worker Nodes with AWS Managed Nodes

Phu Hoang — Sat, 10 Jan 2026 08:03:00 +0000

Introduction

In Part 1, we successfully finished building the EKS Control Plane, we set up the VPC, Subnets, NAT Gateway, the Kubernetes API Server, and configured kubectl connectivity.

VPC, Subnets, NAT
Kubernetes API Server
An IAM role with enough permissions for the Control Plane
kubectl already connected to the cluster

But if you run:

kubectl get nodes

you’ll see… nothing.

That’s the real state of the cluster we created: it has a brain (Control Plane), but no hands and feet (Worker Nodes). Without compute capacity, the Kubernetes Scheduler can only stare at Pods stuck in the Pending state.

In this Part 2, we’ll “grow limbs” for your EKS cluster by deploying Worker Nodes using AWS Managed Nodes - the most practical middle ground between production readiness and operational effort.

This post won’t stop at “click and it works.” We’ll also explore what AWS quietly builds for you in a Managed Node Group, and contrast it with the Self-managed way so you understand the fundamentals and can debug with confidence.

📝A Quick Primer on Worker Nodes

What is a Worker Node, really?

In the most accurate - and simplest - definition:

A Worker Node is an EC2 instance configured to run Kubernetes workloads (Pods).

At minimum, every worker node runs:

kubelet — the agent that talks to the Kubernetes API Server
a container runtime — typically containerd
kube-proxy — manages service/network rules
AWS VPC CNI — the plugin that allocates Pod IPs from your VPC

Following production best practices, EKS worker nodes usually live inside private subnets without public IPs. That significantly reduces exposure: workloads are not directly reachable from the Internet, and outbound connectivity is handled through a NAT Gateway.

Worker Node Deployment Models

AWS offers three main ways to run compute for EKS, each with a different balance of control, operational cost, and “serverless-ness”:

Criteria	Managed Nodes	Self-managed Nodes	AWS Fargate
Infrastructure	EC2 Instances (partly managed by AWS)	EC2 Instances (fully managed by you)	Serverless (no nodes to manage)
Operational cost (OpEx)	Low	High	Very low
Control	Medium	Highest	Lowest
Billing	Per EC2 instance	Per EC2 instance	Per Pod (CPU/Mem)

Fargate hides almost everything—including the “node layer.” That’s convenient, but if your goal is to understand EKS fundamentals, the interesting engineering happens in Managed Nodes vs Self-managed Nodes.

Criteria	Managed Nodes (Managed Node Group)	Self-managed Nodes
Node creation	EKS Console or CLI command creates a Node Group	EC2 Launch Template + ASG + User Data (bootstrap)
ASG management	AWS manages the Auto Scaling Group lifecycle	You manage the ASG entirely
Cluster Join	AWS automatically handles the wiring and credentials for join	You must provide user data calling `/etc/eks/bootstrap.sh`
Node auth mapping (IAM→ RBAC)	AWS typically maps the Node Role automatically	You must manually update `aws-auth` to map the Node Role
Upgrades/Updates	Built-in, managed rolling update workflows	You design and manage `drain` / `replace` strategies
Debug level	Fewer common traps, higher abstraction	More control, but more responsibility for low-level configuration

What AWS does for you in a Managed Node Group

When you click Create Node Group, AWS typically handles a long checklist that Self-managed nodes would require you to build manually:

Creates an Auto Scaling Group
Picks an EKS-Optimized AMI compatible with your cluster version
Creates/manages a Launch Template (or uses one you provide)
Attaches an IAM Instance Profile using your Node IAM Role
Injects the bootstrap configuration so nodes can join the cluster
Automates the node registration path
Provides a rolling update workflow for node group upgrades
Typically handles node role mapping into the cluster auth mechanism

Recommendation: Use Managed Nodes for most production setups to reduce operational overhead. Choose Self-managed only when you have very specific requirements (custom OS hardening, special bootstrap, deep control of the node lifecycle).

Cluster IAM Role vs Node IAM Role

This is one of the most common points of confusion, so let’s make it crystal clear.

Cluster IAM Role

Used by the EKS Control Plane
Allows the Control Plane to manage ENIs and interact with your VPC resources

This role is not meant for workloads.

Node IAM Role

Worker nodes need a Node IAM Role to:

Join the cluster
Allow the VPC CNI to attach ENIs and allocate Pod IPs
Pull images from ECR
Access other required AWS APIs (later: secrets, parameters, logs, etc.)

Your worker nodes won’t become Ready without (at least) these managed policies:

Policy	Purpose
`AmazonEKSWorkerNodePolicy`	Join cluster, talk to the API Server
`AmazonEKS_CNI_Policy`	Attach ENIs, allocate Pod IPs
`AmazonEC2ContainerRegistryReadOnly`	Pull images from ECR

AWS IAM & Kubernetes Authentication

Access to an EKS cluster is a combination of two layers: AWS IAM & Kubernetes RBAC.

IAM = Authentication

IAM answers: “Who are you?”

When a principal (an EC2 node or a human user) calls the Kubernetes API Server, EKS uses IAM authentication to verify:

Which IAM principal (Role/User) the request comes from
Whether the request has a valid SigV4 signature

✅ This is why worker nodes must have a proper Node IAM Role attached.

Kubernetes RBAC = Authorization

RBAC answers: “What are you allowed to do?”

Even if IAM authentication succeeds, the API call can still fail with Forbidden if RBAC doesn’t grant the required permissions.

Bridging the two worlds: mapping IAM → Kubernetes identity

After IAM authentication, EKS maps IAM identity into Kubernetes users/groups so RBAC can evaluate permissions. Two common mechanisms exist:

aws-auth ConfigMap (classic, still widely used), example:

mapRoles: |
  - rolearn: arn:aws:iam::<account-id>:role/EKSNodeRole
    username: system:node:{{EC2PrivateDNSName}}
    groups:
      - system:bootstrappers
      - system:nodes

EKS Access Entries / Access Policies (newer Console-based approach)

For this article:

Nodes must be mapped into groups like system:bootstrappers and system:nodes
Humans/admins are commonly mapped to system:masters or granted an equivalent Access Policy

Hands-on Time

The AWS architecture after completing the steps below:

Step 0 — Preparation

Make sure all resources from Part 1 are created correctly and your cluster is ACTIVE.

Step 1 — Create the IAM Role for Worker Nodes

Open the AWS Console:

→ Go to IAM

→ Choose Roles → Create role

→ Configure Trusted entity:

Trusted entity type: AWS service
Service or use case: EC2
Use case: EC2

→ Click Next

→ Attach policies:

AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly

→ Role name: EKSNodeRole

→ Click Create role

Step 2 — Create a Managed Node Group

→ Open the EKS service

→ Select cluster demo-eks-cluster

→ Go to Compute → Add node group

Configure node group

Name: eks-mng-general
Node IAM role: EKSNodeRole

→ Click Next.

Configure compute & scaling

AMI type: EKS optimized (Amazon Linux / Bottlerocket)
Capacity type: On-Demand
Instance type: t3.medium
Disk size: 20 GiB
Scaling:
- Desired: 2
- Min: 1
- Max: 3

→ Keep other settings as default.

→ Click Next.

Configure networking

Select only the two private subnets.

This is critical: subnet selection here determines where your worker nodes live. Private subnets are ideal for production worker nodes because they don’t expose instances to the Internet.

→ Click Next → Create.

Wait until the node group status becomes Active.

Step 3 - Join the Cluster - AWS handles it

You don’t need to manually configure bootstrap steps for a Managed Node Group - but understanding the join flow is what makes you effective at troubleshooting.

3.1 Node join flow

When you create a Managed Node Group, AWS launches EC2 worker nodes. Each instance gets an Instance Profile containing your Node IAM Role (EKSNodeRole). The join flow looks like this:

The EC2 instance boots using an EKS-Optimized AMI
Bootstrap config provides kubelet with:
- cluster name
- API endpoint
- cluster CA certificate
kubelet calls the Kubernetes API Server to register the node
EKS performs IAM authentication and identifies the IAM Role from the instance profile
EKS maps IAM identity → Kubernetes identity via aws-auth / Access Entries
If mapping is valid (node is in system:nodes), the node becomes Ready

In short: nodes don’t “join by Kubernetes magic.” They join because:

IAM authentication proves identity + Kubernetes group mapping allows the node role to function.

3.2 What Managed Node Group eliminates compared to Self-managed

With Self-managed nodes, you must build the join path yourself:

Create Launch Template (AMI, instance profile, user data)
Ensure bootstrap via /etc/eks/bootstrap.sh <cluster-name>
Create ASG and subnet placement
Manually update aws-auth to map the node role into:
- system:bootstrappers
- system:nodes

Managed Node Groups remove most of this plumbing.

Step 4 — Verify

This is the moment your EKS cluster finally starts to feel alive.

4.1 Verify with kubectl

→ Open a terminal and run:

kubectl get nodes -o wide

→ Then check system pods:

kubectl get pods -n kube-system

Expected results:

You should see two nodes (based on desired size), in Ready state
coredns, metrics-server, and kube-proxy should transition to Running

4.2 What AWS resources were created behind the scenes?

Now let’s satisfy curiosity and confirm what AWS created inside your account.

(1) Auto Scaling Group

→ Open EKS Service

→ Select EKS cluster demo-eks-cluster

→ Click Compute tab

→ Select node group eks-mng-general

→ In Details, click the Auto Scaling Group

Inside the ASG page, you’ll find:

Desired / Min / Max configuration
EC2 instances in InService
Launch Template reference
Security Groups

(2) Launch Template

From the ASG page:

→ Click the Launch template link.

You’ll see:

AMI ID
Instance type
Security groups attached
User data/bootstrap wiring (partly hidden, but it’s there)

(3) Security Group for worker nodes

From the ASG details page

→ Click the Security group IDs.

Review inbound/outbound rules applied to worker nodes.

Common Pitfalls

1) Node group is Active, but `kubectl get nodes` shows nothing

Likely causes:

The node group is using the wrong IAM role (not EKSNodeRole)
Node IAM role is missing required policies
Wrong subnet selection or private subnet route tables are incorrect

2) Instances keep launching and terminating in the ASG

Likely causes:

Instance type capacity shortage → try a more common type (t3.large, m5.large, etc.)
Subnet/AZ constraints → expand to more AZs/subnets
EC2 quota limits → request quota increase

3) Pods stuck in `Pending`

Likely causes:

Insufficient node resources (CPU/memory) → choose a larger instance type
Taints/labels preventing scheduling → remove taints or adjust selectors

4) `ImagePullBackOff` / `ErrImagePull`

Likely causes:

Private subnets have no NAT gateway, or routes are wrong
DNS resolution is broken → check VPC settings (DNS resolution and DNS hostnames)

Summary

In Part 2, we:

Added production-style worker nodes (private subnets) so workloads finally have somewhere to run
Clearly separated Cluster Role vs Node Role
Covered the IAM → Kubernetes authentication story
Explored what a Managed Node Group creates behind the scenes

Next, in Part 3, we’ll go deeper into EKS networking: VPC CNI, ENIs, Pod IP allocation, and traffic flow debugging.

Amazon EKS From The Ground Up - Part 1: Control Plane & Infrastructure

Phu Hoang — Thu, 25 Dec 2025 10:25:30 +0000

Introduction

As a DevOps engineer, the ability to provision and manage Kubernetes clusters efficiently is essential. Amazon EKS (Elastic Kubernetes Service) makes this significantly easier—you can create an EKS cluster with just a single command or a few clicks in the AWS Console.

While you’re sipping your coffee, however, a lot is happening behind the scenes.

This article is the first part of a deep-dive series that breaks down those behind-the-scenes actions. We will examine the components that make up an Amazon EKS cluster, focusing specifically on the Infrastructure layer and IAM & Security, which form the foundation of everything that comes later.

A Brief Concept of EKS

Amazon EKS is a managed Kubernetes service that allows you to run Kubernetes without having to operate the Control Plane yourself.

In simple terms, EKS handles the two most difficult aspects of running Kubernetes:

Setting up and managing the Control Plane

The Control Plane is the “brain” of Kubernetes. It consists of the API server, the etcd database (which stores cluster state), the scheduler, and the controller manager. AWS ensures that this Control Plane is highly available, secure, and continuously patched.
Deep integration with AWS services

EKS tightly integrates Kubernetes with core AWS services such as VPC networking, IAM-based security, and Elastic Load Balancing.

Building the Foundation – Infrastructure First

A very common misconception among newcomers is the following:

EKS = Kubernetes, and AWS handles everything.

In reality, EKS is split into two distinct responsibility domains:

Component Partition	Components	Management Responsibility
AWS-managed (Control Plane)	API Server, etcd, Scheduler, Controller Manager	AWS (not configurable)
Your AWS Account (Data Plane & Networking)	VPC, Subnets, Security Groups, IAM Roles, EC2 Worker Nodes or Fargate Pods	You (must be designed and maintained)

For this reason, the remainder of this article focuses on the infrastructure that you are responsible for, before the EKS Control Plane is even created.

Step 0 – Preparing the Tools

Before creating the EKS cluster, ensure you have the following ready:

An AWS account with access to the AWS Console
AWS CLI installed and configured
kubectl installed

Although this article is console-first, we will use the CLI to verify and explain what happens under the hood.

Step 1 – Preparing the VPC

The EKS Control Plane itself does not run inside your VPC. However, you still need a VPC for:

EC2 worker nodes
Pod IP address allocation
Security groups to control traffic
Internet or NAT access for outbound communication

In short: without a properly designed VPC, EKS cannot function.

1.1. Actions in the AWS Console

→ Open the Amazon VPC service

→ Click Create VPC

→ Choose VPC and more

Recommended Configuration (Practice / Production-Ready Baseline)

Component	Value	Notes
Name	`eks-demo-vpc`
IPv4 CIDR	`10.0.0.0/16`
Availability Zones	2	High availability
Public subnets	2	Used for NAT Gateways and public load balancers
Private subnets	2	Where worker nodes will run
NAT Gateway	1 per AZ	Required for outbound access
DNS Hostnames	Enabled	Required for nodes to resolve the API server

Step 2 – Adding Tags to Subnets

Amazon EKS does not automatically infer which subnets it should use. Many EKS features—especially the AWS Load Balancer Controller—depend entirely on subnet tags.

2.1. Required Tags

Cluster discovery tag (required on all subnets):

kubernetes.io/cluster/demo-eks-cluster = shared

Public load balancer subnets (public subnets only):
```
kubernetes.io/role/elb=1
```
Internal load balancer subnets (private subnets only):
```
kubernetes.io/role/internal-elb =1
```

Incorrect or missing tags are one of the most common causes of load balancer failures in EKS.

2.2. Actions in the AWS Console

→ Open VPC → Subnets

→ Locate subnets named eks-demo-subnet-*

→ Select eks-demo-subnet-public1

→ Open the Tags tab → Manage tags

→ Add tags:

kubernetes.io/cluster/demo-eks-cluster = shared
kubernetes.io/role/elb = 1

→ Save

Repeat for eks-demo-subnet-public2.

For each private subnets, add tags:

kubernetes.io/cluster/demo-eks-cluster = shared
kubernetes.io/role/internal-elb = 1

Step 3 - Add a dedicated Security Group for the EKS Cluster

In this step, we’ll create a separate Security Group for the EKS cluster. The goal is to establish a clean security boundary early (even if you’re still in the “control plane only” stage).

Inbound: allow internal traffic within the same Security Group so EKS-related components that share this SG can communicate with each other.
Outbound: allow outbound traffic so the cluster and its components can reach required endpoints (e.g., AWS APIs, image registries, etc.).

Note: This is a learning/bootstrapping-friendly baseline. In production, you typically tighten inbound/outbound rules based on actual traffic requirements.

3.1. Actions in the AWS Console

→ Open VPC service

→ Go to Security Groups

→ Configure the Security Group

Security group name: eks-cluster-sg
VPC: eks-demo-vpc

→ Add (or keep) this outbound rule:

Type	Protocol	Port range	Destination
All traffic	All	All	0.0.0.0/0

→ Then click Create security group.

Now we’ll allow “intra-SG” traffic:

→ Select the Security Group eks-cluster-sg

→ Open the Inbound rules tab → click Edit inbound rules

→ Add a new rule:

Type	Protocol	Port range	Source
All traffic	All	All	Custom → Security group: `eks-cluster-sg`

→ Save the rule.

Step 4 – Creating the IAM Role for the EKS Control Plane

Although AWS manages the Control Plane, it still needs permissions to interact with resources in your AWS account, such as:

Creating and managing ENIs
Attaching security groups
Communicating with VPC resources

This is achieved through an IAM Role that the EKS service assumes.

4.1. Actions in the AWS Console

→ Open IAM → Roles

→ Click Create role

→ Trusted entity:

Type: AWS service
Service: EKS
Use case: EKS – Cluster

→ Continue to permissions

AmazonEKSClusterPolicy is attached automatically

→ Name the role: EKSClusterRole

→ Create the role

4.2. Trust Policy Explanation

The role’s trust policy contains the following statement:

{
    "Effect":"Allow",
    "Principal":{
        "Service":"eks.amazonaws.com"
    },
  "Action":"sts:AssumeRole"
}

This means:

“Allow the EKS service to assume this role and call AWS APIs using the permissions defined in AmazonEKSClusterPolicy.”

Granting only this policy follows the Principle of Least Privilege and is sufficient for the Control Plane to operate correctly.

Step 5 – Creating the EKS Cluster

With the infrastructure and IAM role in place, we can now create the Control Plane.

5.1. Actions in the AWS Console

→ Open Amazon EKS

→ Click Create cluster

→ Choose Custom configuration

→ Disable EKS Auto Mode

In Cluster Configuration step:

→ Name: demo-eks-cluster

→ IAM role: EKSClusterRole

→ Kubernetes version: 1.34 (latest stable at the time of writing)

→ Leave other settings as default

In Networking Configuration step:

→ VPC: eks-demo-vpc

→ Subnets: select all 4 subnets

→ Security group: eks-cluster-sg

→ Endpoint access: Public and Private

Public + Private access allows management from your local machine while ensuring worker nodes communicate internally within the VPC.

In Logging Configuration step:

→ Enable the following logs (recommended):

API server
Audit
Authenticator

In Add-ons Configuration step:

→ Keep the AWS-recommended defaults:

kube-proxy
Amazon VPC CNI
Node monitoring agent
CoreDNS
Amazon EKS Pod Identity Agent
External DNS
Metrics Server

→ Click Create and wait for the cluster status to become Healthy.

Behind the scenes, this is equivalent to:

aws eks create-cluster \
  --name demo-eks-cluster \
  --role-arn arn:aws:iam::<account-id>:role/EKSClusterRole \
  --resources-vpc-config subnetIds=<subnet-public1>,<subnet-public2>,<subnet-private1>,<subnet-private2>

Which do you prefer AWS Console or AWS CLI? 😉

Step 6 – Connecting and Verifying the Cluster

Once the cluster is ACTIVE, connect to it from your local machine.

6.1 Fetching the kubeconfig

aws eks update-kubeconfig --name demo-eks-cluster --region us-east-1

This command:

Retrieves the API endpoint
Updates ~/.kube/config
Configures authentication using your current IAM identity

6.2 Verifying the Cluster

kubectl cluster-info
kubectl get namespaces
kubectl get nodes

Expected Results

cluster-info: API server is reachable
get namespaces: default namespaces are listed
get nodes: no nodes returned

This is expected.

The Control Plane is ready, but no worker nodes exist yet, so Kubernetes has no compute capacity to schedule Pods. Core system Pods such as CoreDNS remain in the Pending state.

Conclusion

In this first part of the series, we built the foundation of an Amazon EKS cluster from the ground up:

Designed a production-ready VPC with properly tagged subnets
Created an IAM role for the EKS Control Plane
Provisioned the EKS Control Plane with secure endpoint access
Verified connectivity while intentionally stopping before worker nodes

At this point, we have built the brain of Kubernetes.

It is powered on, networked, and authorized—but it is still waiting for its “hands”.

Forem: Phu Hoang

Amazon EKS From The Ground Up - Part 2: Worker Nodes with AWS Managed Nodes

Introduction

📝A Quick Primer on Worker Nodes

What is a Worker Node, really?

Worker Node Deployment Models

What AWS does for you in a Managed Node Group

Cluster IAM Role vs Node IAM Role

Cluster IAM Role

Node IAM Role

AWS IAM & Kubernetes Authentication

IAM = Authentication

Kubernetes RBAC = Authorization

Bridging the two worlds: mapping IAM → Kubernetes identity

Hands-on Time

Step 0 — Preparation

Step 1 — Create the IAM Role for Worker Nodes

Step 2 — Create a Managed Node Group

Configure node group

Configure compute & scaling

Configure networking

Step 3 - Join the Cluster - AWS handles it

3.1 Node join flow

3.2 What Managed Node Group eliminates compared to Self-managed

Step 4 — Verify

4.1 Verify with kubectl

4.2 What AWS resources were created behind the scenes?

Common Pitfalls

1) Node group is Active, but kubectl get nodes shows nothing

2) Instances keep launching and terminating in the ASG

3) Pods stuck in Pending

4) ImagePullBackOff / ErrImagePull

Summary

Amazon EKS From The Ground Up - Part 1: Control Plane & Infrastructure

Introduction

A Brief Concept of EKS

Building the Foundation – Infrastructure First

Step 0 – Preparing the Tools

Step 1 – Preparing the VPC

1.1. Actions in the AWS Console

Step 2 – Adding Tags to Subnets

2.1. Required Tags

2.2. Actions in the AWS Console

Step 3 - Add a dedicated Security Group for the EKS Cluster

3.1. Actions in the AWS Console

Step 4 – Creating the IAM Role for the EKS Control Plane

4.1. Actions in the AWS Console

4.2. Trust Policy Explanation

Step 5 – Creating the EKS Cluster

5.1. Actions in the AWS Console

Step 6 – Connecting and Verifying the Cluster

6.1 Fetching the kubeconfig

6.2 Verifying the Cluster

Expected Results

Conclusion

1) Node group is Active, but `kubectl get nodes` shows nothing

3) Pods stuck in `Pending`

4) `ImagePullBackOff` / `ErrImagePull`