Forem: Adedamola Ajibola

The EKS 1.32 to 1.33 Upgrade That Broke Everything (And How I Fixed It)

Adedamola Ajibola — Fri, 26 Dec 2025 15:40:55 +0000

Introduction
Upgrading Kubernetes should be boring.

This one wasn’t.

I recently upgraded a production Amazon EKS cluster from 1.32 to 1.33, expecting a routine change. Instead, it triggered a cascading failure:

Nodes went NotReady
Add-ons stalled indefinitely
Karpenter stopped provisioning capacity
The cluster deadlocked itself

This post walks through what broke, why it broke, and the exact steps that stabilized the cluster so you don’t repeat my mistakes.

Critical Issues

If you're upgrading to EKS 1.33, know this:

Amazon Linux 2 is NOT supported - Must migrate to AL2023 first
Anonymous auth is restricted - New RBAC required for kube-apiserver
Karpenter needs eks:DescribeCluster permission - Missing this breaks everything
Addons can get stuck "Updating" - managed node groups are your escape hatch

Part 1: The Failed First Attempt

What I Did Wrong
I started with what looked like a standard Terraform upgrade:

module "damola_eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = local.name
  cluster_version = "1.33"

What happened

Error: AMI Type AL2_x86_64 is only supported for kubernetes versions 1.32 or earlier

The Root Cause
EKS 1.33 drops Amazon Linux 2 completely:

AL2 reaches end of support on Nov 26, 2025 and no AL2 AMIs exist for 1.33.

The Fix: AL2 → AL2023 Migration
For Karpenter users, this is actually simple. Update your EC2NodeClass:

# EC2NodeClass
spec:
  amiSelectorTerms:
    - alias: al2023@latest

Managed node groups (Terraform)

ami_type = "AL2023_x86_64_STANDARD"

Wait until all nodes are AL2023, then upgrade the control plane.

Part 2: The Karpenter Catastrophe
After migrating to AL2023, I cordoned old nodes, no new nodes came up.

Karpenter was completely stuck.

The Error
Checking Karpenter logs revealed:


{
  "message": "failed to detect the cluster CIDR",
  "error": "not authorized to perform: eks:DescribeCluster"
}

The Root Cause

Starting with Karpenter v1.0, the controller requires eks:DescribeCluster to:

Detect cluster networking (CIDR)
Discover API endpoint configuration
Validate authentication mode

Without this permission, provisioning silently fails.

The Fix
Add the permission to your Karpenter controller IAM role:

{
  "Effect": "Allow",
  "Action": "eks:DescribeCluster",
  "Resource": "*"
}

Then restart:

kubectl rollout restart deployment/karpenter -n karpenter
kubectl rollout status deployment/karpenter -n karpenter

Karpenter recovered but the cluster still wasn’t healthy.

Part 3: The Addon Deadlock

After the control plane upgraded:

Add-ons started updating (vpc-cni, kube-proxy, coredns)
They got stuck in Updating
All nodes went NotReady
No new nodes could join

Classic deadlock:

Nodes need add-ons → add-ons need healthy nodes.

The Error

kubectl get nodes
# All showed: NotReady

kubectl logs -n kube-system -l k8s-app=kube-dns
# Error: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

The Root Causes

Anonymous auth restricted (EKS 1.33)

Anonymous API access is now limited to health endpoints only.
The kube-apiserver requires explicit RBAC to communicate with kubelet.

Add-on update deadlock

Add-ons need healthy nodes to update.
Nodes need working add-ons to become Ready.
When all nodes are NotReady, everything gets stuck.

The Fix Part 1: RBAC for kube-apiserver
Create the missing RBAC:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-apiserver-to-kubelet
rules:
- apiGroups: [""]
  resources: ["nodes/proxy","nodes/stats","nodes/log","nodes/spec","nodes/metrics"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
- kind: User
  name: kube-apiserver-kubelet-client
  apiGroup: rbac.authorization.k8s.io
EOF

Errors stopped but add-ons were still stuck.

The Fix Part 2: Breaking the Deadlock with Managed Nodes

With broken Karpenter nodes, I had no way out.

Solution: temporarily scale up managed node groups.

desired_size = 1
ami_type     = "AL2023_x86_64_STANDARD"

Why this works

Managed nodes bootstrap independently
They come up with working VPC CNI
Add-ons get healthy replicas
Karpenter recovers
Broken nodes can be safely deleted

Within ~10 minutes, the cluster recovered.

Part 4: Final Cleanup & Validation
Once stable verify all nodes healthy:

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
STATUS:.status.conditions[-1].type,\
OS:.status.nodeInfo.osImage,\
VERSION:.status.nodeInfo.kubeletVersion

# All showed:
# Ready | Amazon Linux 2023.9.20251208 | v1.33.5-eks-ecaa3a6

# Verify addons
kubectl get daemonset -n kube-system
# All showed READY = DESIRED

# Clean up stuck terminating pods
kubectl delete pod -n kube-system --force --grace-period=0 <stuck-pod-names>

Recommended Add-on Versions for EKS 1.33

CoreDNS: v1.12.4-eksbuild.1
kube-proxy: v1.33.5-eksbuild.2
VPC CNI: v1.21.1-eksbuild.1
EBS CSI: v1.54.0-eksbuild.1

Key Takeaways

AL2023 is mandatory for EKS 1.33
Karpenter needs eks:DescribeCluster
kube-apiserver RBAC must be updated
Keep managed node groups as a safety net

Note
Looking back, the main issue wasn’t just missing permissions it was configuration drift.

While the cluster was still running EKS 1.32, I manually added eks:DescribeCluster during the AL2023 migration. Everything worked, so I forgot to codify it in Terraform.

During the upgrade to EKS 1.33, Terraform re-applied the IAM role and removed the permission right when Karpenter started requiring it.

The upgrade didn’t introduce the bug.

Environment: EKS, Terraform, Karpenter v1.x

Resources

How to Fix Karpenter Migration Issues During Upgrade (v0.25.0 v1.5.0)

Adedamola Ajibola — Thu, 04 Dec 2025 13:51:27 +0000

This blog post covers the real issues I ran into while upgrading Karpenter from v0.25.0 → v1.5.0 in production, why they happened, and the exact fixes. If you're planning this upgrade, this guide will save you hours of debugging.

Background

Upgrading Karpenter from 0.25.0 to 1.5.0 is not a simple version bump. It requires migrating from v1alpha5 APIs to the new v1 APIs a breaking change.

Starting point:

Karpenter v0.25.0
EKS 1.31
v1alpha5 CRDs (Provisioner, AWSNodeTemplate)

Target:

Karpenter v1.5.0
v1 CRDs (NodePool, EC2NodeClass)
EKS 1.32 compatibility

Skipping the CRD migration step leads to controller crashes, stuck resources, and broken uninstalls all of which I learned the hard way.

Problems I Faced During Migration

Problem 1: Chart not found in Helm repository

Error:

helm upgrade karpenter karpenter/karpenter --version 1.5.0
# Error: chart "karpenter" matching 1.5.0 not found

Why: The old Helm repo only contains versions up to 0.16.3. Karpenter v1.x was moved to an OCI registry.

Fix:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 --namespace karpenter --wait

Problem 2: OCI registry tag not found

Error:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter --version v1.5.0
# Error: ... v1.5.0: not found

Why: Starting in v0.35.0, OCI tags no longer use the v prefix.

Fix:

--version 1.5.0   # NOT v1.5.0

Problem 3: Controller crash on startup (missing CRDs)

Error:

# ERROR: no matches for kind "NodeClaim" in version "karpenter.sh/v1"
# panic: unable to retrieve the complete list of server APIs

Why: The v1.5.0 controller requires new v1 CRDs (NodePool, NodeClaim, EC2NodeClass). Your cluster still contains only v1alpha5 CRDs.

Fix: Install CRDs first

helm upgrade --install karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version 1.5.0 \
  --namespace karpenter --create-namespace

Then upgrade the controller:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 --namespace karpenter --wait

Most common migration failure CRDs must be upgraded first.

Problem 4: IAM permission denied

Error:

"error": "not authorized to perform: ec2:DescribeImages"

Why: Karpenter v1 introduces new instance profile and AMI discovery workflows.

Fix: Add this to the Karpenter controller IAM role:

{
  "Effect": "Allow",
  "Action": [
    "ec2:DescribeImages",
    "iam:GetInstanceProfile",
    "iam:CreateInstanceProfile",
    "iam:DeleteInstanceProfile",
    "iam:AddRoleToInstanceProfile",
    "iam:RemoveRoleFromInstanceProfile"
  ],
  "Resource": "*"
}

Restart the deployment:

kubectl rollout restart deployment karpenter -n karpenter

Step-by-Step Migration

1. Backup existing resources

kubectl get provisioners -A -o yaml > provisioners-backup.yaml
kubectl get awsnodetemplates -A -o yaml > awsnodetemplates-backup.yaml

2. Install v1 CRDs

helm upgrade --install karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version 1.5.0 --namespace karpenter

3. Update IAM permissions

(Add the policy from Problem 4.)

4. Upgrade the controller

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 --namespace karpenter --wait

5. Convert v1alpha5 → v1 resources

Provisioner → NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

AWSNodeTemplate → EC2NodeClass

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  role: "karpenter-node-role-name"

6. Apply and verify

kubectl apply -f ec2nodeclass.yaml
kubectl apply -f nodepool.yaml

kubectl get ec2nodeclass
kubectl get nodepools

7. Migrate nodes

# Test provisioning
kubectl run test --image=nginx --requests=cpu=1,memory=1Gi

# Drain old nodes

8. Clean up old resources

kubectl delete provisioner default
kubectl delete awsnodetemplate default

Results

Karpenter v1.5.0 running smoothly
All nodes migrated to NodePools
Cluster ready for EKS 1.32
Zero downtime

Useful Resources

Monitor logs during upgrade:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f

W.TEC’S EARLY INNOVATORS CAMP: Where Creativity Meets Technology

Adedamola Ajibola — Thu, 11 Sep 2025 07:30:47 +0000

When I first signed up to volunteer at W.TEC’s Early Innovators Camp, I thought it would simply be a chance to give back. What I didn’t realize was how much it would give back to me in return. This camp isn’t just about summer activities. It’s about nurturing creativity, problem-solving skills, and confidence in technology for the next generation.

For two weeks, I had the privilege of stepping into a dual role as both facilitator and sponsor and it turned out to be one of the most meaningful experiences of my journey in tech so far.

GIVING BACK IN MY OWN WAY

Volunteering has always been important to me, but this year I wanted to take it a step further. I decided to sponsor two children to attend the camp, giving them the opportunity to be part of an environment they might not have accessed otherwise.

Watching them dive into activities, raise their hands eagerly during sessions, and share their excitement with their peers was incredibly rewarding. It reminded me that impact doesn’t always have to come from grand gestures sometimes, it’s the small steps that create ripples of change. Seeing those children grow in confidence over the two weeks reaffirmed for me that access to opportunities is often the difference between potential left dormant and potential unlocked.

HANDS-ON LEARNING AND MY ROLE

I also got the chance to anchor a few fun, hands-on sessions. Two of my Favorites were:

The Magic Tissue Paper Flower Experiment: Blending science and art, the activity encouraged kids to think about how everyday materials can transform into something unexpected, a simple reminder that science is everywhere if we pay attention.
The Balloon Rocket Experiment: A playful demonstration of Newton’s third law of motion. As balloons zoomed across the room, the children laughed, shouted, and chased after them, all while unknowingly grasping a key principle of physics.

The excitement on the children’s faces as the balloon rockets zoomed across the room was priceless. Their endless why and how questions reminded me why curiosity is the engine of innovation.

LEARNING FROM OTHERS

One of the most enriching parts of the camp was learning alongside the kids. I got to sit in on sessions led by my colleagues Nifesimi and Abraham, who taught robotics, remote-controlled cars, and art drawing. Watching them break down complex concepts into digestible, engaging lessons was inspiring.

Even outside structured sessions, learning continued through anime watch and discuss gatherings, spontaneous debates about which superhero had the best powers, and laughter-filled walks after lunch. The camp created a community where kids could explore ideas in a relaxed and open environment. These moments reminded me that some of the most powerful lessons aren’t formally taught they’re experienced and shared together.

THE W.TEC COMMUNITY

Behind the scenes, the camp was powered by a team of passionate volunteers: Abraham, Nifesimi, Joy, Debbie, and Stella, who poured their energy into creating a safe, fun, and inspiring space. Each person brought something unique: from patience in guiding the kids through challenges, to humour that kept the atmosphere light, to creativity that turned ordinary lessons into memorable experiences.

Working alongside them reminded me how much stronger we are when united by a shared mission. Together, we weren’t just facilitators; we were mentors, role models, cheerleaders, and sometimes even students ourselves. That spirit of collaboration made the camp feel less like work and more like a family.

A LASTING IMPACT

As the camp ended, I found myself feeling deeply nostalgic. I even sent a message to my fellow facilitators:

I’ve been feeling quite nostalgic since the program ended. I truly miss the kids already. It was such a meaningful two weeks.

And I meant every word. Volunteering at the camp was a privilege, but the real reward was seeing the spark of possibility light up in young minds. Witnessing their growth reaffirmed the importance of creating access to opportunities like this.

CLOSING THOUGHTS

W.TEC’s Early Innovators Camp proves that when you blend fun, technology, and mentorship, you don’t just create summer activities, you shape futures.

I left the camp with a heart full of gratitude, inspired by the children, my fellow facilitators, and the W.TEC team. And I’m already looking forward to being part of this journey again.

To the W.TEC community, the facilitators, and most importantly, the kids, thank you for making it the best two weeks ever.

Fixing “Invalid Credentials” in AWS SSM Fleet Manager RDP

Adedamola Ajibola — Sun, 31 Aug 2025 06:29:21 +0000

When granting RDP access to a Windows EC2 instance, it’s tempting to open port 3389 to the world 0.0.0.0/0. That’s a major security risk. Instead, AWS SSM Fleet Manager lets you connect over a secure channel without exposing RDP to the internet.

Recently, I ran into an issue where Fleet Manager failed with this error:

Unable to establish Remote Desktop connection. Verify that valid credentials were provided, and that the user you specified has been granted permission to log in through Remote Desktop

Root Cause

On the Windows Server 2022 Base AMI, the default Administrator account was present, but its password had already expired. Since RDP connections including those tunneled through SSM Fleet Manager require a valid and active password, the expired credentials caused the login failure.

The Fix

Reset the Administrator password via SSM Run Command:

net user Administrator "xxxxx28xx@xxxx!73"
net localgroup "Remote Desktop Users" Administrator /add

Then log in through Fleet Manager RDP with your username and the new password.

Best Practices

Never expose RDP 3389 to 0.0.0.0/0
Use SSM Fleet Manager for secure access
Enforce strong passwords and rotate them regularly
Ensure EC2 has the IAM role: AmazonSSMManagedInstanceCore

Takeaway
If Fleet Manager RDP shows Invalid credentials, it’s usually not an SSM issue but a Windows password problem. Just reset the password through SSM and you’re good to go.

References

Demystifying AWS Security: IAM Password Policies vs. Automated Access Key Rotation

Adedamola Ajibola — Mon, 10 Jun 2024 12:42:39 +0000

Are you new to managing security in your AWS environment? Navigating the intricacies of AWS Identity and Access Management (IAM) can be overwhelming, especially when it comes to ensuring strong security practices. In this beginner-friendly blog post, we'll explore two fundamental aspects of AWS security: IAM password policies and automatically rotating IAM access keys using a Lambda function.

IAM Password Policy: Strengthening Your Authentication

Let's start with IAM password policies. These policies define the rules and requirements for user passwords within your AWS account. By enforcing strong password policies, you can significantly enhance the security of your AWS environment. Here's what you need to know:

Complexity Requirements: IAM password policies allow you to specify complexity requirements such as minimum length, the inclusion of special characters, and the prohibition of common passwords.

Password Expiry: You can set password expiry periods to ensure that users regularly update their passwords. This helps mitigate the risk of compromised credentials.

Preventing Password Reuse: IAM password policies can also prevent users from reusing previous passwords, further bolstering security.

By configuring a robust IAM password policy, you establish a strong foundation for authentication security within your AWS account.

Automatically Rotating IAM Access Keys: Enhancing Key Security

In addition to strong password policies, it's essential to regularly rotate IAM access keys. Access keys are used to authenticate programmatic access to AWS services, and regularly rotating them helps mitigate the risk of unauthorized access. Here's how you can automate this process using a Lambda function:

Lambda Function: AWS Lambda allows you to run code in response to various triggers. By creating a custom Lambda function, you can automate the rotation of IAM access keys.

Key Rotation Logic: The Lambda function checks the age of existing access keys associated with IAM users. If a key exceeds a specified age threshold, the function generates a new access key and deactivates the old one.

Scheduled Execution: You can schedule the Lambda function to run regularly, ensuring that access keys are rotated at predefined intervals without manual intervention.

By automatically rotating IAM access keys, you maintain a higher level of security in your AWS environment and reduce the risk of unauthorized access due to compromised credentials.

Conclusion

IAM password policies and automated access key rotation are essential components of AWS security. By enforcing strong password policies and regularly rotating access keys, you significantly reduce the risk of security breaches and unauthorized access in your AWS environment.

Upgrading an EKS Cluster: A Step-by-Step Guide

Adedamola Ajibola — Sat, 18 May 2024 00:02:19 +0000

Upgrading an Amazon EKS (Elastic Kubernetes Service) cluster can seem daunting, especially in a production environment. However, with a well-defined strategy and the right tools, the process can be smooth and minimally disruptive. In this post, I'll walk you through the upgrade process from version 1.27 to 1.28 using Terraform, ensuring your EKS cluster remains functional and resilient throughout.

Why Upgrading EKS Matters

EKS clusters need to stay updated to leverage the latest features, security patches, and performance improvements. However, EKS-managed clusters can only be upgraded one minor version at a time, making a systematic approach essential.

The Upgrade Process

If you're managing your EKS cluster configuration with Terraform, it's essential to understand that EKS-managed clusters can only undergo upgrades one minor version at a time. The terraform-aws-eks module is specifically designed to handle upgrades in the correct order when changing the cluster version.

Here's a simplified overview of what we'll cover:

Upgrading the EKS Cluster Version
Updating EKS Add-ons
Upgrading EKS Managed Node Groups
Upgrade Other Resources (e.g., Karpenter)

Step 1: Upgrading the Control Plane

The control plane is the brain of your Kubernetes cluster, managing all the operations within the cluster. To upgrade:

Update the Control Plane: This is the first step in the upgrade process. The control plane version dictates the compatibility of the cluster's components.
Initiate the Upgrade: Use Terraform to apply the new configuration. This ensures that the control plane updates to the desired version without manual intervention.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = local.name
  cluster_version = "1.27" => "1.28"

  cluster_endpoint_public_access = true

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
  }

Step 2: Updating EKS Add-ons

Add-ons in EKS are additional features that enhance your cluster's capabilities, such as DNS management or monitoring tools. These add-ons are tightly coupled with the cluster's version.

Automatic Compatibility: When the control plane is updated, EKS automatically aligns the add-on versions with the new cluster version.
Verify Add-on Versions: Ensure that each add-on is updated and compatible with the new version of the control plane.

Step 3: Upgrading EKS Managed Node Groups

Node groups are the workers in your cluster, running your applications. These groups need to be in sync with the control plane.

Node Group Update Process: After the control plane is updated, node groups are updated to match the new version. This ensures that all nodes run the compatible Kubernetes version.
Minimize Disruption: EKS handles node group updates in a way that minimizes disruption. By default, it limits the number of unavailable nodes during the upgrade to 33%, ensuring that most of your applications remain operational.

Ensuring Minimal Disruption

Our upgrade approach prioritizes minimal disruption, making it suitable for production environments. By systematically updating the control plane, add-ons, and node groups, you ensure that your cluster remains functional and efficient throughout the upgrade process.

Upgrade Other Resources

If you are using additional tools like Karpenter, an open-source cluster autoscaler, you will need to upgrade these as well:

Verify the compatibility of these resources with the new EKS version.

Conclusion

Upgrading your EKS cluster is paramount to maintaining security, stability, and accessing the latest Kubernetes features. It's imperative to follow the recommended upgrade procedure to seamlessly transition from v1.27 to v1.28 while mitigating disruptions. Before implementing upgrades in production, thorough testing in a staging environment is essential. Leveraging the terraform-aws-eks module can streamline the upgrade process, ensuring efficiency and accuracy. Stay informed about Kubernetes releases and adhere to best practices to uphold the success of your AWS EKS deployment.

Solving Kubernetes CronJob Stuck on Pending with Pod Affinity

Adedamola Ajibola — Tue, 06 Feb 2024 07:00:53 +0000

Introduction

Running cron jobs on a Kubernetes cluster provides an automated way to perform periodic tasks. However, when your cluster is dynamically managed, and new nodes come into play, you might encounter an issue where your cron job pods get stuck in the Pending state. In such cases, applying pod affinity can be a powerful solution to ensure successful pod scheduling. This blog post delves into how to address this issue using pod affinity and the advantages it offers.

Setting Up a CronJob in a Dynamic Kubernetes Cluster

CronJob Configuration: Define a CronJob resource in your Kubernetes manifest, specifying the schedule and the container to run the job. Ensure you have your data and resources correctly configured.

Dynamic Node Scaling: In dynamic Kubernetes clusters, tools like Karpenter automatically provision new nodes based on resource demands. This scaling mechanism is efficient but can lead to pod scheduling issues.

The Problem

When a new node spins up close to the time of your cron job execution, the scheduler may place the job on the newly created node. This can lead to pods getting stuck in a Pending state because the resources required by the CronJob are not immediately available.

The Solution

To address this problem, you can leverage pod affinity rules.

Using Pod Affinity to Ensure Successful Scheduling: Pod affinity allows you to influence where your pods are scheduled, making it a valuable tool in managing the scheduling of your cron jobs in dynamic clusters.

Affinity Rules: Define pod affinity rules that indicate where the cron job pods should be scheduled. For example, you can specify that they should be scheduled on nodes running specific labels or running pods of a particular type.

Topology Constraints: You can also apply topology constraints, ensuring that your cron job pods are only scheduled on nodes that meet certain criteria. For instance, you can specify that they should run on nodes in a different availability zone to improve high availability.

Node Selection: Utilize node selectors and node affinity to further fine-tune the selection of nodes for your pods.

Advantages of Pod Affinity
By applying pod affinity in your Kubernetes cluster, you gain several advantages:

Predictable Scheduling: Pod affinity ensures that your cron job pods are scheduled in a way that aligns with your requirements, even in dynamic clusters. This leads to predictable and reliable execution of your tasks.

Resource Utilization: It optimizes resource utilization by avoiding overloading new nodes that might not have sufficient resources for your cron jobs.

Improved High Availability: Through topology constraints and affinity rules, you can enhance fault tolerance and high availability by spreading your pods across different nodes or zones.

Conclusion

In dynamic Kubernetes clusters where nodes can be spun up and down automatically, scheduling cron jobs without proper consideration can lead to pod stuck-in-pending issues. Applying pod affinity rules allows you to regain control over pod placement, ensuring your cron jobs run reliably and efficiently. This approach offers improved high availability, resource utilization, and predictable scheduling in your Kubernetes environment, ultimately enhancing the overall stability and performance of your workloads.

Kubernetes Affinity Basics

Adedamola Ajibola — Fri, 19 Jan 2024 07:23:59 +0000

In Kubernetes, affinity is like a set of rules that decide where your pods go. Think of pods as small units of your app. There are two types:

Node Affinity: Decides where to put pods.
Pod Affinity: Decides where to group pods together. These rules help you control how pods are placed based on specific conditions.

Node Affinity VS Pod Affinity

Node Affinity

RequiredDuringSchedulingIgnoredDuringExecution: Pods must follow these rules to be scheduled on a node. If the rules are not met, the pod won't be placed on that node. But once scheduled, the pod won't be moved even if the rules are broken.

PreferredDuringSchedulingIgnoredDuringExecution: These rules are preferred, but not mandatory. The scheduler will try to follow them, but if it can't, the pod might still be placed on a node that breaks the rules.

Here's an example of a node affinity rule that requires a pod to be scheduled on a node with a specific label. Think of node affinity like choosing where your pod should live.

Pod Affinity

RequiredDuringSchedulingIgnoredDuringExecution: Similar to node affinity, but it's about pods wanting to be on the same node as other pods. Rules must be satisfied for the pods to be scheduled together on a node.

PreferredDuringSchedulingIgnoredDuringExecution: Similar to preferred node affinity, but at the pod level.

Here's an illustration of a pod affinity guideline below

Conclusion

Affinity rules in Kubernetes use labels (key-value pairs) to guide where pods should be placed in the cluster. By setting these rules, you enhance efficiency and performance, essentially telling Kubernetes where you prefer your pods to be based on the labels you've defined. This optimization helps in better resource utilization, improves performance, and ensures that related pods share the same nodes for efficient communication. For more details, check out the official documentation. It's a powerful tool for managing workload placement in Kubernetes.

Deploying a Next.js Static Site on DigitalOcean's App Platform

Adedamola Ajibola — Wed, 29 Nov 2023 09:10:03 +0000

Wondering if deploying a Next.js static site on DigitalOcean can deliver a seamless experience? 😂

I was assigned the task of deploying a static site and initially, I assumed it would be as simple as deploying a React Native app. I diligently followed the steps outlined in the documentation.

After successfully completing the build process, I eagerly clicked on the URL, only to encounter a frustrating 404 error. It was then that I delved into the build logs to troubleshoot the underlying issue.

Identifying the Issue

The root cause of the problem lies in the absence of a clear definition for the output directory of the static site. As a result, it was searching for static files and, by default, resorting to the standard 404 error document provided by the App Platform.

Finding a Solution

To resolve this issue, head to the settings and make a modification in the appsec file by adding the line output_dir: /out. Don't forget to clear the build cache before initiating a redeployment.

The build process concluded successfully, and the static files were exported to the designated /out directory.

Managing URL Path Rewrites and Redirects

If your application involves URL path rewrites or redirects, consult the documentation for the specific steps. You'll either need to manually configure these routes or include them in the appsec file under the ingress rule section.

Handling Domain Management in the App Platform

For guidance on effectively managing domains while using the App Platform, refer to the documentation, which offers a clear and straightforward guide.

Conclusion

Deploying a Next.js static site on DigitalOcean's App Platform is easy when you have the right guidance. This guide is here to help you make the process smooth and enjoyable. Additionally, the GitHub integration automates the build and deployment process, making it even more convenient.

App platforms offer many benefits like simplified deployment, automatic scaling, managed infrastructure, security features, CI/CD support, isolation, monitoring, and more. They boost developer productivity and reduce costs. The choice of the right platform depends on your project's needs and your organization's goals.

Resolving "The Node Had Condition [Node Disk-Pressure]" & "The Node Was Low On Resource: Ephemeral-storage" In Kubernetes

Adedamola Ajibola — Mon, 30 Oct 2023 01:01:22 +0000

Kubernetes is a powerful platform for managing containerized workloads, but it can be challenging to ensure that your applications are running efficiently and without issue. One common issue that you may encounter is Node Disk-Pressure & Node Was Low On Resource, which occurs when a node in your Kubernetes cluster runs out of disk space. In this blog post, we'll explore how to identify and resolve Node Disk-Pressure issues in your Kubernetes environment.

Identifying The Issue

A pod was evicted because of DiskPressure issues, and upon inspecting the specific pod, two prevalent issues were observed:

Node conditions: [DiskPressure]
The node was low on resources: ephemeral-storage.

To identify the source of the Node Disk-Pressure issue, there are a couple of different methods you can use. One way is to use the Kubernetes Kubectl top command to see which processes are consuming the most disk space on your node. Another way is to SSH into the node and run the df -h command to check the disk usage on the node and identify which directories are consuming the most disk space.

Once you've identified the processes or directories that are causing the disk usage, you can determine whether they are essential to the system or whether they can be deleted or moved to another location. By removing unnecessary files and directories or moving them to a different disk or node, you can free up disk space and resolve the Node Disk-Pressure issue.

Resolving the Issue

To resolve the Node Disk-Pressure issue, you can take several steps, such as increasing the EBS volume size, deleting unused resources, implementing resource limits, implementing pod anti-affinity, or adding more nodes to your cluster.

In my case, I resolved the issue by deleting unused resources and increasing the EBS volume size.

One way to resolve the issue is to increase the EBS volume size. To do this, you can follow the steps outlined in the AWS documentation. Essentially, you will need to modify the volume size in the EC2 console or using the AWS CLI or Terraform, and then extend the file system on the node to utilize the additional space.
Another way to free up disk space and resolve the Node Disk-Pressure issue is to delete any unused resources on your node. This can include pods, images, or volumes that are no longer required. By removing these resources, you can free up additional disk space on the node and prevent future Disk Pressure issues.
Finally, if none of the above steps works, you may need to consider adding more nodes to your cluster to distribute the workload across a larger number of machines. This will reduce the likelihood of any one node experiencing Disk-Pressure issues.

Conclusion

Node Disk-Pressure can be a challenging issue to resolve, but by following the steps outlined in this blog post, you can ensure that your Kubernetes cluster is running efficiently and that your applications can run successfully without being evicted due to Disk-Pressure issues. It's important to monitor your Kubernetes environment and take proactive steps to prevent Disk-Pressure issues from occurring in the first place.

Reducing Data Transfer Costs with S3 Gateway Endpoint

Adedamola Ajibola — Mon, 22 May 2023 11:00:43 +0000

In today's world, data is king. Businesses across all industries rely on the storage, retrieval, and analysis of data to make informed decisions and remain competitive. However, with the explosion of big data, cloud storage has become a necessary tool for businesses to store and analyze large volumes of data. Amazon S3 is a popular cloud storage service used by millions of businesses worldwide due to its scalability, high durability, and reliability.

However, using S3 from within a VPC can incur significant data transfer costs. To access S3 from within a VPC, traffic must be routed over the internet and back, which can add up to high data transfer costs, especially for businesses with large volumes of data.

Fortunately, Amazon has a solution to this problem - the S3 Gateway Endpoint. In this blog post, we will explore what an S3 Gateway Endpoint is, how it works, and how it has helped reduce data transfer costs.

What is an S3 Gateway Endpoint?

An S3 Gateway Endpoint is a service that allows you to access S3 from within a VPC without incurring any data transfer costs. It provides a secure and private connection between your VPC and S3 over the AWS network, bypassing the public internet. This allows you to access your S3 buckets and objects directly from your VPC as if they were hosted within your VPC.

How Does An S3 Gateway Endpoint Work?

When you create an S3 Gateway Endpoint, AWS creates an endpoint in your VPC that maps to the S3 service. Traffic between the S3 Gateway Endpoint and S3 is routed through the AWS network, avoiding the public internet. This provides a secure and private connection between your VPC and S3, ensuring that your data is protected from unauthorized access.

To use the S3 Gateway Endpoint, you simply need to update your routing tables to direct traffic destined for S3 to the endpoint. This ensures that any traffic to S3 from within your VPC is sent over the AWS network to the S3 Gateway Endpoint and then on to the S3 service.

Note that VPC and S3 buckets need to be in the same AWS region.

For a visual demonstration and detailed setup instructions, I recommend watching this video tutorial.

How S3 Gateway Endpoint Helped Reduce Data Transfer Costs

Using S3 extensively to store and analyze data can incur significant data transfer costs. Routing traffic over the internet and back to access S3 buckets can add up to high data transfer costs. However, implementing an S3 Gateway Endpoint can significantly reduce data transfer costs.

By implementing an S3 Gateway Endpoint, it was possible to access S3 buckets and objects directly from within a VPC, without incurring any data transfer costs. This allowed us to store and access data more efficiently, reducing overall data transfer costs.

Conclusion

In conclusion, Amazon's S3 Gateway Endpoint is an excellent solution to the problem of data transfer costs when accessing S3 from within a VPC. It provides a secure and private connection between your VPC and S3, ensuring that your data is protected from unauthorized access, while also allowing you to access your S3 buckets and objects directly from within your VPC, without incurring any data transfer costs.

Software Documentation Part2

Adedamola Ajibola — Mon, 17 Oct 2022 05:24:07 +0000

This post will discuss the importance, limitations, and different types of software documentation created for different audiences.

Software Documentation Limitation

The majority of process documents are tailored to a specific moment or phase of the process. As a result, these documents become quickly out of date and obsolete. However, they should be kept in development because they may be useful in the future for similar tasks or maintenance.

Documentation IS Essential

It allows new users to quickly learn how to use the software, simplifies the product, and reduces support costs.
It aids in knowledge transfer to other developers who may wish to modify or maintain the software.
It assists in keeping track of all aspects of an application and improves software product quality.

Here are some examples of software documentation types created for various audiences:

End User Documentation: This refers to documentation written specifically for end users. It should explain in as few words as possible how the software can help users solve their problems. Many large customer-based products replace some parts of user documentation, such as tutorials and onboarding, with onboarding training. Furthermore, online delivery of user documentation is becoming increasingly popular. Technical writers must be more creative when creating user documentation for the web. The following sections should be included in online end-user documentation:

Video demonstrations
In-built assistance
Help Portals

Technical documentation: Entails the documentation of software source codes, algorithms, APIs, and so on. It is typically written for a technical audience, such as software developers, technicians, and maintenance engineers. Technical documentation in software engineering is an umbrella term for all written documents and materials dealing with the development of software products. All software development products, whether developed by a small team or a large corporation, necessitate some level of documentation.

Architecture Documentation: Specifies the high-level architecture of the software system under development. May describe the system's main components, their roles and functions, as well as the data and control flow between those components. The main architectural decisions are included in software architecture design documents. We do not recommend listing everything, but rather focusing on the most important and difficult ones. The following information sections make up an effective design and architecture document.

Requirements Documentation: Usually created at the start of a software development project. The goal is to clearly and precisely specify the expectations for the software being developed. During the analysis phase of the SDLC, the requirements for software development are created and documented. The requirements are a description of the functionality of a software application that is used throughout the software development process to explain how the software is supposed to work.

Summary

Documentation is an important part of the software development process because it makes it easier for users to use new software and also helps transfer knowledge to another developer who may want to modify the software.