Forem: Mohammed Kashif Rafi

KAAR AI-Powered Kubernetes Cluster Analysis and Remediation

Mohammed Kashif Rafi — Fri, 23 May 2025 03:12:45 +0000

Introducing KAAR AI-Powered Kubernetes Cluster Analysis and Remediation

KAAR (Kubernetes AI-powered Analysis and Remediation), a tool that automates Kubernetes Pod issue detection and resolution using k8sgpt and AWS Bedrock.

In this post, I’ll introduce KAAR, explain its value, and guide you on getting started with it.

The Kubernetes Pod Challenge

Pods are the core of Kubernetes, but they often encounter issues that halt your applications:

ImagePullBackOff: Pods fail to start due to invalid or inaccessible container images.
CrashLoopBackOff: Containers crash repeatedly from misconfigured commands or errors like “executable file not found.”
OOMKilled: Pods are terminated for exceeding memory limits.
Pending: Pods can’t schedule due to resource constraints or node affinity issues.

Manually debugging these with kubectl describe and logs is time-consuming, especially in large clusters.

KAAR automates this process, leveraging AI to resolve Pod issues quickly and reliably, saving valuable time for DevOps teams.

What is KAAR?

KAAR is an open-source tool designed to simplify Kubernetes Pod management. It integrates:

k8sgpt: A diagnostic tool that scans Kubernetes clusters for Pod issues.
AWS Bedrock: Uses the Claude v2 model to classify issues and recommend fixes.
AWS SNS and CloudWatch: Sends notifications and logs results for team visibility.
kubectl: Applies automated fixes to get Pods back on track.

KAAR is perfect for DevOps engineers, SREs, and architects who want to minimize manual intervention. It currently focuses on Pod remediation, with plans to support Services and Deployments in future releases.

How KAAR Works

KAAR follows a streamlined, AI-powered workflow to fix Pod issues:

Cluster Scanning: KAAR uses k8sgpt to analyze your cluster and identify Pod issues.
Issue Classification: Findings are processed by AWS Bedrock to determine issue type.
Remediation: Applies fixes via kubectl, like updating images or adjusting limits.
Verification: Ensures Pods are Running and healthy.
Notification: Logs results to CloudWatch and sends SNS notifications.

Real-World Examples

Scenario 1: CrashLoopBackOff

A Pod nginx-pod is stuck due to an invalid command. KAAR:

Detects the issue with k8sgpt.
Uses Bedrock to classify it as CrashLoopBackOff.
Updates the Pod’s command.
Verifies it is Running.
Sends an SNS alert: > “Pod nginx-pod in default is Healthy.”

Scenario 2: OOMKilled

A Pod memory-hog is terminated for low memory. KAAR:

Identifies the OOMKilled issue.
Increases the memory limit.
Confirms stability.
Sends an alert: > “Pod memory-hog in default is Healthy.”

Python 3.11 with boto3 and pyyaml:

Why Choose KAAR?

KAAR offers compelling benefits for Kubernetes admins:

Time Savings: Automates remediation.
Accuracy: AI-driven classification and suggestions.
AWS Integration: Works with Bedrock, SNS, CloudWatch.
Future-Ready: Coming support for Services and Deployments.

As an AWS Community Builder, I built KAAR to reflect AWS’s commitment to automation and innovation.

Getting Started with KAAR

Prerequisites

Ensure you have:

Kubernetes Cluster: AWS EKS, minikube, or kind.

   minikube start --driver=docker
   kubectl get nodes

AWS Credentials:

  aws configure

Python 3.11 with boto3 and pyyaml:

  python3.11 -m venv python311_env
  source python311_env/bin/activate
  pip install boto3 pyyaml

Install KAAR via pip:

 pip install kaar

Create my_config.yaml:

 aws:
  region: us-east-1
  sns_topic_arn: arn:aws:sns:us-east-1:YOUR_AWS_ACCOUNT_ID:KAARAlerts
  log_group: /kaar/notifications
  log_stream: kaar-notifier

k8sgpt:
  backend: amazonbedrock
  explain: true

bedrock:
  model: anthropic.claude-v2:1
  max_tokens: 50
  temperature: 0.5

remediation:
  max_attempts: 5
  retry_interval_seconds: 5

Future Plans

KAAR is poised to become a comprehensive Kubernetes management tool. Planned enhancements include:

Service Remediation: Support for issues like SelectorMismatch and LoadBalancerPending.
Deployment Support: Fixes for Deployment misconfigurations.
EventBridge Integration: Scheduled runs for proactive monitoring, inspired by my KAAR project.
Local LLMs: Support for Ollama as an alternative to Bedrock.

KAAAS: AI-Powered Kubernetes Cluster Analysis and Solutions

Mohammed Kashif Rafi — Mon, 19 May 2025 16:24:04 +0000

KAAAS: AI-Powered Kubernetes Cluster Analysis and Solutions

Kubernetes has revolutionized container orchestration, enabling organizations to manage complex, distributed applications at scale. However, with great power comes great complexity—troubleshooting Kubernetes clusters can be a daunting task, especially in production environments with thousands of resources.

Enter KAAAS (Kubernetes AI-powered Cluster Analysis and Solution), a cutting-edge tool designed to simplify Kubernetes management by harnessing the power of artificial intelligence (AI). In this blog post, we’ll explore what KAAAS is, how it works, and why it’s a game-changer for DevOps teams and Site Reliability Engineers (SREs) looking to maintain healthy Kubernetes clusters.

What is KAAAS?

KAAAS is an AI-driven framework that automates the analysis, diagnosis, and resolution of issues in Kubernetes clusters. Built with the goal of reducing the operational burden on teams, KAAAS integrates with Kubernetes environments to scan clusters, identify potential problems, and provide actionable solutions in a user-friendly format.

Unlike traditional monitoring tools that often overwhelm users with raw data, KAAAS leverages AI to interpret complex diagnostic information and present it in plain English, making it accessible to both seasoned Kubernetes experts and newcomers.

At its core, KAAAS combines the capabilities of tools like K8sGPT—a CNCF Sandbox project that uses AI to diagnose Kubernetes issues—with custom agents for cluster analysis, issue identification, and notification. By integrating with AI backends such as Amazon Bedrock or Ollama, KAAAS transforms raw cluster data into meaningful insights, helping teams maintain cluster health without getting bogged down in the intricacies of Kubernetes internals.

Why Do We Need AI in Kubernetes Management?

Kubernetes clusters are inherently complex, often spanning multiple nodes and managing thousands of resources like pods, services, and deployments. Common issues—such as pods stuck in a CrashLoopBackOff state, image pull failures, or resource misconfigurations—can be time-consuming to diagnose and fix manually.

Traditional tools like kubectl provide raw data, but interpreting logs, events, and metrics requires deep expertise and significant time investment.

AI-powered tools like KAAAS address these challenges by automating the diagnostic process. They can analyze vast amounts of cluster data, identify patterns, and pinpoint the root cause of issues faster than a human could. Moreover, AI tools can suggest solutions based on best practices and historical data, reducing the mean time to resolution (MTTR).

For organizations running mission-critical workloads on Kubernetes, this automation is invaluable—it minimizes downtime, improves reliability, and frees up engineering teams to focus on innovation rather than firefighting.

How Does KAAAS Work?

KAAAS operates through a series of modular agents, each responsible for a specific aspect of cluster analysis and management.

1. Cluster Scanning with K8sGPT

KAAAS uses K8sGPT to scan Kubernetes clusters for issues. K8sGPT, a powerful open-source tool, integrates with AI backends to analyze cluster resources like pods, deployments, and services. It identifies critical issues—such as a pod failing to schedule due to resource constraints or a service with no endpoints—and generates detailed explanations in plain English.

KAAAS builds on this capability by automating the scanning process and tailoring the analysis to specific cluster configurations.

2. Custom Agents for Analysis

KAAAS employs several custom agents to process the raw output from K8sGPT and other tools:

ClusterLimitAgent: Checks the number of clusters based on node count, ensuring compliance with licensing limits (e.g., KAAAS is free for one cluster).
K8sGPTAnalysisAgent: Runs K8sGPT scans and captures detailed output, including error messages and suggested fixes.
OutputParserAgent: Parses K8sGPT output to extract structured information about issues, such as error details, affected resources, and potential solutions.
IssueIdentificationAgent: Identifies the type of issue (e.g., ImagePullBackOff, CrashLoopBackOff) and provides context about the affected resource, including its namespace and name.

These agents work together to transform raw data into actionable insights, ensuring that users don’t have to sift through logs manually.

3. Notification and Logging

Once issues are identified, KAAAS uses AWS services like SNS (Simple Notification Service) and CloudWatch to notify users and log events.

The NotificationAgent sends alerts via SNS, summarizing the issues and their solutions in a concise format.
It also logs detailed information to CloudWatch, providing a historical record for auditing and further analysis.

This integration ensures that teams are promptly informed of critical issues and can take action without delay.

4. Integration with AWS Resources

KAAAS seamlessly integrates with AWS resources, leveraging tools like AWS Controllers for Kubernetes (ACK) to manage not only Kubernetes resources but also associated AWS services. For example, if a cluster issue is related to an AWS-managed resource (e.g., an RDS database or an S3 bucket), KAAAS can analyze and provide recommendations for those resources as well, offering a holistic view of the environment.

Benefits of Using KAAAS

KAAAS offers several advantages for teams managing Kubernetes clusters:

1. Simplified Troubleshooting

By automating the analysis and diagnosis of cluster issues, KAAAS eliminates the need for manual log diving. Its AI-driven approach provides clear, actionable insights, making it easier for teams to resolve issues quickly.

2. Proactive Issue Detection

KAAAS continuously monitors clusters, identifying potential problems before they escalate. For example, it can detect a pod in a Pending state due to resource constraints and suggest adjustments to resource limits or node scaling.

3. AI-Powered Automatic Analysis and Notifications:

KAAAS scans your cluster regularly, finds problems, suggests solutions using AI, and notifies the admin—keeping your environment healthy with minimal effort..

4. Enhanced Collaboration

The notification system in KAAAS fosters collaboration by keeping all team members informed of cluster health. Whether you’re a DevOps engineer, an SRE, or a developer, you’ll have access to the same insights, enabling faster decision-making.

High level flow

Prerequisites for Using KAAAS

Before you can use KAAAS, ensure you have the following prerequisites in place:

Kubernetes Cluster: A running Kubernetes cluster (self-managed, on a cloud provider like AWS EKS, or a local setup like Minikube).
kubectl: The Kubernetes command-line tool installed and configured to interact with your cluster.
K8sGPT: KAAAS relies on K8sGPT for cluster scanning. Install it using a package manager like brew install k8sgpt or download the binary from the K8sGPT GitHub repository.
Python 3.11+: KAAAS is a Python-based tool, and we recommend using Python 3.11 or above for optimal compatibility and performance.
pip: The Python package manager to install KAAAS and its dependencies.
AWS Credentials: Configure your AWS credentials using the AWS CLI (aws configure) or environment variables. Ensure you have permissions for SNS and CloudWatch.
K8sGPT Configuration: Configure K8sGPT with an AI backend (e.g., Ollama or Amazon Bedrock). For example, set up a k8sgpt.yaml file with your backend details, such as model name, base URL, and authentication tokens.
kubectl Access: Ensure kubectl is configured to access your cluster (kubectl config view to verify).

Getting Started with KAAAS

KAAAS is available on PyPI, making it easy to install and use. We recommend setting up a Python virtual environment to manage dependencies cleanly.

Step 1: Set Up a Python Virtual Environment

Create and activate a virtual environment using Python 3.11 or above:

python3.11 -m venv kaaas_env
source kaaas_env/bin/activate

Step 2: With the virtual environment activated, install KAAAS using pip:

pip install kaaas

This will install KAAAS and its dependencies, including boto3 for AWS integration and pyyaml for configuration parsing.

Step 3: Set Up AWS Credentials

Configure your AWS credentials to enable KAAAS to interact with SNS and CloudWatch:

aws configure

Provide your AWS Access Key ID, Secret Access Key, region (e.g., us-east-1), and output format.

Step 4: Configure KAAAS

Create a config.yaml file specifying your backend LLM, AWS region, SNS topic ARN, and CloudWatch log group/stream. Example:

backend_llm: ollama #Your AI backend
aws_region: us-east-1
sns_topic_arn: arn:aws:sns:us-east-1:xxxxxxxx:kaaasAlerts # your SNS arn
log_group: /kaaas/notifications
log_stream: kaas

Step 5: Run KAAAS

After installation, run KAAAS by providing the path to your configuration file:

kaaas --config config.yaml

KAAAS will scan your cluster, analyze issues using K8sGPT, and send notifications via SNS while logging details to CloudWatch.

Step 6: Automating KAAAS with a Cron Job

To ensure your Kubernetes cluster is regularly monitored and analyzed, you can automate KAAAS using a cron job. This helps maintain cluster health by running automated scans at scheduled intervals without manual intervention.

Here’s an example bash script to use in your cron job:

#!/bin/bash

# Set PATH explicitly
export PATH="/root/python311_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Activate virtual environment
source /root/python311_env/bin/activate

# Define config and log file paths
CONFIG_FILE="/root/KAAAS/config.yaml"
LOG_FILE="/root/KAAAS/kaaas-$(date +%F).log"

# Run the command
/root/python311_env/bin/kaaas --config "$CONFIG_FILE" >> "$LOG_FILE" 2>&1

# Deactivate virtual environment
deactivate

To run this script daily at 2 AM, add the following line to your crontab using crontab -e

crontab -e
chmod +x /root/KAAAS/run_kaaas_cron.sh

Make sure to give execute permission to your script:

chmod +x /root/KAAAS/run_kaaas_cron.sh

Automated Guide to VPC Lattice Setup and East-West Traffic Testing in Amazon EKS

Mohammed Kashif Rafi — Sun, 26 Jan 2025 05:33:08 +0000

Modern applications require seamless and secure service-to-service communication, especially when operating at scale in the cloud. Amazon Elastic Kubernetes Service (EKS) combined with AWS VPC Lattice provides an efficient way to implement scalable, east-west communication and streamline traffic management between services.

This script is a step-by-step guide to deploying the AWS Gateway API Controller for EKS using VPC Lattice. It simplifies the setup process by automating key configurations, such as IAM policies, service accounts, and Gateway API installations, ensuring a smooth integration with your Kubernetes environment.

By following this script, you'll:

Set up the AWS Gateway API Controller with Helm.
Deploy GatewayClass and Gateway configurations for traffic routing.
Establish and verify east-west connectivity using HTTPRoutes.
Test service-to-service communication with DNS-based routing.


!/bin/bash

# Define Variables (Modify these as per your setup)
CLUSTER_NAME="xxxxxxxxxxxxx" # Your cluster name
AWS_REGION="xxxxxx"  # Modify as per your region
AWS_ACCOUNT_ID="xxxxxxxxxxxxxxx" # Your account id
VPC_ID="vpc-xxxxxxxxxxxxxxxxx"  # Modify with your VPC ID

# Download the recommended inline policy for the controller installation
echo "Downloading recommended inline policy..."
curl https://raw.githubusercontent.com/aws/aws-application-networking-k8s/main/files/controller-installation/recommended-inline-policy.json -o recommended-inline-policy.json

# Create the IAM policy
echo "Creating IAM policy..."
aws iam create-policy \
    --policy-name VPCLatticeControllerIAMPolicy-eks-2 \
    --policy-document file://recommended-inline-policy.json

# Get the Policy ARN
VPCLatticeControllerIAMPolicyArn=$(aws iam list-policies --query 'Policies[?PolicyName==`VPCLatticeControllerIAMPolicy-eks-2`].Arn' --output text)

# Apply the controller installation manifest for the namespace
echo "Applying the controller installation manifest..."
kubectl apply -f https://raw.githubusercontent.com/aws/aws-application-networking-k8s/main/files/controller-installation/deploy-namesystem.yaml

# Enable OIDC provider for EKS
echo "Enabling OIDC provider..."
eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --approve --region $AWS_REGION

# Create IAM Service Account for the pod
echo "Creating IAM service account..."
eksctl create iamserviceaccount \
    --cluster=$CLUSTER_NAME \
    --namespace=aws-application-networking-system \
    --name=gateway-api-controller \
    --attach-policy-arn=$VPCLatticeControllerIAMPolicyArn \
    --override-existing-serviceaccounts \
    --region $AWS_REGION \
    --approve

# Login to AWS ECR Public and install the controller using Helm
echo "Login to AWS ECR Public and installing Helm chart..."
aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws

# Install the Gateway API Controller with Helm
helm install gateway-api-controller \
    oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart \
    --version=v1.0.6 \
    --set=serviceAccount.create=false \
    --namespace aws-application-networking-system \
    --set=awsRegion=$AWS_REGION \
    --set=clusterVpcId=$VPC_ID \
    --set=awsAccountId=$AWS_ACCOUNT_ID \
    --set=clusterName=$CLUSTER_NAME \
    --set=log.level=info

# Wait for the Pod to be in RUNNING state
echo "Waiting for the gateway-api-controller pod to be in RUNNING state..."
kubectl wait --namespace aws-application-networking-system \
  --for=condition=ready pod -l app.kubernetes.io/instance=gateway-api-controller \
  --timeout=300s  # Timeout after 5 minutes

# Wait for 3 minutes after Helm installation
echo "Waiting for 3 minutes after Helm installation..."
sleep 180  # Sleep for 180 seconds (3 minutes)

# Apply the GatewayClass manifest
echo "Creating GatewayClass..."
kubectl apply -f https://raw.githubusercontent.com/aws/aws-application-networking-k8s/main/files/controller-installation/gatewayclass.yaml

# Check if the Git repository exists, if not, clone it
GIT_REPO_DIR="aws-application-networking-k8s"  # Directory to check for Git clone
if [ ! -d "$GIT_REPO_DIR" ]; then
  echo "Cloning AWS Gateway API Controller repository..."
  git clone https://github.com/aws/aws-application-networking-k8s.git
else
  echo "Repository already cloned. Skipping git clone."
fi

# Navigate to the repository directory
cd $GIT_REPO_DIR

# Update the Helm chart for default service network
echo "Updating Helm chart for service network..."
aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws

helm upgrade gateway-api-controller \
    oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart \
    --version=v1.0.6 \
    --reuse-values \
    --namespace aws-application-networking-system \
    --set=defaultServiceNetwork=my-hotel

# Wait for 3 minutes after updating the Helm chart
echo "Waiting for 3 minutes after updating Helm chart..."
sleep 180  # Sleep for 180 seconds (3 minutes)

# Test Deployemnt and North-South Connectivity Testing

# Apply the "my-hotel" gateway configuration
echo "Creating the 'my-hotel' gateway..."
kubectl apply -f /prod-eks-2/test/aws-application-networking-k8s/files/examples/my-hotel-gateway.yaml

# Verify that the Gateway was created successfully
echo "Verifying Gateway creation..."
kubectl get gateway

# Apply the HTTPRoute configurations
echo "Applying HTTPRoute configurations..."
kubectl apply -f /aws-application-networking-k8s/files/examples/parking.yaml
kubectl apply -f /aws-application-networking-k8s/files/examples/review.yaml
kubectl apply -f /aws-application-networking-k8s/files/examples/rate-route-path.yaml

# Apply additional HTTPRoute for inventory
echo "Applying inventory HTTPRoute..."
kubectl apply -f /aws-application-networking-k8s/files/examples/inventory-ver1.yaml
kubectl apply -f /aws-application-networking-k8s/files/examples/inventory-route.yaml

# Wait for 3 minutes after applying inventory HTTPRoute
echo "Waiting for 3 minutes after applying inventory HTTPRoute..."
sleep 180  # Sleep for 180 seconds (3 minutes)

# Check DNS names of HTTPRoutes
echo "Fetching DNS names for HTTPRoutes..."
ratesFQDN=$(kubectl get httproute rates -o json | jq -r '.metadata.annotations."application-networking.k8s.aws/lattice-assigned-domain-name"')
inventoryFQDN=$(kubectl get httproute inventory -o json | jq -r '.metadata.annotations."application-networking.k8s.aws/lattice-assigned-domain-name"')

# Display DNS names
echo -e "Rates FQDN: $ratesFQDN\nInventory FQDN: $inventoryFQDN"

# Verify service-to-service communication from inventory to parking and review services
echo "Verifying service-to-service communication..."
kubectl exec deploy/inventory-ver1 -- curl -s $ratesFQDN/parking $ratesFQDN/review
kubectl exec deploy/parking -- curl -s $inventoryFQDN