Forem: Muhammad Ahmad Khan

Monitoring & Alerting for AWS EKS Using Grafana, Prometheus & Alertmanager with SNS Integration

Muhammad Ahmad Khan — Sun, 02 Feb 2025 20:53:27 +0000

In this blog post, we'll walk through setting up a robust monitoring and alerting system for a Kubernetes cluster on AWS EKS. We'll use the kube-prometheus-stack to deploy Prometheus and Grafana, configure an ALB Ingress for external access, and set up Amazon SNS as the receiver for Alertmanager to handle alerts.

Using Amazon SNS as the alert receiver is particularly useful when an organization does not use Slack or wants a centralized way to distribute alerts via email. While Alertmanager supports email notifications, setting up email as a direct receiver requires configuring an SMTP configuration resulting in administrative overhead. With SNS, you can subscribe multiple email addresses to a single topic without additional SMTP setup, making it a simpler and more scalable solution.

However, it's important to note that SNS has limitations when handling HTML content. Alerts sent via SNS will be received as plain text rather than formatted HTML, which may impact readability.

Prerequisites

Before we begin, ensure you have the following:

An AWS account with a running EKS cluster.
kubectl and helm installed on your local machine.
AWS CLI configured with the necessary credentials.
AWS Load Balancer Controller installed in the EKS cluster.
kubectl pointing to the correct cluster context.

Step 1: Install the `kube-prometheus-stack` Using Helm

The kube-prometheus-stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules that provide easy deployment and management of Prometheus and Grafana on Kubernetes.

1.1 Add the Prometheus Community Helm Repository

First, add the Prometheus Community Helm repository and update it:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

1.2 Deploy the `kube-prometheus-stack`

Create a prom-operator-values.yaml file to customize the deployment. For example, set up the Grafana admin password, some custom dashboards, grafana ingress and other configurations:

grafana:
  adminPassword: "your-secure-password"
  ### Provision grafana-dashboards-kubernetes ###
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'grafana-dashboards-kubernetes'
        orgId: 1
        folder: 'Kubernetes'
        type: file
        disableDeletion: true
        editable: true
        options:
          path: /var/lib/grafana/dashboards/grafana-dashboards-kubernetes
  dashboards:
    grafana-dashboards-kubernetes:
      k8s-system-api-server:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json
        token: ''
      k8s-system-coredns:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json
        token: ''
      k8s-views-global:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json
        token: ''
      k8s-views-namespaces:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json
        token: ''
      k8s-views-nodes:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json
        token: ''
      k8s-views-pods:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json
        token: ''
  ingress:
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/load-balancer-name: grafana-alb
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/ssl-redirect: '443'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<AWS_REGION>:<ACCOUNT_ID>:certificate/a0ff498e-62fd-4397-8d4a-626360465d32
    enabled: true
    hosts:
    - grafana-test.apps.xyz-company.com
    labels: {}
    path: /

Deploy the kube-prometheus-stack using Helm:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

This command installs Prometheus, Grafana, and Alertmanager in the monitoring namespace.

1.3 Access Prometheus and Alertmanager UIs

To access the Prometheus and Alertmanager web UIs, use kubectl port-forward:

Prometheus UI:

  kubectl port-forward svc/prometheus-operated 9090:9090 --namespace monitoring

Open your browser and go to http://localhost:9090.

Alertmanager UI:

  kubectl port-forward svc/alertmanager-operated 9093:9093 --namespace monitoring

Open your browser and go to http://localhost:9093.

To access the Grafana web UI:

Create a DNS entry mapping the domain define in ingress with created loadbalancer url.
Use admin as the username and the password defined in prom-operator-values.yaml to log in to Grafana.

Step 2: Set Up Amazon SNS for Alertmanager

To receive alerts via email or other channels, we'll configure Amazon SNS as the receiver for Alertmanager.

2.1 Create an SNS Topic

Create an SNS topic to receive alerts:

aws sns create-topic --name alertTopic

Note the ARN (Amazon Resource Name) from the output, as it will be needed in later steps.

2.2 Create an Email Subscription for SNS

Subscribe your email to the SNS topic to receive notifications:

aws sns subscribe \
    --topic-arn arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic \
    --protocol email \
    --notification-endpoint <your-email@example.com>

Check your email inbox for a confirmation message and click the link to activate the subscription.

2.3 Create a New IAM Role for EKS Node Group to Assume and Publish to SNS

2.3.1 Create a Trust Relationship

Create a trust relationship policy to allow the EKS node group role to assume the new role:

cat << EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/eks-node-group-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the IAM role:

aws iam create-role \
    --role-name alertmanager_role \
    --assume-role-policy-document file://trust-policy.json

2.3.2 Attach Permissions to Publish to SNS

Create a policy to allow the role to publish messages to the SNS topic:

cat << EOF > sns-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic"
    }
  ]
}
EOF

Attach the policy to the new role:

aws iam put-role-policy \
    --role-name alertmanager_role \
    --policy-name SNSPublishPolicy \
    --policy-document file://sns-policy.json

2.4 Create a Resource-Based Policy for SNS

Allow this new role named alertmanager_role to publish alerts to the SNS topic by attaching a resource-based policy:

cat << EOF > sns-resource-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role"
      },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic"
    }
  ]
}
EOF

Apply the policy to the SNS topic:

aws sns set-topic-attributes \
    --topic-arn arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic \
    --attribute-name Policy \
    --attribute-value file://sns-resource-policy.json

2.5 Update the IAM Role for the EKS Node Group

Allow the EKS node group to assume this new role named alertmanager_role by attaching a policy:

cat << EOF > eks-assume-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role"
    }
  ]
}
EOF

Attach the policy to the EKS node group role:

aws iam put-role-policy \
    --role-name eks-node-group-role \
    --policy-name assume_alertmanager_role_policy \
    --policy-document file://eks-assume-policy.json

Note:

your EKS node Group role might be different than mine e.g. eks-node-group-role

Step 3: Configure Alertmanager to Send Alerts to SNS

Update the alertmanager.yml configuration to send alerts to the SNS topic in same helm values file named as prom-operator-values.yaml:

alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job', 'alertname', 'priority']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'sns-receivers'
    receivers:
    - name: 'null'  # Add this to your config as well
    - name: sns-receivers
      sns_configs:
        - api_url: https://sns.<AWS_REGION>.amazonaws.com
          topic_arn: arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic
          sigv4:
            region: <AWS_REGION>
            role_arn: arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role
          subject: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]'
          message: |-
            {{ if gt (len .Alerts.Firing) 0 }}
            Alerts Firing:
            {{ template "__text_alert_list_markdown" .Alerts.Firing }}
            {{ end }}
            {{ if gt (len .Alerts.Resolved) 0 }}
            Alerts Resolved:
            {{ template "__text_alert_list_markdown" .Alerts.Resolved }}
            {{ end }}
          send_resolved: true  # Sends notification when alert is resolved

Apply the updated configuration:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

Step 4: Configure custom rules for custom alerts

Update the prom-operator-values.yaml to add:

additionalPrometheusRulesMap:
  custom-rules:
    groups:
    - name: customGroupA.rules
      rules:
      - alert: Custom-Alert Instance High CPU Utilization
        annotations:
          description: CPU usage had been over 75% for 5 minutes | Current usage is {{ $value | printf "%.2f" }}%
          summary: CPU usage is over 75% (instance {{ $labels.instance }})
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 75
        for: 5m
        labels:
          severity: critical
      - alert: Custom-Alert Instance High Memory Utilization
        annotations:
          description: Memory usage had been over 75% for 5 minutes | Current usage is {{ $value | printf "%.2f" }}%
          summary: Memory usage is over 75% (instance {{ $labels.instance }})
        expr: 100 - (sum by(instance) (node_memory_MemAvailable_bytes) / sum by(instance) (node_memory_MemTotal_bytes) * 100) > 75
        for: 5m
        labels:
          severity: critical

Apply the updated configuration:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

The complete helm values file can be found on GitHub here

Conclusion

In this blog post, we set up a comprehensive monitoring and alerting system for a Kubernetes cluster running on AWS EKS. We deployed Prometheus and Grafana using the kube-prometheus-stack, configured an ALB Ingress for external access to Grafana, and set up Amazon SNS as the receiver for Alertmanager to receive alerts.

With this setup, you can now monitor your Kubernetes cluster, visualize metrics using Grafana, and receive alerts via Alertmanager and SNS. This ensures that your cluster is both observable and resilient to issues.

Rapidly Deploy ECS Infrastructure on AWS with AWS CDK (TypeScript)

Muhammad Ahmad Khan — Sat, 25 Jan 2025 02:22:52 +0000

When building and deploying containerized applications on AWS, Amazon ECS (Elastic Container Service) provides a powerful and flexible solution. AWS CDK (Cloud Development Kit) simplifies the process of creating ECS infrastructure by providing a programmatic approach to define cloud resources in code. This blog post walks through how to rapidly deploy ECS infrastructure using the aws-cdk-lib/aws-ecs-patterns module, which abstracts complex tasks like load balancing, service management, and autoscaling.

We’ll cover how to create an ECS Fargate service, set up an Application Load Balancer (ALB), enable auto-scaling, and integrate AWS Secrets Manager for securely managing sensitive data, all with just a few lines of TypeScript code.

Prerequisites

Before getting started, make sure you have the following:

Node.js and npm installed.
AWS CLI configured with appropriate permissions to create resources. You can configure it with:

   aws configure

AWS CDK installed globally:

   npm install -g aws-cdk

Docker installed (optional but useful for container builds).

Setting Up the Project

To begin, create a new directory for your CDK project and initialize it with TypeScript:

mkdir ecs-fargate
cd ecs-fargate
cdk init app --language typescript

This will generate the basic project structure for an AWS CDK application. Next, install the necessary dependencies for ECS, VPC, ECR, and other related services:

npm install @aws-cdk/aws-ec2 @aws-cdk/aws-ecs @aws-cdk/aws-ecr @aws-cdk/aws-ecs-patterns @aws-cdk/aws-iam @aws-cdk/aws-elasticloadbalancingv2 @aws-cdk/aws-certificatemanager @aws-cdk/aws-secretsmanager

Now you're ready to start defining your infrastructure!

Using `aws-cdk-lib/aws-ecs-patterns`

The aws-cdk-lib/aws-ecs-patterns module simplifies ECS service setup by providing high-level abstractions for common ECS patterns. It includes constructs for easy configuration of services like Application Load Balanced Fargate services, which handle the complexity of setting up load balancers, auto-scaling, and other integrations.

For example, the ApplicationLoadBalancedFargateService construct automatically creates an ECS Fargate service with an Application Load Balancer (ALB) in front of it, managing much of the networking and routing for you.

Here’s a quick look at some key benefits of using the ApplicationLoadBalancedFargateService construct from the aws-cdk-lib/aws-ecs-patterns module:

Simplified Infrastructure Setup: You don't need to manually define an Application Load Balancer or its target group—everything is handled for you.
Built-in Auto-Scaling: Easily configure auto-scaling for your Fargate tasks based on CPU or memory usage.
HTTPS Setup: You can easily set up SSL certificates via ACM (AWS Certificate Manager) and configure the load balancer to use HTTPS.

For more details on other architectural patterns, you can refer to the official AWS CDK ECS Patterns Documentation.

Defining the ECS Fargate Infrastructure

Here’s an example that demonstrates how to quickly deploy ECS infrastructure with AWS CDK using TypeScript.

Example: ECS Fargate Service with Application Load Balancer

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecr from 'aws-cdk-lib/aws-ecr';
import * as ecs_patterns from 'aws-cdk-lib/aws-ecs-patterns';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';
import * as acm from 'aws-cdk-lib/aws-certificatemanager';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';

interface EcsFargateStackProps extends cdk.StackProps {
  vpcId: string; // VPC Id
  ecrRepositoryName: string; // ECR repository name
  acmCertificateArn: string; // ACM certificate ARN for HTTPS
  demoServiceSecretArn: string; // Secret ARN for demoService
}

export class EcsFargateStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: EcsFargateStackProps) {
    super(scope, id, props);

    // Lookup existing VPC using VPC ID from props
    const vpc = ec2.Vpc.fromLookup(this, 'EcsVpc', {
      vpcId: props.vpcId,
    });

    // Reference the existing ECR repository
    const ecrRepo = ecr.Repository.fromRepositoryName(this, 'ExistingEcrRepository', props.ecrRepositoryName);

    // ECS Cluster
    const cluster = new ecs.Cluster(this, 'EcsCluster', {
      clusterName: 'demo-service-EcsFargateCluster',
      vpc,
    });

    // Task Definition
    const taskDefinition = new ecs.FargateTaskDefinition(this, 'FargateTaskDef', {
      memoryLimitMiB: 512,
      cpu: 256,
    });

    // Reference the secret from AWS Secrets Manager
    const taskSecret = secretsmanager.Secret.fromSecretCompleteArn(this, 'FargateTaskSecret', props.demoServiceSecretArn);

    // Add a container to the task definition
    const container = taskDefinition.addContainer('AppContainer', {
      image: ecs.ContainerImage.fromEcrRepository(ecrRepo, 'latest'),
      logging: ecs.LogDriver.awsLogs({ streamPrefix: 'ecs-app' }),
      environment: {
        DEBUG: 'true', // Simple environment variable
      },
      secrets: {
        SECRET_KEY: ecs.Secret.fromSecretsManager(taskSecret, 'SECRET_KEY'), // Referencing secret for SECRET_KEY
      },
    });

    container.addPortMappings({
      containerPort: 5000, // Container runs on port 5000
    });

    // ACM Certificate for HTTPS
    const certificate = acm.Certificate.fromCertificateArn(this, 'AcmCertificate', props.acmCertificateArn);

    // Application Load Balanced Fargate Service
    const fargateService = new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'FargateService', {
      cluster,
      taskDefinition,
      enableExecuteCommand: true, // Enable execute command
      certificate,
      publicLoadBalancer: true,
      loadBalancerName: 'demo-service-alb',
      sslPolicy: elbv2.SslPolicy.RECOMMENDED,
      serviceName: 'demo-service-EcsFargateService',
      redirectHTTP: true,
      protocol: elbv2.ApplicationProtocol.HTTPS, // Default HTTPS
    });

    // Adjust target group port to match the container port
    fargateService.targetGroup.configureHealthCheck({
      path: '/',
      port: '5000', // Health check targets container port 5000
    });

    // Autoscaling
    const scaling = fargateService.service.autoScaleTaskCount({
      minCapacity: 1,
      maxCapacity: 3,
    });

    scaling.scaleOnCpuUtilization('CpuScaling', {
      targetUtilizationPercent: 80,
    });

    scaling.scaleOnMemoryUtilization('MemoryScaling', {
      targetUtilizationPercent: 80,
    });

    // Outputs
    new cdk.CfnOutput(this, 'ClusterName', {
      value: cluster.clusterName,
      description: 'ECS Cluster Name',
    });

    new cdk.CfnOutput(this, 'ServiceName', {
      value: fargateService.service.serviceName,
      description: 'ECS Service Name',
    });
  }
}

Breakdown of the Code:

VPC and ECS Cluster: We use ec2.Vpc.fromLookup() to reference an existing VPC based on the VPC ID passed in as a parameter, and create an ECS cluster within that VPC.
ECR Repository: The Docker image for the application is stored in an existing ECR repository. We reference this repository using ecr.Repository.fromRepositoryName().
Task Definition: We define an ECS Fargate task with specified memory and CPU limits, referencing a Docker image from ECR. We also integrate environment variables and sensitive data from AWS Secrets Manager.
Load Balancer and HTTPS: The ApplicationLoadBalancedFargateService construct from aws-cdk-lib/aws-ecs-patterns automatically provisions an ALB, configuring SSL termination using an ACM certificate for HTTPS.
Auto-Scaling: We configure auto-scaling for the Fargate service based on CPU and memory utilization, ensuring the service can dynamically adjust to demand.
Outputs: Finally, we output the ECS cluster name and service name, so they can be easily referenced.

Deploying the Stack

Step 1: Install Dependencies

Ensure that all dependencies are installed:

npm install

Step 2: Bootstrap the CDK Environment

Run the bootstrap command to prepare your AWS account for CDK:

cdk bootstrap --profile your-aws-profile

Step 3: Synthesize the Stack

Synthesize the CloudFormation template to verify the stack:

cdk synth --context vpcId=vpc-xxxxxxxx --context ecrRepositoryName=demo-service --context acmCertificateArn=arn:aws:acm:region:account-id:certificate/certificate-id --context demoServiceSecretArn=arn:aws:secretsmanager:region:account-id:secret:secret-id --profile your-aws-profile

Step 4: Deploy the Stack

Deploy the ECS Fargate stack to AWS:

cdk deploy --context vpcId=vpc-xxxxxxxx --context ecrRepositoryName=demo-service --context acmCertificateArn=arn:aws:acm:region:account-id:certificate/certificate-id --context demoServiceSecretArn=arn:aws:secretsmanager:region:account-id:secret:secret-id --profile your-aws-profile

Step 4: Verify the Deployment

After the stack is successfully deployed, AWS will automatically provision the ECS service, ALB, and other necessary resources. You can access the service via the load balancer’s public endpoint.

Conclusion

In this guide, we’ve rapidly provisioned ECS infrastructure using AWS CDK and TypeScript. By leveraging the ApplicationLoadBalancedFargateService construct from aws-cdk-lib/aws-ecs-patterns, we were able to simplify the creation of a fully managed ECS service that includes an ALB, auto-scaling, and HTTPS support.

This approach significantly reduces the complexity of deploying containerized applications on AWS, allowing you to focus on developing your application instead of managing infrastructure.

For more detailed information, check out the official AWS CDK ECS Patterns Documentation.

GitHub Repository

You can find the complete code for this deployment in my GitHub repository.

Integrating Multiple EKS Clusters with ArgoCD for Simplifying Kubernetes Operations

Muhammad Ahmad Khan — Tue, 26 Mar 2024 19:49:52 +0000

In the rapidly evolving landscape of cloud-native technologies, Kubernetes has emerged as the de facto standard for container orchestration. As organizations scale their infrastructure, managing multiple Kubernetes clusters becomes inevitable. With this growth comes the challenge of ensuring consistency, reliability, and efficiency across all clusters. Enter ArgoCD, a powerful tool for continuous delivery and GitOps workflows in Kubernetes. In this blog post, we'll explore why integrating multiple clusters in ArgoCD is essential and how we can integrate multiple AWS EKS clusters in ArgoCD.

Why Integration is Required

Centralized Management: Managing multiple Kubernetes clusters manually can be daunting and error-prone. Integrating them with ArgoCD provides a centralized platform for managing and deploying applications across all clusters, streamlining operations and reducing complexity.
Consistency and Standardization: Different clusters may have varying configurations, making it difficult to maintain consistency in deployments. ArgoCD ensures that configurations and deployments are standardized across all clusters, promoting best practices and ensuring uniformity.
Scalability: As organizations grow, they often adopt a multi-cluster strategy to distribute workloads and improve fault tolerance. Integrating these clusters with ArgoCD enables seamless scaling of applications across clusters, allowing organizations to leverage resources efficiently.
Monitoring [Health Checks and Logging]: ArgoCD offers insights into the deployment status and health of applications across clusters through its user interface and API. By integrating multiple clusters, organizations gain centralized visibility and can monitor the status of applications across all clusters from a single dashboard.

How to Integrate Multiple AWS EKS Clusters in ArgoCD

Let's say we have AWS accounts as follows:

Account A with account id: 111111111111
Account B with account id: 222222222222
Account C with account id: 333333333333

Account A is where ArgoCD runs.

To authenticate and access the external cluster we need to add the configuration as follows:

In Account A:

Create an IAM role named argocd-manager.
Create a role policy named argocd-role-policy and attach it to a role named argocd-manager having the assume role policy given below

RolePolicyDocument

cat >argocd-role-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "*"
        }
    ]
}
EOF

AssumeRolePolicyDocument

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": [ "system:serviceaccount:argocd:argocd-server", "system:serviceaccount:argocd:argocd-application-controller" ]
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Now In Account B:

Create an IAM role named deployer having trust relationship as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::11111111111:role/argocd-manager"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Map this role in aws auth-config configmap Kubernetes object in Account B EKS cluster

kubectl edit -n kube-system configmap/aws-auth


# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::222222222222:role/my-role
      username: system:node:{{EC2PrivateDNSName}}
    - groups:
      - system:masters
      rolearn: arn:aws:iam::222222222222:role/deployer # deployer role arn
      username: deployer
  mapUsers: |
    - groups:
      - system:masters
      userarn: arn:aws:iam::222222222222:user/admin
      username: admin
    - groups:
      - system:masters      
      userarn: arn:aws:iam::222222222222:user/alpha-user
      username: my-user

Follow the same procedure in Account C as we have followed in Account B.

In Account A (where argocd is installed), add the following configuration in argocd helm chart values

Note: IAM role deployer must be created first in Account B or C

arn:aws:iam::222222222222:role/deployer
arn:aws:iam::333333333333:role/deployer

global:
  securityContext: # Set deployments securityContext/fsGroup to 999 so that the user of the docker image can use IAM Authenticator. We need this because the IAM Authenticator will try to mount a secret on /var/run/secrets/eks.amazonaws.com/serviceaccount/token. If the correct fsGroup (999 corresponds to the argocd user) isn’t set, this will fail.
    runAsGroup: 999
    fsGroup: 999

controller:
  serviceAccount:
    create: true
    name: argocd-application-controller
    annotations: {eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/argocd-manager} # Account A - IAM role service account
    automountServiceAccountToken: true

server:
  serviceAccount:
    create: true
    name: argocd-server
    annotations: {eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/argocd-manager} # Account A - IAM role service account
    automountServiceAccountToken: true

 configs:
  # -- Provide one or multiple [external cluster credentials]
  # @default -- `[]` (See [values.yaml])
  ## Ref:
  ## - https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
  ## - https://argo-cd.readthedocs.io/en/stable/operator-manual/security/#external-cluster-credentials
  ## - https://argo-cd.readthedocs.io/en/stable/user-guide/projects/#project-scoped-repositories-and-clusters
  clusterCredentials:
    - name: development
      server: https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.abc.region.eks.amazonaws.com # EKS cluster API server endpoint of Account B
      config:
        awsAuthConfig:
          clusterName: eks-development
          roleARN: arn:aws:iam::222222222222:role/deployer # Deployer role arn of Account B
        tlsClientConfig:
          # Base64 encoded PEM-encoded bytes (typically read from a client certificate file).
          caData: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx........==" # EKS cluster certificate authority
    - name: staging
      server: https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.abc.region.eks.amazonaws.com # EKS cluster API server endpoint of Account C
      config:
        awsAuthConfig:
          clusterName: eks-staging
          roleARN: arn:aws:iam::333333333333:role/deployer # Deployer role arn of Account C
        tlsClientConfig:
          # Base64 encoded PEM-encoded bytes (typically read from a client certificate file).
          caData: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx........==" # EKS cluster certificate authority

Obtain the EKS certificate of the respective cluster using AWS CLI:

aws eks describe-cluster \
        --region=${AWS_DEFAULT_REGION} \
        --name=${CLUSTER_NAME} \
        --output=text \
        --query 'cluster.{certificateAuthorityData: certificateAuthority.data}' | base64 -D

The important thing to note is that we need to set deployments securityContext/fsGroup to 999 so that the user of the docker image can use IAM Authenticator. We need this because the IAM Authenticator will try to mount a secret on /var/run/secrets/eks.amazonaws.com/serviceaccount/token. If the correct fsGroup (999 corresponds to the argocd user) isn’t set, this will fail.

Upgrading EKS Version with Zero Downtime Incorporating Custom Blue-Green Methodology

Muhammad Ahmad Khan — Sat, 23 Mar 2024 21:22:21 +0000

One of the most significant operational overheads of EKS is its version upgrade, which we have to tackle each quarter. Although AWS has announced extended support for older versions of EKS, the cost of extended support is something you do not want to incur. AWS maintains its standard support and extended support timeline in the following documentation: Kubernetes Release Calendar.

Keeping EKS clusters up to date with the latest versions is essential for security, performance, and access to new features. However, upgrading Kubernetes clusters can be challenging, especially in production environments where downtime is not an option. In this blog post, we'll explore how you can leverage the custom blue-green methodology to perform version upgrades on AWS EKS clusters seamlessly and with minimal disruption.

The blue-green deployment strategy is a technique used to reduce downtime and risk when deploying updates to applications or infrastructure. In the context of AWS EKS version upgrades, the traditional blue-green methodology involves creating a new cluster with the updated Kubernetes version (green), migrating workloads from the old cluster (blue) to the new one, and then decommissioning the old cluster. However, in our case, we will maintain a blue cluster only to serve traffic when we plan to upgrade our original cluster (green). This means that our original cluster will always remain green (before and after the EKS upgrade), and traffic will only shift from the green to the blue cluster while we perform upgrades on the original cluster. After the successful upgrade, traffic will shift back from the blue cluster to the original cluster (green).

Steps to Perform EKS Version Upgrade using Custom Blue-Green Methodology:

Prepare for Upgrade: Before initiating the upgrade process, thoroughly review the release notes for the new Kubernetes version to understand any changes or potential compatibility issues with your workloads. This includes checking deprecated resources and APIs, whether in your self-maintained application chart or in some open-source community chart. If you are using an open-source community chart, please ensure that the EKS version you are upgrading to has been tested with your installed chart.
Create a New Blue Cluster or Upgrade the Already Created Blue Cluster: Using AWS EKS, provision a new Kubernetes cluster with the desired version. Ensure that the cluster configuration, including node groups, networking, and IAM roles, aligns with your requirements. If you have created a new EKS cluster, make sure to configure DNS and Load Balancers by updating DNS records or adjusting load balancer settings to direct traffic to the new cluster gradually when workloads are migrated. If your cluster is large and you have many Kubernetes Ops tools installed in Kubernetes using Helm charts, it is a good idea to maintain this separate cluster for each upgrade cycle to save time. If you are already maintaining the cluster, then you have to prepare this cluster for upgrade by performing the following tasks:
- Upgrade the EKS control plane from the console.
- Upgrade each EKS Node group by setting the right update config as per the priority from the console and using Force Update.
- Upgrade add-ons from the console.
Deploy/Update Workloads: Utilize Kubernetes tools such as kubectl or Helm charts to deploy workloads onto the new blue cluster or update in the existing blue cluster. Monitor performance and functionality to ensure a smooth transition.
Validate and Test: Conduct thorough testing on this new/upgraded blue cluster to verify that all workloads and applications function as expected. Use automated testing tools and perform manual checks to validate performance, scalability, and reliability.
Cut Over Traffic: Once validation is complete, gradually shift production traffic from the old cluster to the new upgraded one using weighted routing. Monitor logs, pod scaling, and AWS Load Balancer metrics such as RequestCount as traffic increases. Monitor closely for any anomalies or performance issues during the transition.
Upgrade Original Cluster: Now perform the same process for the original cluster, including upgrading the EKS control plane, node groups, add-ons, as well as Kubernetes workloads. Verify if everything is working fine and transition traffic gradually back from the blue cluster to this original one after the full upgrade.
Decommission/Scale Down Blue Cluster: After confirming that the original cluster is handling all traffic effectively, decommission/scale down all node groups of the blue cluster to release resources and reduce operational overhead.

Best Practices for EKS Version Upgrades:

Automate wherever possible to streamline the upgrade process and reduce manual errors.
Implement monitoring and alerting to detect and respond to any issues promptly.
Maintain clear communication with stakeholders throughout the upgrade process to manage expectations and address concerns proactively.

AWS EKS provides a robust platform for managing Kubernetes clusters, and staying current with the latest versions is crucial for leveraging new features and ensuring security and performance. By adopting the blue-green methodology for version upgrades, organizations can minimize downtime, reduce risks, and maintain a seamless experience for end-users. By following best practices and leveraging automation and monitoring capabilities, you can confidently navigate EKS version upgrades and keep your Kubernetes infrastructure up to date with minimal disruption.

References: Kubernetes Deprecation Guide.

Distributing restricted static content through CloudFront using signed cookies.

Muhammad Ahmad Khan — Mon, 21 Nov 2022 14:47:55 +0000

Many times it's a business requirement that the content you share over the internet through your application including documents, business data, or media streams is intended to be consumed by selected users, for example, users who have paid a fee. Suppose that you have a similar situation and you are using an AWS CloudFront distribution (CDN) to distribute the static content including documents, images, and videos. But you want a restriction that this content is accessible through your app only to a set of selected users. Then we can use signed URLs and signed cookies.
CloudFront signed URLs and signed cookies provide the same basic functionality. They allow you to control who can access your content but both are beneficial in entirely different situations based on different requirements.
E.g.

Using signed URLs is beneficial when you want to restrict access to individual files, for example, an installation download for your application.
Using signed cookies is beneficial when you want to provide access to multiple restricted files, and don't want to change your current URLs.

In this blog, we will discuss about using signed cookies to grant access to your private content to the relevant user using CloudFront.
Signed URLs take precedence over signed cookies if you use both signed URLs and signed cookies to control access to your content.

How signed cookies works?
Here is an overview of how signed cookies work.

In your CloudFront distribution, specify one or more trusted key groups, which contain the public keys that CloudFront can use to verify the URL signature. You use the corresponding private keys to sign the URLs. Since you use private keys to sign the URLs you must store them somewhere and we will store them in AWS Secrets Manager. To create a KeyPair and to add a signer (trusted key group) in CloudFront distribution see the procedure here.
You develop your application to determine whether a user should have access to your content and if so, to send three Set-Cookie headers to the viewer. (Each Set-Cookie header can contain only one name-value pair, and a CloudFront signed cookie requires three name-value pairs.) You must send the Set-Cookie headers to the viewer before the viewer requests your private content. Typically, your CloudFront distribution will have at least two cache behaviors, one that doesn't require authentication and one that does. The error page for the secure portion of the site includes a redirector or a link to a login page. If you configure your distribution to cache files based on cookies, CloudFront doesn't cache separate files based on the attributes in signed cookies.
A user signs in to your website and either pays for content or meets some other requirement for access.
Your application returns the Set-Cookie headers in the response, and the viewer stores the name-value pairs.
The user requests a file. The user's browser or other viewer gets the name-value pairs from step 4 and adds them to the request in a Cookie header. This is the signed cookie.
CloudFront uses the public key to validate the signature in the signed cookie and to confirm that the cookie hasn't been tampered with. If the signature is invalid, the request is rejected. If the signature in the cookie is valid, CloudFront looks at the policy statement in the cookie (or constructs one if you're using a canned policy) to confirm that the request is still valid. If the request meets the requirements in the policy statement, CloudFront serves your content as it does for content that isn't restricted: it determines whether the file is already in the edge cache, forwards the request to the origin if necessary, and returns the file to the user.

There are two kinds of policies used for signed cookies

Canned policies
Custom policies

When you create a signed cookie, you write a policy statement in JSON format that specifies the restrictions on the signed cookie, for example, how long the cookie is valid. You can use canned policies or custom policies.
Canned policies allow you to specify an expiration date only. Custom policies allow more complex restrictions.

Canned policy example:

{
    "Statement": [
        {
            "Resource": "base URL or stream name",
            "Condition": {
                "DateLessThan": {
                    "AWS:EpochTime": ending date and time in Unix time format and UTC
                }
            }
        }
    ]
}

Custom policy example:

{
    "Statement": [
        {
            "Resource": "URL of the file",
            "Condition": {
                "DateLessThan": {
                    "AWS:EpochTime":required ending date and time in Unix time format and UTC
                },
                "DateGreaterThan": {
                    "AWS:EpochTime":optional beginning date and time in Unix time format and UTC
                },
                "IpAddress": {
                    "AWS:SourceIp": "optional IP address"
                }
            }
        }
    ]
}

Since we want to give access to all of the files located on a certain path of our CloudFront URL using wildcard * therefore we will use a custom policy.
For example, we want to give access to all of the files located on this URL: https://content.mysite.com/private-content/*
Here is the simple code in PHP for creating signed cookies using custom policy.

<?php

require 'vendor/autoload.php';

use Aws\Credentials\Credentials;
use Aws\SecretsManager\SecretsManagerClient;
use Aws\CloudFront\CloudFrontClient;
use Aws\Exception\AwsException;

// Check if user is login or check if user is paid the fee


// Initialize AWS credentials from IAM role
$credentials_uri = getenv('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI');
$credentials_url = 'http://169.254.170.2' . $credentials_uri;
$get_credentials = file_get_contents($credentials_url);
$credentials_array = json_decode($get_credentials, true);
$credentials = new Credentials($credentials_array["AccessKeyId"], $credentials_array["SecretAccessKey"], $credentials_array["Token"], strtotime($credentials_array["Expiration"]));

// Create a Secrets Manager Client
$secretManagerClient = new SecretsManagerClient([
    'credentials' => $credentials,
    'version' => '2017-10-17',
    'region' => 'us-east-1',
]);

$secretName = 'private-content-access-key';

try {
    $secretResult = $secretManagerClient->getSecretValue([
        'SecretId' => $secretName,
    ]);

}
catch (AwsException $e) {
    // output error message if fails
    echo $e->getAwsErrorMessage();
    echo "\n";
}


function signCookie($cloudFrontClient, $policy, $privateKey, $keyPairId)
{
    try {
        $result = $cloudFrontClient->getSignedCookie([
            'policy' => $policy,
            'private_key' => $privateKey,
            'key_pair_id' => $keyPairId
        ]);

        return $result;

    }
    catch (AwsException $e) {
        return ['Error' => $e->getAwsErrorMessage()];
    }
}

function signACookie($privateKey)
{
    $resourcePath = 'https://content.mysite.com/private-content/*';
    $expires = time() + 3600; // 60 minutes (60 * 60 seconds) from now.
    $keyPairId = 'ABCDWXYZ';
    $policy =
    '{'.
        '"Statement":['.
            '{'.
                '"Resource":"'. $resourcePath . '",'.
                '"Condition":{'.
                    '"DateLessThan":{"AWS:EpochTime":' . $expires . '}'.
                '}'.
            '}'.
        ']' .
    '}';

    $cloudFrontClient = new CloudFrontClient([
        'credentials' => $credentials,
        'version' => '2014-11-06',
        'region' => 'us-east-1'
    ]);

    $result = signCookie(
        $cloudFrontClient,
        $policy,
        $privateKey,
        $keyPairId
    );

    $cookies = array();

    foreach ($result as $key => $value) {
        array_push(
            $cookies,
            array(
            'name' => $key,
            'value' => $value,
            'expires' => $expires,
            'path' => '/',
            'domain' => ".mysite.com",
            'secure' => true,
            'httpOnly' => true
        )
        );

    }

    foreach ($cookies as $cookie) {
        setcookie
            (
            $cookie['name'],
            $cookie['value'],
            $cookie['expires'],
            $cookie['path'],
            $cookie['domain'],
            $cookie['secure'],
            $cookie['httpOnly']
        );
    }
}

if (isset($secretResult['SecretString'])) {
    $privateKey = $secretResult['SecretString'];
    signACookie($privateKey);
}

// Give access to the content e.g https://content.mysite.com/private-content/doc.pdf

You can easily write similar code in Node.js (Javascript).
The most important function to consider from the CloudFront client is getSignedCookie() which takes three parameters: policy, private key, and key pair id (this is not key group Id).
This function returns a response with the following values.

CloudFront-Policy=base64 encoded version of the policy statement;
CloudFront-Signature=hashed and signed version of the policy statement;
CloudFront-Key-Pair-Id=public key ID for the CloudFront public key whose corresponding private key you're using to generate the signature;

Finally, we will set the cookies by using values returned in the response.

SQS depth based ECS task auto-scaling using step scaling.

Muhammad Ahmad Khan — Sun, 20 Nov 2022 21:20:10 +0000

As mentioned in a previous blog post here, we can easily apply scaling using step scaling on SQS queue depth.

Step scaling policies increase or decrease the current capacity of a scalable target based on a set of scaling adjustments, known as step adjustments. The adjustments vary based on the size of the alarm breach. All alarms that are breached are evaluated by Application Auto Scaling as it receives the alarm messages.

With step scaling, you choose scaling metrics and threshold values for the CloudWatch alarms that trigger the scaling process and thus it requires you to create CloudWatch alarms.

When you create a step scaling policy, you add one or more step adjustments that enable you to scale based on the size of the alarm breach. Each step adjustment specifies the following:

A lower bound for the metric value
An upper bound for the metric value
The amount by which to scale, based on the scaling adjustment type

Application Auto Scaling supports the following adjustment types for step scaling policies:

ChangeInCapacity—Increase or decrease the current capacity of the scalable target by the specified value. A positive value increases the capacity and a negative value decreases the capacity. For example: If the current capacity is 3 and the adjustment is 5, then Application Auto Scaling adds 5 to the capacity for a total of 8.

ExactCapacity—Change the current capacity of the scalable target to the specified value. Specify a positive value with this adjustment type. For example: If the current capacity is 3 and the adjustment is 5, then Application Auto Scaling changes the capacity to 5.

PercentChangeInCapacity—Increase or decrease the current capacity of the scalable target by the specified percentage. A positive value increases the capacity and a negative value decreases the capacity. For example: If the current capacity is 10 and the adjustment is 10 percent, then Application Auto Scaling adds 1 to the capacity for a total of 11.

Below is the CloudFormation template with the implementation of step scaling with SQS.

AWSTemplateFormatVersion: 2010-09-09
Description: Creates a fargate based auto-scaling environment that processes work from an SQS queue
Parameters:
  DockerImageUrl:
    Type: String
    Default: latest

  DockerContainerName:
    Type: String
    Default: consumer-service

  EnvironmentName:
    Type: String
    Default: dev

  Memory:
    Type: String
    Default: 8GB

  Cpu:
    Type: Number
    Default: 2048 # 2 vCPU

  ContainerPort:
    Type: Number
    Default: 3000

  HealthCheckPath:
    Type: String
    Default: http://localhost:3000/check

  FaragateScalingEnvSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/FARGATE_SCALING_ENV"

  QueueDepthScaleOutAlarmThresholdSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD"

  CpuUtilizationScaleInAlarmThresholdSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD"

  CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS"

  ComputeAutoScalingTargetMaxCapacitySSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY"

Conditions:
  CreateNonProdResources: !Equals [!Ref FaragateScalingEnvSSM, 'non-prod']
  CreateProdResources: !Equals [!Ref FaragateScalingEnvSSM, 'prod']

Resources:
  SQSQueue:
    Type: 'AWS::SQS::Queue'
    # Properties:
    #   ReceiveMessageWaitTimeSeconds: 20
    #   VisibilityTimeout: 1200 # 20 minutes
    #   MessageRetentionPeriod: 1209600 # 14 Days

  QueueUrlParameter:
    Type: 'AWS::SSM::Parameter'
    Properties:
      Name: !Join
        - ''
        - - /
          - !Ref EnvironmentName
          - /services/
          - !Ref DockerContainerName
          - /SQS_QUEUE_URL
      Type: String
      Value: !Ref SQSQueue

  ComputeTaskLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Join
        - /
        - - /x-org
          - ecs
          - !Sub '${AWS::StackName}'
          - logs

  ComputeTaskRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: Required_Access
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'sqs:*'
                  - 'secretsmanager:*'
                  - 'ssm:*'
                  - 'logs:*'
                  - 'dynamodb:*'
                  - 's3:*'
                  - 'ecs:*'
                Resource: '*'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'

  ComputeTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    DependsOn: ComputeTaskLogGroup
    Properties:
      TaskRoleArn: !GetAtt ComputeTaskRole.Arn
      ExecutionRoleArn: !GetAtt ComputeTaskRole.Arn
      RequiresCompatibilities:
        - FARGATE
      NetworkMode: awsvpc
      Cpu: !Ref Cpu
      Memory: !Ref Memory
      ContainerDefinitions:
        - Name: !Sub '${AWS::StackName}'
          Image: !Ref DockerImageUrl
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: us-east-1
              awslogs-group: !Ref ComputeTaskLogGroup
              awslogs-stream-prefix: ecs
          HealthCheck:
            Command:
              - CMD-SHELL
              - !Sub 'curl -f ${HealthCheckPath} || exit 1'
            Interval: 30
            Retries: 3
            StartPeriod: 300
          PortMappings:
            - ContainerPort: !Ref ContainerPort
              Protocol: tcp
          Environment:
            - Name: EnvironmentName
              Value: !Ref EnvironmentName
            - Name: SQS_QUEUE_URL
              Value: !Ref SQSQueue

  ComputeCluster:
    Type: 'AWS::ECS::Cluster'
    # Properties:
    #   ClusterName: !Join ['-', [!Ref DockerContainerName, cluster]]

  NonProdComputeService:
    Type: 'AWS::ECS::Service'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      Cluster: !Ref ComputeCluster
      TaskDefinition: !Ref ComputeTaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      # Desired count should be 0; Otherwise the Task Scheduler will restart number of desired containers once they are stopped
      DesiredCount: 0
      # This may need to be adjusted if the container takes a while to start up
      # HealthCheckGracePeriodSeconds: 30
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          # Change it to DISABLED if you're using private subnets that have access to a NAT gateway
          AssignPublicIp: ENABLED
          Subnets:
            - !ImportValue ComputeSubnetA
            - !ImportValue ComputeSubnetB
            - !ImportValue ComputeSubnetC
          SecurityGroups:
            - !ImportValue ComputeSecurityGroup

  ProdComputeService:
    Type: 'AWS::ECS::Service'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      Cluster: !Ref ComputeCluster
      TaskDefinition: !Ref ComputeTaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      # This may need to be adjusted if the container takes a while to start up
      # HealthCheckGracePeriodSeconds: 30
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          # Change it to DISABLED if you're using private subnets that have access to a NAT gateway
          AssignPublicIp: ENABLED
          Subnets:
            - !ImportValue ComputeSubnetA
            - !ImportValue ComputeSubnetB
            - !ImportValue ComputeSubnetC
          SecurityGroups:
            - !ImportValue ComputeSecurityGroup

  ComputeAutoScalingRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
                - application-autoscaling.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: "/"
      Policies:
      - PolicyName: !Sub ${DockerContainerName}-ECSAutoScalingRole
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - ecs:UpdateService
            - ecs:DescribeServices
            - application-autoscaling:*
            - cloudwatch:DescribeAlarms
            - cloudwatch:GetMetricStatistics
            Resource: "*"
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole'

  NonProdComputeAutoScalingTarget:
    Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      MinCapacity: 0 # As desired task can be 0
      MaxCapacity: !Ref ComputeAutoScalingTargetMaxCapacitySSM
      ResourceId: !Join
          - '/'
          - - service
            - !Ref ComputeCluster
            - !GetAtt NonProdComputeService.Name
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      RoleARN: !GetAtt ComputeAutoScalingRole.Arn

  ProdComputeAutoScalingTarget:
    Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      MinCapacity: 1 # As desired task can be 1 but not 0
      MaxCapacity: !Ref ComputeAutoScalingTargetMaxCapacitySSM 
      ResourceId: !Join
          - '/'
          - - service
            - !Ref ComputeCluster
            - !GetAtt ProdComputeService.Name
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      RoleARN: !GetAtt ComputeAutoScalingRole.Arn

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdNoComputeAutoScalingPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdNoComputeAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ExactCapacity # Can use PercentChangeInCapacity but then need to come up with configuration including some estimated change in percent
        Cooldown: 60
        MetricAggregationType: Average # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average. 
        StepAdjustments: 
        - MetricIntervalLowerBound: !Ref AWS::NoValue
          MetricIntervalUpperBound: 0
          ScalingAdjustment: 0

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdInitialComputeAutoScalingPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdInitialComputeAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay 
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: 0
          MetricIntervalUpperBound: !Ref AWS::NoValue
          ScalingAdjustment: 1 # scaling up by 1 container when the alarm is greater than or equal to the Metric Threshold

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdComputeAutoScalingScaleOutPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdComputeAutoScalingScaleOutPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average. 
        StepAdjustments: 
        - MetricIntervalLowerBound: 0  # 0 means exactly equal to Metric Threshold which is 10 defined using SSM parameter
          MetricIntervalUpperBound: 15 # [Metrice Threshold + 15]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 15 # [Metrice Threshold + 15]
          MetricIntervalUpperBound: 25 # [Metrice Threshold + 25]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 25 # [Metrice Threshold + 25]
          MetricIntervalUpperBound: 35 # [Metrice Threshold + 35]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 35 # [Metrice Threshold + 35]
          MetricIntervalUpperBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  ProdComputeAutoScalingScaleInPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-ProdComputeAutoScalingScaleInPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ExactCapacity # Can use PercentChangeInCapacity but then need to come up with configuration including some estimated change in percent
        Cooldown: 60
        MetricAggregationType: Average # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: !Ref AWS::NoValue
          MetricIntervalUpperBound: 0
          ScalingAdjustment: 1

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  ProdComputeAutoScalingScaleOutPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-ProdComputeAutoScalingScaleOutPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: 0  # 0 means exactly equal to Metric Threshold which is 10 defined using SSM parameter
          MetricIntervalUpperBound: 15 # [Metrice Threshold + 15]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 15 # [Metrice Threshold + 15]
          MetricIntervalUpperBound: 25 # [Metrice Threshold + 25]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 25 # [Metrice Threshold + 25]
          MetricIntervalUpperBound: 35 # [Metrice Threshold + 35]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 35 # [Metrice Threshold + 35]
          MetricIntervalUpperBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1

# ######### https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html #########
#                                       (Total CPU units used by tasks in service) x 100
# Service CPU utilization =  ----------------------------------------------------------------------------
#                            (Total CPU units specified in task definition) x (number of tasks in service)

  NonProdCPUNoComputeAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdCPUNoComputeAlarm
      AlarmDescription: Alarm if container utilize low CPU based on specified threshold!
      Namespace: AWS/ECS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: CPUUtilization
      Dimensions:
        - Name: ServiceName
          Value:
            Fn::GetAtt:
            - NonProdComputeService
            - Name
        - Name: ClusterName
          Value:
            Ref: ComputeCluster
      Statistic: Average # Not using Sum since the metric is CPUUtilization
      Period: 60  # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: !Ref CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM # setting evaluation period 3 because when there is no task at all usually cpu starts with 0 (this way first evaluation period will already hit even before the task start doing anything) & 2 more as part of taking extra precautions! Change it to 2 if needed but not 1
      Threshold: !Ref CpuUtilizationScaleInAlarmThresholdSSM
      ComparisonOperator: LessThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdNoComputeAutoScalingPolicy

  NonProdInitialQueueDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdInitialQueueDepthAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 60 # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: 1 # Threshold is 1 for initial depth
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdInitialComputeAutoScalingPolicy

  NonProdQueueDepthScaleOutAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdQueueDepthScaleOutAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 120 # 120 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: !Ref QueueDepthScaleOutAlarmThresholdSSM # change this as needed
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdComputeAutoScalingScaleOutPolicy

  ProdCPUScaleInAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateProdResources # only create alarm if it is Prod env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-ProdCPUScaleInAlarm
      AlarmDescription: Alarm if container utilize low cpu based on specified threshold!
      Namespace: AWS/ECS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: CPUUtilization
      Dimensions:
        - Name: ServiceName
          Value:
            Fn::GetAtt:
            - ProdComputeService
            - Name
        - Name: ClusterName
          Value:
            Ref: ComputeCluster
      Statistic: Average # Not using Sum since the metric is CPUUtilization
      Period: 60  # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: !Ref CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM # setting this 3 as part of taking extra precautions! Change it to 1 if needed
      Threshold: !Ref CpuUtilizationScaleInAlarmThresholdSSM 
      ComparisonOperator: LessThanOrEqualToThreshold
      AlarmActions:
        - Ref: ProdComputeAutoScalingScaleInPolicy

  ProdQueueDepthScaleOutAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateProdResources # only create alarm if it is Prod env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-ProdQueueDepthScaleOutAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 120 # 120 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: !Ref QueueDepthScaleOutAlarmThresholdSSM # change this as needed
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: ProdComputeAutoScalingScaleOutPolicy

In the above template, I have created two sets of resources NonProd and Prod. In lower environments, the DesiredCount of ECS service is set as zero to save cost. Since CloudWatch Alarm takes at least one minute to respond to scaling events and in prod we do not want any delay that's why the DesiredCount is set as one in prod.
One more thing to note is that the scale-out is based on queue depth but scale-in is based on CPU utilization because the consumer-service consumes a message which can take several minutes to finish and I don't want Application Auto-Scaling to reduce tasks when the queue is empty and they are still processing something. In this regard, EC2 has something called
instance scale-in protection which allows you to have control over which queue workers are terminated when your Auto Scaling group scales in. It was not available for Fargate based ECS clusters but AWS just recently introduced a new feature for ECS called task scale-in protection. You can see the blog post here.

Here is the bash script for creating SSM parameters.

#!/usr/bin/env bash

while (($# > 1)); do
  case $1 in
  --profile) PROFILE="$2" ;;
  *) break ;;
  esac
  shift 2
done

echo "Deleting Parameters..."
aws ssm delete-parameter --profile $PROFILE --name "/$PROFILE/services/consumer-service/AWS_REGION"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/FARGATE_SCALING_ENV"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY"

echo "Creating parameters..."
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/'$PROFILE'/services/consumer-service/AWS_REGION", "Value": "us-east-1"}'
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/FARGATE_SCALING_ENV", "Value": "non-prod"}' # valid values are: non-prod or prod
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD", "Value": "10"}' # if you are increasing this then make sure you are also adjusting step scaling criteria
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD", "Value": "2"}' # 2% 
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS", "Value": "3"}' # do not set this as 1 in non-prod
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY", "Value": "6"}' # put 1 if you want to disable autoscaling at all

ECS auto-scaling based on Amazon MQ for RabbitMQ’s queue depth.

Muhammad Ahmad Khan — Tue, 01 Nov 2022 15:15:40 +0000

There is a very common use case in the industry about scaling the application based on queue depth. In this regard, AWS itself has published the documentation which can be found here.

In this blog post we will discuss about ECS auto-scaling based on queue depth through the use of AWS application auto-scaling.

There are three solutions you can use with AWS application auto-scaling.

Target Tracking (Policy)
Step Scaling (Policy)
Scheduled scaling (Action)

Here we will discuss about first two only.

Target Tracking Scaling Policies:

Target tracking scaling policy increases or decreases the number of tasks that your service runs based on a target value for a specific CloudWatch metric (predefined or customized).
Target tracking scaling policy assumes that it should scale out your ECS tasks when the specified metric is above the target value and scale in when it is below the target value.
In short, in target tracking scaling policies, Amazon ECS creates and manages the CloudWatch alarms that trigger the scaling policy and calculates the scaling adjustment based on the CloudWatch metric and the target value that you specify. With target tracking, AWS controls the scaling adjustments automatically based on your target specified.

Step Scaling Policies:

Step scaling policy increases or decreases the number of tasks that your service runs by considering the set of scaling adjustments, known as step adjustments, which means you can set multiple actions to vary the scaling depending on the size of the alarm breach.
In short, step scaling policies increase or decrease current capacity based on a set of scaling adjustments (known as step adjustments) that you specify. With step scaling, you control the scaling adjustments.

When working with both policies, it must be noted that you can easily scale your application based on queue depth using step scaling but not with target scaling because target scaling will only allow these three predefined metrics in ECS.

ALBRequestCountPerTarget (load balancer metric)
ECSServiceAverageCPUUtilization
ECSServiceAverageMemoryUtilization

Further, since in target scaling AWS itself create CloudWatch Alarm and manage its scaling therefore you do not have the autonomy to set your desired metric for scaling without creating a custom metric. Thus you need to make a custom metric as described in AWS documentation.

In order to monitor the number of unconsumed message with SQS we can use metric named as ApproximateNumberOfMessagesVisible and if we want to monitor messages from Amazon MQ for RabbitMQ we can use metric named as MessageReadyCount.
One more problem with using a CloudWatch SQS metric like ApproximateNumberOfMessagesVisible for target tracking is that the number of messages in the queue might not change proportionally to the number of tasks within a service that processes messages from the queue. That's because the number of messages in your SQS queue does not solely define the number of tasks needed. The number of tasks in your ECS service can be driven by multiple factors, including how long it takes to process a message and the acceptable amount of latency (queue delay).
AWS proposes a solution to this problem by introducing a backlog per instance metric with the target value being the acceptable backlog per instance.

In AWS documentation, it has stated that:
“Backlog per instance: To calculate your backlog per instance, start with the ApproximateNumberOfMessages queue attribute to determine the length of the SQS queue (number of messages available for retrieval from the queue). Divide that number by the fleet's running capacity, which for an Auto Scaling group is the number of instances in the InService state, to get the backlog per instance.
Acceptable backlog per instance: To calculate your target value, first determine what your application can accept in terms of latency. Then, take the acceptable latency value and divide it by the average time that an EC2 instance takes to process a message.”

In other words, backlog per instance is the value obtained by dividing the number of messages yet to be consumed within the queue with the total number of tasks running at the moment.
Similarly, acceptable backlog per instance is the target value that we use in our scaling policy, and it is the acceptable latency value divided by the average time that an ECS task takes to process a message.
This acceptable backlog per instance (target value) varies from application to application and use case to use case.

Example from AWS documentation:
“As an example, let's say that you currently have an Auto Scaling group with 10 instances and the number of visible messages in the queue (ApproximateNumberOfMessages) is 1500. If the average processing time is 0.1 seconds for each message and the longest acceptable latency is 10 seconds, then the acceptable backlog per instance is 10 / 0.1, which equals 100 messages. This means that 100 is the target value for your target tracking policy. When the backlog per instance reaches the target value, a scale-out event will happen. Because the backlog per instance is already 150 messages (1500 messages / 10 instances), your group scales out, and it scales out by five instances to maintain proportion to the target value.”

Following the AWS documentation, I have created a lambda that creates a custom metric in a custom namespace named as “Queue Based Scaling Metrics”. But in this scenario, our application is using AmazonMQ for RabbitMQ rather than SQS.

See the simple flow of architecture in the below diagram.

In our lambda, we will calculate all the values for our custom metric including backlog per instance as well as acceptable backlog per instance as mentioned in AWS documentation.
For this purpose, we need to set two parameter values (acceptable_latency, time_process_per_message) as mentioned above and these parameters will vary from application to application. In my scenario, I have set these values as follows:

acceptable latency = 250
time process per message = 0.8

Here is the lambda code for all the calculations:

src/app.py

import boto3
import dateutil
from datetime import date, timedelta, datetime

def lambda_handler(event, context):
    cw_client = boto3.client('cloudwatch')
    ecs_client = boto3.client('ecs')
    cluster_name = event["cluster_name"]
    service_name = event['service_name']
    mq_cluster_name = event["mq_cluster_name"]
    mq_queue_name = event["mq_queue_name"]
    acceptable_latency = (event["acceptable_latency"])
    time_process_per_message = (event["time_process_per_message"])
    queue_attribute_calculation(cw_client, ecs_client, cluster_name, service_name, mq_cluster_name, mq_queue_name, acceptable_latency,
                                time_process_per_message)


def queue_attribute_calculation(cw_client, ecs_client, cluster_name, service_name, mq_cluster_name, mq_queue_name, acceptable_latency,
                                time_process_per_message):
    response = ecs_client.describe_services(cluster=cluster_name, services=[service_name])
    running_task_count = response['services'][0]['runningCount']
    print("Running Task: " + str(running_task_count))

    yesterday=date.today() - timedelta(days=2)
    tomorrow=date.today() + timedelta(days=1)
    response = cw_client.get_metric_data(
        MetricDataQueries=[
            {
                'Id': 'mq1',
                'MetricStat': {
                    'Metric': {
                        'Namespace': 'AWS/AmazonMQ',
                        'MetricName': 'MessageReadyCount',
                        'Dimensions': [
                            {
                                'Name': 'Broker',
                                'Value': mq_cluster_name
                            },
                            {
                                'Name': "VirtualHost",
                                'Value': "/"
                            },
                            {
                                'Name': "Queue",
                                'Value': mq_queue_name
                            }
                        ]
                    },
                    'Period': 1,
                    'Stat': 'Sum',
                    'Unit': 'Count'
                },
            },
        ],
        StartTime=datetime(yesterday.year, yesterday.month, yesterday.day),
        EndTime=datetime(tomorrow.year, tomorrow.month, tomorrow.day)
    )

    queue_size =  response['MetricDataResults'][0]['Values'][0]

    print("Running Task: " + str(running_task_count))
    print("Queue Message Count (per second): " + str(queue_size))

    """
    Backlog Per Capacity Unit = Queue Size (MessageReadyCount) / Running Capacity of ECS Task Count
    """
    try:
        backlog_per_capacity_unit = int(int(queue_size) / running_task_count)
    except ZeroDivisionError as err:
        print('Handling run-time error:', err)
        backlog_per_capacity_unit = 0
    print("Backlog Per Capacity Unit: " + str(backlog_per_capacity_unit))

    """
    Acceptable backlog per capacity unit = Acceptable Message Processing Latency (seconds) / Average time to Process a Message each Task (seconds)
    """
    acceptablebacklogpercapacityunit = float((int(acceptable_latency) / float(time_process_per_message)))
    print("Acceptable backlog per capacity unit: " + str(acceptablebacklogpercapacityunit))

    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'MessageReadyCount', int(queue_size),
                  'Queue Based Scaling Metrics')
    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'BackLogPerCapacityUnit', backlog_per_capacity_unit,
                  'Queue Based Scaling Metrics')
    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'AcceptableBackLogPerCapacityUnit', acceptablebacklogpercapacityunit,
                  'Queue Based Scaling Metrics')


def putMetricToCW(cw, dimension_name, dimension_value, dimension_sub_attribute_name, dimension_sub_attribute_value, metric_name, metric_value, namespace):
    cw.put_metric_data(
        Namespace=namespace,
        MetricData=[
            {
                'MetricName': metric_name,
                'Dimensions': [
                    {
                        'Name': dimension_name,
                        'Value': dimension_value
                    },
                    {
                        'Name': dimension_sub_attribute_name,
                        'Value': dimension_sub_attribute_value
                    }
                ],
                'Timestamp': datetime.now(dateutil.tz.tzlocal()),
                'Value': metric_value
            }
        ]
    )

Note that in order to get AmazonMQ for RabbitMQ metric (MessageReadyCount) I have used CloudWatch client SDK call because I was unable to find any API call to get this metric directly from AmazonMQ for RabbitMQ broker client SDK.

I have deployed this lambda using SAM template and here is the solution for it.
template.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Stack which creates a lambda function that is invoked by a CloudWatch Event Rule every minute to update custom metric datapoints through aws sdk calls.

Parameters:
  Env:
    Type: String
    Description: Values can be dev, qa, devops, test, staging and prod respectively
  ClusterNameParameter:
    Type: String
  ServiceNameParameter:
    Type: String
  MqClusterNameParameter:
    Type: String
  MqQueueNameParameter:
    Type: String
  AcceptableLatencyParameter:
    Type: Number
  TimeProcessPerMessageParameter:
    Type: Number
  SecurityGroupIdParameter:
    Type: String
  SubnetIdAParameter:
    Type: String
  SubnetIdBParameter:
    Type: String
  SubnetIdCParameter:
    Type: String

Resources:
  CustomMetricRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - 'lambda.amazonaws.com'
                - 'events.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: 'LambdaBasicExecutionPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: LambdaExecutionPolicy
                Effect: Allow
                Action:
                - ec2:Describe*
                - ec2:CreateNetworkInterface
                - ec2:DeleteNetworkInterface
                - ec2:DescribeNetworkInterfaces
                - cloudwatch:GetMetricData
                - cloudwatch:PutMetricData
                - "ecs:*"
                Resource: '*'
              - Sid: ExtraPolicy
                Effect: Allow
                Action:
                - ssm:GetParameters
                - ssm:GetParameter
                Resource: '*'
        - PolicyName: 'LogPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'

  CustomMetricFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      FunctionName: !Sub create-custom-metric-for-scaling-${Env}-cluster # FunctionName uses Env variable to identify its belonging to the Env specific cluster
      Description: A function that is invoked by a CloudWatch Event every minute to update custom metric datapoints through aws sdk calls.
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.9
      Timeout: 900
      Role: !GetAtt CustomMetricRole.Arn
      Events:
        CloudWatchEventRule:
          Type: Schedule
          Properties:
            Input: !Sub '{"cluster_name": "${ClusterNameParameter}","service_name": "${ServiceNameParameter}","mq_cluster_name": "${MqClusterNameParameter}","mq_queue_name": "${MqQueueNameParameter}","acceptable_latency": "${AcceptableLatencyParameter}","time_process_per_message": "${TimeProcessPerMessageParameter}"}'
            # Read more about schedule expressions here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
            Schedule: cron(* * * * ? *) # Invoke function every minute
      VpcConfig:
        SecurityGroupIds:
          - !Ref SecurityGroupIdParameter
        SubnetIds:
          - !Ref SubnetIdAParameter
          - !Ref SubnetIdBParameter
          - !Ref SubnetIdCParameter

Here is the SAM template deployment command:
First, create the SAM template’s parameter file.

.sam-params-dev

Env=dev
ClusterNameParameter=dev-worker-cluster
ServiceNameParameter=dev-worker-service
MqClusterNameParameter=scan-dev
MqQueueNameParameter=scans_dev
AcceptableLatencyParameter=250
TimeProcessPerMessageParameter=0.8
SecurityGroupIdParameter=sg-123456789
SubnetIdAParameter=subnet-123456789
SubnetIdBParameter=subnet-123456789
SubnetIdCParameter=subnet-123456789

sam deploy --stack-name create-custom-metric --template-file template.yml --s3-bucket test-deployment-bucket --capabilities CAPABILITY_IAM --region us-east-1 --parameter-overrides $(cat .sam-params-dev)

Finally, we will use AWS CLI to put the auto-scaling policy with a target value on ECS.
First, create the JSON policy.

config-dev.json

{
    "TargetValue":312.5,
    "ScaleOutCooldown":1,
    "ScaleInCooldown":1,
    "CustomizedMetricSpecification":{
       "MetricName":"BackLogPerCapacityUnit",
       "Namespace":"Queue Based Scaling Metrics",
       "Dimensions":[
          {
             "Name":"AmazonMQ",
             "Value":"scan-dev"
          },
          {
            "Name":"Queue",
            "Value":"scans_dev"
         }
       ],
       "Statistic":"Average"
    }
 }

In above policy, the target value is Acceptable backlog per instance. This means it is:
acceptable latency/time process per message = 250/0.8 = 312.5 BackLogPerCapacityUnit is the custom metric that we have created through our lambda in Queue Based Scaling Metrics namespace.
scan-dev is the RabbitMQ cluster name and scans_dev is the queue name.

aws application-autoscaling register-scalable-target \ --service-namespace ecs \ --scalable-dimension ecs:service:DesiredCount \ --resource-id service/dev-worker-cluster/dev-worker-service \ --min-capacity 1 \ --max-capacity 10 \ --profile dev

aws application-autoscaling put-scaling-policy \ --policy-name ecs-scaling-based-on-message-consumption-rate \ --service-namespace ecs \ --resource-id service/dev-worker-cluster/dev-worker-service \ --scalable-dimension ecs:service:DesiredCount \ --policy-type TargetTrackingScaling \ --target-tracking-scaling-policy-configuration file://config-dev.json \ --profile dev

We can automate the creation of the application auto-scaling policy with Terraform and CloudFormation if required.
Note that we are just applying the autoscaling policy and AWS will create CloudWatch Alarm itself.

See the git repository here.

Forem: Muhammad Ahmad Khan

Monitoring & Alerting for AWS EKS Using Grafana, Prometheus & Alertmanager with SNS Integration

Prerequisites

Step 1: Install the kube-prometheus-stack Using Helm

1.1 Add the Prometheus Community Helm Repository

1.2 Deploy the kube-prometheus-stack

1.3 Access Prometheus and Alertmanager UIs

Step 2: Set Up Amazon SNS for Alertmanager

2.1 Create an SNS Topic

2.2 Create an Email Subscription for SNS

2.3 Create a New IAM Role for EKS Node Group to Assume and Publish to SNS

2.3.1 Create a Trust Relationship

2.3.2 Attach Permissions to Publish to SNS

2.4 Create a Resource-Based Policy for SNS

2.5 Update the IAM Role for the EKS Node Group

Step 3: Configure Alertmanager to Send Alerts to SNS

Step 4: Configure custom rules for custom alerts

Conclusion

Rapidly Deploy ECS Infrastructure on AWS with AWS CDK (TypeScript)

Prerequisites

Setting Up the Project

Using aws-cdk-lib/aws-ecs-patterns

Defining the ECS Fargate Infrastructure

Example: ECS Fargate Service with Application Load Balancer

Breakdown of the Code:

Deploying the Stack

Step 1: Install Dependencies

Step 2: Bootstrap the CDK Environment

Step 3: Synthesize the Stack

Step 4: Deploy the Stack

Step 4: Verify the Deployment

Conclusion

GitHub Repository

Integrating Multiple EKS Clusters with ArgoCD for Simplifying Kubernetes Operations

Why Integration is Required

How to Integrate Multiple AWS EKS Clusters in ArgoCD

Upgrading EKS Version with Zero Downtime Incorporating Custom Blue-Green Methodology

Distributing restricted static content through CloudFront using signed cookies.

SQS depth based ECS task auto-scaling using step scaling.

ECS auto-scaling based on Amazon MQ for RabbitMQ’s queue depth.

Target Tracking Scaling Policies:

Step Scaling Policies:

Step 1: Install the `kube-prometheus-stack` Using Helm

1.2 Deploy the `kube-prometheus-stack`

Using `aws-cdk-lib/aws-ecs-patterns`