Forem: Muhammad Ahmad Khan

From Static to Runtime: Choosing Between CloudFront + S3, Amplify, and ECS for Frontends on AWS

Muhammad Ahmad Khan — Sun, 26 Apr 2026 14:20:44 +0000

Most AWS frontend journeys begin with the same honest instinct: keep the first version simple. A build pipeline pushes static assets, S3 holds the files, CloudFront handles TLS and caching, and the team ships. For a genuinely static site, that decision is still excellent. It is cheap to operate, easy to explain, and fast all over the world.

The problem appears later. The frontend starts asking for behavior that is no longer static: deeper routing, request-time rendering, image or runtime optimizations, environment-specific configuration, or production controls that need to look like the rest of the platform. At that point, the hosting model is no longer a packaging detail. It becomes architecture.

The mistake is usually not starting with CloudFront + S3. The mistake is keeping a static-hosting model after the frontend begins to need request-time behavior, runtime configuration, or deployment controls that look like the rest of the production platform.
The real question: can this frontend be fully generated at build time and served as files, or does it now need server behavior? That single question usually points you to the right AWS model faster than any brand comparison does.

My practical framing is simple. Use CloudFront + S3 when the app is truly static. Use Amplify Hosting when you want a managed path to modern frontend behavior. Use ECS on Fargate when the frontend has crossed the line into being an application service.

Option 1 — CloudFront + S3

Why teams start here

CloudFront + S3 is still the cleanest answer for static frontends. That includes documentation portals, marketing sites, lightweight SPAs, and Next.js projects that can be exported as static output via the output: 'export' setting in next.config.js. Operationally, this model stays small: no app server fleet, no container runtime, and no load balancer in the middle.

It also scales in the most comfortable way. Most requests are served from the edge, the origin is just a bucket, and your deployment story is mostly about building assets and invalidating cache when needed.

Figure 1 — Build artifacts flow into S3; users hit CloudFront; the edge cache absorbs most traffic.

Figure 2 — S3 is only contacted when the edge cache misses.

Where the model starts to hurt

This is where many teams get tripped up. They do not make a bad initial choice; they simply keep a static model after the app stops behaving like a static site. Once request-time behavior enters the picture, CloudFront + S3 begins to accumulate workarounds.

Two details matter more than most teams expect. First, S3 website endpoints support only HTTP, not HTTPS. Second, if you use a website endpoint as the CloudFront origin so you can get website-style behavior such as automatic index document handling, CloudFront treats it as a custom origin, which means Origin Access Control (OAC) is not available.

Deep-link routing for SPAs also needs care. Since S3 only serves objects that actually exist, a direct browser request to a route such as /dashboard or /products/123 may not map to a real object in the bucket and may return a 403 or 404. For SPA-style routing, this is usually handled with CloudFront custom error responses that return /index.html for 403 and 404 responses, or with a CloudFront Function that rewrites clean URLs to the appropriate index.html path.

That works for static sites and SPAs, but it is still a routing workaround, not a runtime application model. If routes require request-time rendering, server-side data fetching, middleware, personalization, or framework runtime behavior, CloudFront + S3 starts to become a poor fit.

Option 2 — Amplify Hosting

The middle ground many teams should evaluate earlier

Amplify is the answer when CloudFront + S3 feels too static, but running containers for a frontend still feels unnecessary. It gives you Git-based deployments, a managed CDN, and managed SSR support without asking the team to own the underlying runtime in the same way ECS does.

That makes Amplify particularly attractive for modern frontend teams: product keeps a fast deploy loop, QA gets predictable branch-based delivery, and engineering can support SSR behavior without building a bespoke hosting stack first.

Figure 3 — Git push triggers a managed build; the CDN routes requests to either static assets or managed SSR compute.

Figure 4 — Static routes are served from cache; SSR routes are rendered on managed compute.

Why this is not just "CloudFront + S3 with nicer buttons"

Amplify is opinionated in a useful way. AWS documents managed SSR support for modern Next.js versions, automatic framework detection, and a deployment model that is closer to a frontend platform than a raw infrastructure stack. For many teams, that is exactly the point.

The tradeoff is that you are accepting the boundaries of a managed service. Server-side environment variables, for example, are not exposed automatically the way client-side ones are; they need to be set up explicitly via the Amplify console or build spec. And new framework features sometimes ship to self-hosted Next.js before they reach Amplify. If your goal is to minimize hosting decisions and keep the frontend delivery experience as product-friendly as possible, Amplify is often the cleanest middle path.

Option 3 — ECS on Fargate

When the frontend becomes part of the application platform

There is a moment when the right question stops being "How do we host this site?" and becomes "How do we run this service?" That is the handoff point to ECS on Fargate.

This model makes sense when the frontend needs runtime secrets, deliberate health checks, explicit rollout control via mechanisms like the ECS deployment circuit breaker, target-tracking autoscaling, private networking, and the same operational standards as the rest of your production estate. It is not the lowest-effort option. It is the highest-control option.

Figure 5 — The pipeline pushes images to ECR and updates the ECS service; the ALB routes user traffic to healthy tasks.

Figure 6 — Deployment phase rolls out new tasks behind health checks; steady-state requests reach only healthy targets.

The thing Next.js teams underestimate in containers

When teams move Next.js into containers, they usually think first about deployment. They should also think about cache behavior. Self-hosted Next.js is straightforward on a single instance, but multi-instance behavior is a design topic, not an implementation detail. Once multiple tasks can serve the same content, the Next.js incremental cache (used by ISR and the data cache) needs an explicit shared cache handler — a Redis or S3 backend, for example — instead of the per-task default. Otherwise you get inconsistent pages between replicas.

That is why ECS is best reserved for frontends that have clearly joined the application platform. Once you truly need runtime control, the extra operational responsibility is justified. Before that point, it is often more platform than the problem requires.

A simple decision matrix

Model	Best when	Biggest strength	Main tradeoff	Default advice
CloudFront + S3	The app is truly static or can be fully generated at build time.	Very low ops, very fast, globally cached.	Starts to fight you once request-time behavior enters the picture.	Start here for static sites.
Amplify Hosting	You want modern frontend behavior without running containers.	Managed Git-based delivery with built-in SSR support.	Opinionated platform; some Next.js features land later.	Best middle ground.
ECS on Fargate	The frontend now behaves like an application service.	Full runtime, rollout, and scaling control.	Highest operational responsibility.	Use when the frontend joins the platform.

What I would actually choose

If the frontend is still static, I would not move away from CloudFront + S3. It remains a strong design. If the team needs modern frontend behavior but wants to stay out of container operations, I would take Amplify seriously and early. If the frontend now needs to live under the same release, security, and observability standards as backend services, I would move it to ECS and treat it accordingly.

The best AWS frontend architecture is usually not the most powerful one. It is the one whose operating model still matches the app's actual behavior.

Amazon ECS Deployment Strategies: Legacy CodeDeploy and the Native ECS Controller

Muhammad Ahmad Khan — Sun, 26 Apr 2026 13:37:41 +0000

If you have worked with Amazon ECS long enough, you have probably heard two different stories that both sound true.

The first says that if you want blue/green on ECS, you use AWS CodeDeploy. The second says that Amazon ECS now supports rolling, blue/green, canary, and linear deployments directly. Both are true — they just belong to different generations of the ECS deployment model.

The real distinction is not the rollout names; it is the control plane. In the legacy path, CodeDeploy owns the blue/green workflow and ECS participates through task sets. In the newer path, the ECS deployment controller owns the service deployment directly and tracks service revisions inside ECS itself.

That shift matters because most real environments are mixed. Existing services may still depend on the older CodeDeploy pattern, while new services can often standardize on the native ECS strategies. If you lead platform or DevOps work, you need to be comfortable in both worlds.

The one distinction that clears up most confusion

Amazon ECS separates the deployment controller from the deployment strategy.

A deployment controller is the control plane that runs the release. In practice, that means ECS for native ECS-managed deployments, CODE_DEPLOY for CodeDeploy-managed ECS blue/green, and EXTERNAL for a third-party deployment controller.

A deployment strategy is the rollout shape used by that controller. With the ECS deployment controller, ECS supports ROLLING, BLUE_GREEN, CANARY, and LINEAR.

Older write-ups often equate "ECS blue/green" with "CodeDeploy," but current AWS docs also describe native ECS blue/green, canary, and linear. Advanced ECS traffic shifting used to be a CodeDeploy-only capability. It is now also an ECS-native one.

Quick comparison: what changes in the traffic layer?

The fastest way to explain the entire topic is through the traffic layer.

Built-in rolling usually keeps the same listener and target-group path, with ECS replacing tasks in place.

Legacy CodeDeploy blue/green introduces the dual-target-group model, a production listener, and an optional test listener, with CodeDeploy deciding how traffic moves between blue and green task sets.

Built-in ECS blue/green, canary, and linear also use a dual-environment traffic model for managed shifting, but ECS now owns the listener-rule changes and the rollout behavior.

Aspect	Native rolling	Legacy CodeDeploy blue/green	Native blue/green, canary, linear
Controller	ECS	CODE_DEPLOY	ECS
Target groups	Same target group; in-place	Two target groups (blue and green)	Primary + alternate (ALB)
Listener changes	Usually none	CodeDeploy updates production listener	ECS updates listener rules
Rollout owner	ECS scheduler	CodeDeploy	ECS deployment controller
Unit of deployment	Tasks within service	Task sets	Service revisions
Test environment	Not applicable	Optional test listener	Optional test listener / rule

Quick matrix — what changes in the traffic layer across built-in and legacy ECS deployment models.

Full comparison diagram

The same idea is clearer with the legacy and native models side by side.

Figure 1 — Legacy CodeDeploy controller vs native ECS controller. Both use a dual-target-group pattern; the difference is who owns the rollout.

Part 1: The legacy model — CodeDeploy blue/green for Amazon ECS

For a long time, if a team wanted something safer than a standard ECS rolling update, it usually meant stepping into CodeDeploy-controlled blue/green.

In that model, the ECS service uses deploymentController = CODE_DEPLOY, the deployment is started through CodeDeploy, CodeDeploy creates a replacement green task set, traffic is shifted away from the original blue task set, and after success, blue is terminated according to deployment settings.

From the load-balancer point of view, the pattern is deliberate and explicit. Legacy CodeDeploy ECS blue/green requires two target groups, a production listener, and an optional test listener. One target group serves blue and the other serves green during deployment.

Note that legacy ECS traffic shifting can be all-at-once, canary, or linear — but those are CodeDeploy deployment configurations, not separate ECS-native strategies.

Legacy diagram 1 — CodeDeploy blue/green, all-at-once

The fastest legacy cutover. It is still blue/green because CodeDeploy creates and validates a replacement task set, then moves production traffic in one step.

Figure 2 — Legacy CodeDeploy blue/green, all-at-once. A single 0% → 100% production shift after green is provisioned.

Legacy diagram 2 — CodeDeploy blue/green, canary

The legacy two-phase rollout. CodeDeploy shifts a small percentage first, evaluates health and alarms during the canary interval, and only then shifts the remainder.

Figure 3 — Legacy CodeDeploy blue/green, canary. ALB-based; uses a CodeDeployDefault.ECSCanary* deployment configuration.

Legacy diagram 3 — CodeDeploy blue/green, linear

The most gradual traffic-shift pattern in the legacy model. CodeDeploy increases traffic to green in equal steps, giving you more than one checkpoint before full cutover.

Figure 4 — Legacy CodeDeploy blue/green, linear. ALB-based; uses a CodeDeployDefault.ECSLinear* deployment configuration.

Important legacy caveat — when the legacy model uses a Network Load Balancer, AWS documents all-at-once as the supported traffic-shift mode. That is why legacy canary and linear are presented here as ALB-based examples.

Part 2: The modern model — native Amazon ECS service deployments

The modern ECS deployment story is cleaner because the ECS deployment controller now owns the deployment behavior directly.

Instead of relying on CodeDeploy to manage replacement task sets, the native model works with ECS service deployments and service revisions. That reduces control-plane sprawl and turns the strategy choice into a single ECS concern rather than a multi-service orchestration problem.

For managed traffic shifting, AWS documents support through Application Load Balancer, Network Load Balancer, or Service Connect, depending on the pattern. The diagrams below use Application Load Balancer because that makes the traffic behavior easiest to follow visually.

Native strategy 1 — Rolling

Rolling remains the simplest native option. ECS launches new tasks, stops old tasks as deployment progresses, and follows the deployment configuration controlled by minimumHealthyPercent and maximumPercent.

From the traffic-layer point of view, rolling changes the least. The service typically stays on the same target-group path and ECS replaces tasks in place. ECS can support multiple target groups for some service designs, but that is a service capability, not a blue/green traffic-shift requirement.

Figure 5 — Native ECS rolling deployment. Same target-group path; tasks are replaced in place under min-healthy / max-percent constraints.

Native strategy 2 — Blue/Green

Native ECS BLUE_GREEN is the closest built-in replacement for teams that used blue/green for safer cutovers and quick rollback.

The most important point: native ECS blue/green is not "blue/green with optional canary or linear." In the native model, BLUE_GREEN, CANARY, and LINEAR are separate strategies. Native blue/green performs an all-at-once production cutover after validation.

For ALB-based native blue/green, ECS uses a primary target group for blue, an alternate target group for green, a production listener rule for production traffic, and an optional test listener (or test rule) for validation traffic. ECS updates the listener rules during deployment.

Figure 6 — Native ECS blue/green. ECS owns the listener-rule swap. Primary TG = blue, alternate TG = green.

Native strategy 3 — Canary

Native ECS CANARY is the small-blast-radius strategy.

ECS launches the green service revision, can optionally validate it with test traffic, then shifts a configured small percentage of production traffic to green. ECS waits through the canary bake time, and if the deployment stays healthy, shifts the remaining traffic and waits through the deployment bake time before terminating blue.

The traffic model is straightforward: a small percentage first, the rest later.

Figure 7 — Native ECS canary. A single small slice, a bake window, then a full shift.

Native strategy 4 — Linear

Native ECS LINEAR is the most gradual rollout.

ECS launches green, optionally validates it, then shifts production traffic in equal percentage increments with a configured step bake time between increments. After reaching 100% green, ECS waits through the deployment bake time before terminating blue.

This fits services where you want more than one checkpoint and you expect some issues to appear only as traffic gradually increases.

Figure 8 — Native ECS linear. Equal-step shifts with a bake between each, then a deployment bake before teardown.

What actually changes between rolling and blue/green-style deployments?


Rolling	Usually the same listener and same target-group path. No separate green traffic lane required. The ECS scheduler handles in-place task replacement.
Legacy CodeDeploy blue/green	Two target groups required. A production listener is required. An optional test listener is available. CodeDeploy controls traffic movement between blue and green task sets.
Native ECS blue/green, canary, linear	Managed traffic shifting requires ALB, NLB, or Service Connect. ALB examples use primary and alternate target groups. ECS controls the listener-rule changes and traffic movement.
Key distinction	Canary and linear are NOT sub-modes of blue/green in native ECS. They are separate native strategies.

Summary of the practical differences across deployment models.

What I would choose in practice

If I were setting standards for an AWS platform team today, I would keep the defaults simple.

Use rolling when the service is low risk, backward compatibility is strong, and traffic shaping is unnecessary.

Use native blue/green when you want safer releases than rolling, you want validation before production cutover, and you do not need a gradual production ramp.

Use native canary when blast radius matters most and you want a small real-user slice before full cutover.

Use native linear when you want the smoothest progressive rollout and more checkpoints than canary provides.

Keep legacy CodeDeploy blue/green when services already run it successfully and migration would create more near-term risk than value.

Legacy CodeDeploy still matters because many production estates still run it, but the center of gravity for new ECS design has clearly moved toward native ECS deployment strategies.

The most common mistakes to avoid

Do not say native blue/green "contains" canary and linear. In native ECS, those are separate strategies.
Do not draw legacy canary or linear as NLB-based. The legacy NLB path supports all-at-once.
Do not imply rolling always uses exactly one target group. Rolling usually stays on the same target-group path, but ECS can support multiple target groups for some service designs.
Do not treat native ECS traffic shifting as a CodeDeploy feature. ECS supports native BLUE_GREEN, CANARY, and LINEAR with the ECS deployment controller.
Do not mix task-set language with service-revision language. Legacy CodeDeploy ECS blue/green uses task sets; native ECS deployments use service revisions.

Where to go next

This post compares legacy and ECS-native deployment strategies, but ECS-native deployments now include more capabilities than can comfortably fit into a comparison article. Built-in blue/green, canary, and linear deployment strategies support deployment lifecycle hooks — Lambda functions invoked at defined stages of a deployment, so teams can run automated validation, smoke tests, or custom checks before traffic shifts further.

These hooks pair naturally with a test listener or listener rule for pre-production validation against the green target group, as described in the ECS blue/green deployment workflow. They also work well with CloudWatch alarms and bake windows, where a failing alarm can trigger an automated rollback instead of requiring manual intervention. ECS canary and linear deployments extend the same idea by shifting production traffic gradually, giving teams multiple checkpoints to detect issues before 100% of traffic reaches the new revision.

If you already run CodeDeploy blue/green deployments and are wondering whether to migrate, AWS has published an official AWS Containers Blog post, Migrating from AWS CodeDeploy to Amazon ECS for blue/green deployments, that walks through the practical migration steps and key ECS deployment model changes. Pair it with the Migrate CodeDeploy blue/green deployments to Amazon ECS docs page for the reference details. Migration is not free — CodeDeploy deployments, ECS service revisions, and task sets use different mental models — but for most greenfield ECS services, the ECS-native model is now the better default going forward.

Monitoring & Alerting for AWS EKS Using Grafana, Prometheus & Alertmanager with SNS Integration

Muhammad Ahmad Khan — Sun, 02 Feb 2025 20:53:27 +0000

In this blog post, we'll walk through setting up a robust monitoring and alerting system for a Kubernetes cluster on AWS EKS. We'll use the kube-prometheus-stack to deploy Prometheus and Grafana, configure an ALB Ingress for external access, and set up Amazon SNS as the receiver for Alertmanager to handle alerts.

Using Amazon SNS as the alert receiver is particularly useful when an organization does not use Slack or wants a centralized way to distribute alerts via email. While Alertmanager supports email notifications, setting up email as a direct receiver requires configuring an SMTP configuration resulting in administrative overhead. With SNS, you can subscribe multiple email addresses to a single topic without additional SMTP setup, making it a simpler and more scalable solution.

However, it's important to note that SNS has limitations when handling HTML content. Alerts sent via SNS will be received as plain text rather than formatted HTML, which may impact readability.

Prerequisites

Before we begin, ensure you have the following:

An AWS account with a running EKS cluster.
kubectl and helm installed on your local machine.
AWS CLI configured with the necessary credentials.
AWS Load Balancer Controller installed in the EKS cluster.
kubectl pointing to the correct cluster context.

Step 1: Install the `kube-prometheus-stack` Using Helm

The kube-prometheus-stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules that provide easy deployment and management of Prometheus and Grafana on Kubernetes.

1.1 Add the Prometheus Community Helm Repository

First, add the Prometheus Community Helm repository and update it:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

1.2 Deploy the `kube-prometheus-stack`

Create a prom-operator-values.yaml file to customize the deployment. For example, set up the Grafana admin password, some custom dashboards, grafana ingress and other configurations:

grafana:
  adminPassword: "your-secure-password"
  ### Provision grafana-dashboards-kubernetes ###
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'grafana-dashboards-kubernetes'
        orgId: 1
        folder: 'Kubernetes'
        type: file
        disableDeletion: true
        editable: true
        options:
          path: /var/lib/grafana/dashboards/grafana-dashboards-kubernetes
  dashboards:
    grafana-dashboards-kubernetes:
      k8s-system-api-server:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json
        token: ''
      k8s-system-coredns:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json
        token: ''
      k8s-views-global:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json
        token: ''
      k8s-views-namespaces:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json
        token: ''
      k8s-views-nodes:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json
        token: ''
      k8s-views-pods:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json
        token: ''
  ingress:
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/load-balancer-name: grafana-alb
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/ssl-redirect: '443'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<AWS_REGION>:<ACCOUNT_ID>:certificate/a0ff498e-62fd-4397-8d4a-626360465d32
    enabled: true
    hosts:
    - grafana-test.apps.xyz-company.com
    labels: {}
    path: /

Deploy the kube-prometheus-stack using Helm:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

This command installs Prometheus, Grafana, and Alertmanager in the monitoring namespace.

1.3 Access Prometheus and Alertmanager UIs

To access the Prometheus and Alertmanager web UIs, use kubectl port-forward:

Prometheus UI:

  kubectl port-forward svc/prometheus-operated 9090:9090 --namespace monitoring

Open your browser and go to http://localhost:9090.

Alertmanager UI:

  kubectl port-forward svc/alertmanager-operated 9093:9093 --namespace monitoring

Open your browser and go to http://localhost:9093.

To access the Grafana web UI:

Create a DNS entry mapping the domain define in ingress with created loadbalancer url.
Use admin as the username and the password defined in prom-operator-values.yaml to log in to Grafana.

Step 2: Set Up Amazon SNS for Alertmanager

To receive alerts via email or other channels, we'll configure Amazon SNS as the receiver for Alertmanager.

2.1 Create an SNS Topic

Create an SNS topic to receive alerts:

aws sns create-topic --name alertTopic

Note the ARN (Amazon Resource Name) from the output, as it will be needed in later steps.

2.2 Create an Email Subscription for SNS

Subscribe your email to the SNS topic to receive notifications:

aws sns subscribe \
    --topic-arn arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic \
    --protocol email \
    --notification-endpoint <your-email@example.com>

Check your email inbox for a confirmation message and click the link to activate the subscription.

2.3 Create a New IAM Role for EKS Node Group to Assume and Publish to SNS

2.3.1 Create a Trust Relationship

Create a trust relationship policy to allow the EKS node group role to assume the new role:

cat << EOF > trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/eks-node-group-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the IAM role:

aws iam create-role \
    --role-name alertmanager_role \
    --assume-role-policy-document file://trust-policy.json

2.3.2 Attach Permissions to Publish to SNS

Create a policy to allow the role to publish messages to the SNS topic:

cat << EOF > sns-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic"
    }
  ]
}
EOF

Attach the policy to the new role:

aws iam put-role-policy \
    --role-name alertmanager_role \
    --policy-name SNSPublishPolicy \
    --policy-document file://sns-policy.json

2.4 Create a Resource-Based Policy for SNS

Allow this new role named alertmanager_role to publish alerts to the SNS topic by attaching a resource-based policy:

cat << EOF > sns-resource-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role"
      },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic"
    }
  ]
}
EOF

Apply the policy to the SNS topic:

aws sns set-topic-attributes \
    --topic-arn arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic \
    --attribute-name Policy \
    --attribute-value file://sns-resource-policy.json

2.5 Update the IAM Role for the EKS Node Group

Allow the EKS node group to assume this new role named alertmanager_role by attaching a policy:

cat << EOF > eks-assume-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role"
    }
  ]
}
EOF

Attach the policy to the EKS node group role:

aws iam put-role-policy \
    --role-name eks-node-group-role \
    --policy-name assume_alertmanager_role_policy \
    --policy-document file://eks-assume-policy.json

Note:

your EKS node Group role might be different than mine e.g. eks-node-group-role

Step 3: Configure Alertmanager to Send Alerts to SNS

Update the alertmanager.yml configuration to send alerts to the SNS topic in same helm values file named as prom-operator-values.yaml:

alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job', 'alertname', 'priority']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'sns-receivers'
    receivers:
    - name: 'null'  # Add this to your config as well
    - name: sns-receivers
      sns_configs:
        - api_url: https://sns.<AWS_REGION>.amazonaws.com
          topic_arn: arn:aws:sns:<AWS_REGION>:<ACCOUNT_ID>:alertTopic
          sigv4:
            region: <AWS_REGION>
            role_arn: arn:aws:iam::<ACCOUNT_ID>:role/alertmanager_role
          subject: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]'
          message: |-
            {{ if gt (len .Alerts.Firing) 0 }}
            Alerts Firing:
            {{ template "__text_alert_list_markdown" .Alerts.Firing }}
            {{ end }}
            {{ if gt (len .Alerts.Resolved) 0 }}
            Alerts Resolved:
            {{ template "__text_alert_list_markdown" .Alerts.Resolved }}
            {{ end }}
          send_resolved: true  # Sends notification when alert is resolved

Apply the updated configuration:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

Step 4: Configure custom rules for custom alerts

Update the prom-operator-values.yaml to add:

additionalPrometheusRulesMap:
  custom-rules:
    groups:
    - name: customGroupA.rules
      rules:
      - alert: Custom-Alert Instance High CPU Utilization
        annotations:
          description: CPU usage had been over 75% for 5 minutes | Current usage is {{ $value | printf "%.2f" }}%
          summary: CPU usage is over 75% (instance {{ $labels.instance }})
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 75
        for: 5m
        labels:
          severity: critical
      - alert: Custom-Alert Instance High Memory Utilization
        annotations:
          description: Memory usage had been over 75% for 5 minutes | Current usage is {{ $value | printf "%.2f" }}%
          summary: Memory usage is over 75% (instance {{ $labels.instance }})
        expr: 100 - (sum by(instance) (node_memory_MemAvailable_bytes) / sum by(instance) (node_memory_MemTotal_bytes) * 100) > 75
        for: 5m
        labels:
          severity: critical

Apply the updated configuration:

helm upgrade --install k-prom-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values prom-operator-values.yaml

The complete helm values file can be found on GitHub here

Conclusion

In this blog post, we set up a comprehensive monitoring and alerting system for a Kubernetes cluster running on AWS EKS. We deployed Prometheus and Grafana using the kube-prometheus-stack, configured an ALB Ingress for external access to Grafana, and set up Amazon SNS as the receiver for Alertmanager to receive alerts.

With this setup, you can now monitor your Kubernetes cluster, visualize metrics using Grafana, and receive alerts via Alertmanager and SNS. This ensures that your cluster is both observable and resilient to issues.

Rapidly Deploy ECS Infrastructure on AWS with AWS CDK (TypeScript)

Muhammad Ahmad Khan — Sat, 25 Jan 2025 02:22:52 +0000

When building and deploying containerized applications on AWS, Amazon ECS (Elastic Container Service) provides a powerful and flexible solution. AWS CDK (Cloud Development Kit) simplifies the process of creating ECS infrastructure by providing a programmatic approach to define cloud resources in code. This blog post walks through how to rapidly deploy ECS infrastructure using the aws-cdk-lib/aws-ecs-patterns module, which abstracts complex tasks like load balancing, service management, and autoscaling.

We’ll cover how to create an ECS Fargate service, set up an Application Load Balancer (ALB), enable auto-scaling, and integrate AWS Secrets Manager for securely managing sensitive data, all with just a few lines of TypeScript code.

Prerequisites

Before getting started, make sure you have the following:

Node.js and npm installed.
AWS CLI configured with appropriate permissions to create resources. You can configure it with:

   aws configure

AWS CDK installed globally:

   npm install -g aws-cdk

Docker installed (optional but useful for container builds).

Setting Up the Project

To begin, create a new directory for your CDK project and initialize it with TypeScript:

mkdir ecs-fargate
cd ecs-fargate
cdk init app --language typescript

This will generate the basic project structure for an AWS CDK application. Next, install the necessary dependencies for ECS, VPC, ECR, and other related services:

npm install @aws-cdk/aws-ec2 @aws-cdk/aws-ecs @aws-cdk/aws-ecr @aws-cdk/aws-ecs-patterns @aws-cdk/aws-iam @aws-cdk/aws-elasticloadbalancingv2 @aws-cdk/aws-certificatemanager @aws-cdk/aws-secretsmanager

Now you're ready to start defining your infrastructure!

Using `aws-cdk-lib/aws-ecs-patterns`

The aws-cdk-lib/aws-ecs-patterns module simplifies ECS service setup by providing high-level abstractions for common ECS patterns. It includes constructs for easy configuration of services like Application Load Balanced Fargate services, which handle the complexity of setting up load balancers, auto-scaling, and other integrations.

For example, the ApplicationLoadBalancedFargateService construct automatically creates an ECS Fargate service with an Application Load Balancer (ALB) in front of it, managing much of the networking and routing for you.

Here’s a quick look at some key benefits of using the ApplicationLoadBalancedFargateService construct from the aws-cdk-lib/aws-ecs-patterns module:

Simplified Infrastructure Setup: You don't need to manually define an Application Load Balancer or its target group—everything is handled for you.
Built-in Auto-Scaling: Easily configure auto-scaling for your Fargate tasks based on CPU or memory usage.
HTTPS Setup: You can easily set up SSL certificates via ACM (AWS Certificate Manager) and configure the load balancer to use HTTPS.

For more details on other architectural patterns, you can refer to the official AWS CDK ECS Patterns Documentation.

Defining the ECS Fargate Infrastructure

Here’s an example that demonstrates how to quickly deploy ECS infrastructure with AWS CDK using TypeScript.

Example: ECS Fargate Service with Application Load Balancer

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecr from 'aws-cdk-lib/aws-ecr';
import * as ecs_patterns from 'aws-cdk-lib/aws-ecs-patterns';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';
import * as acm from 'aws-cdk-lib/aws-certificatemanager';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';

interface EcsFargateStackProps extends cdk.StackProps {
  vpcId: string; // VPC Id
  ecrRepositoryName: string; // ECR repository name
  acmCertificateArn: string; // ACM certificate ARN for HTTPS
  demoServiceSecretArn: string; // Secret ARN for demoService
}

export class EcsFargateStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: EcsFargateStackProps) {
    super(scope, id, props);

    // Lookup existing VPC using VPC ID from props
    const vpc = ec2.Vpc.fromLookup(this, 'EcsVpc', {
      vpcId: props.vpcId,
    });

    // Reference the existing ECR repository
    const ecrRepo = ecr.Repository.fromRepositoryName(this, 'ExistingEcrRepository', props.ecrRepositoryName);

    // ECS Cluster
    const cluster = new ecs.Cluster(this, 'EcsCluster', {
      clusterName: 'demo-service-EcsFargateCluster',
      vpc,
    });

    // Task Definition
    const taskDefinition = new ecs.FargateTaskDefinition(this, 'FargateTaskDef', {
      memoryLimitMiB: 512,
      cpu: 256,
    });

    // Reference the secret from AWS Secrets Manager
    const taskSecret = secretsmanager.Secret.fromSecretCompleteArn(this, 'FargateTaskSecret', props.demoServiceSecretArn);

    // Add a container to the task definition
    const container = taskDefinition.addContainer('AppContainer', {
      image: ecs.ContainerImage.fromEcrRepository(ecrRepo, 'latest'),
      logging: ecs.LogDriver.awsLogs({ streamPrefix: 'ecs-app' }),
      environment: {
        DEBUG: 'true', // Simple environment variable
      },
      secrets: {
        SECRET_KEY: ecs.Secret.fromSecretsManager(taskSecret, 'SECRET_KEY'), // Referencing secret for SECRET_KEY
      },
    });

    container.addPortMappings({
      containerPort: 5000, // Container runs on port 5000
    });

    // ACM Certificate for HTTPS
    const certificate = acm.Certificate.fromCertificateArn(this, 'AcmCertificate', props.acmCertificateArn);

    // Application Load Balanced Fargate Service
    const fargateService = new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'FargateService', {
      cluster,
      taskDefinition,
      enableExecuteCommand: true, // Enable execute command
      certificate,
      publicLoadBalancer: true,
      loadBalancerName: 'demo-service-alb',
      sslPolicy: elbv2.SslPolicy.RECOMMENDED,
      serviceName: 'demo-service-EcsFargateService',
      redirectHTTP: true,
      protocol: elbv2.ApplicationProtocol.HTTPS, // Default HTTPS
    });

    // Adjust target group port to match the container port
    fargateService.targetGroup.configureHealthCheck({
      path: '/',
      port: '5000', // Health check targets container port 5000
    });

    // Autoscaling
    const scaling = fargateService.service.autoScaleTaskCount({
      minCapacity: 1,
      maxCapacity: 3,
    });

    scaling.scaleOnCpuUtilization('CpuScaling', {
      targetUtilizationPercent: 80,
    });

    scaling.scaleOnMemoryUtilization('MemoryScaling', {
      targetUtilizationPercent: 80,
    });

    // Outputs
    new cdk.CfnOutput(this, 'ClusterName', {
      value: cluster.clusterName,
      description: 'ECS Cluster Name',
    });

    new cdk.CfnOutput(this, 'ServiceName', {
      value: fargateService.service.serviceName,
      description: 'ECS Service Name',
    });
  }
}

Breakdown of the Code:

VPC and ECS Cluster: We use ec2.Vpc.fromLookup() to reference an existing VPC based on the VPC ID passed in as a parameter, and create an ECS cluster within that VPC.
ECR Repository: The Docker image for the application is stored in an existing ECR repository. We reference this repository using ecr.Repository.fromRepositoryName().
Task Definition: We define an ECS Fargate task with specified memory and CPU limits, referencing a Docker image from ECR. We also integrate environment variables and sensitive data from AWS Secrets Manager.
Load Balancer and HTTPS: The ApplicationLoadBalancedFargateService construct from aws-cdk-lib/aws-ecs-patterns automatically provisions an ALB, configuring SSL termination using an ACM certificate for HTTPS.
Auto-Scaling: We configure auto-scaling for the Fargate service based on CPU and memory utilization, ensuring the service can dynamically adjust to demand.
Outputs: Finally, we output the ECS cluster name and service name, so they can be easily referenced.

Deploying the Stack

Step 1: Install Dependencies

Ensure that all dependencies are installed:

npm install

Step 2: Bootstrap the CDK Environment

Run the bootstrap command to prepare your AWS account for CDK:

cdk bootstrap --profile your-aws-profile

Step 3: Synthesize the Stack

Synthesize the CloudFormation template to verify the stack:

cdk synth --context vpcId=vpc-xxxxxxxx --context ecrRepositoryName=demo-service --context acmCertificateArn=arn:aws:acm:region:account-id:certificate/certificate-id --context demoServiceSecretArn=arn:aws:secretsmanager:region:account-id:secret:secret-id --profile your-aws-profile

Step 4: Deploy the Stack

Deploy the ECS Fargate stack to AWS:

cdk deploy --context vpcId=vpc-xxxxxxxx --context ecrRepositoryName=demo-service --context acmCertificateArn=arn:aws:acm:region:account-id:certificate/certificate-id --context demoServiceSecretArn=arn:aws:secretsmanager:region:account-id:secret:secret-id --profile your-aws-profile

Step 4: Verify the Deployment

After the stack is successfully deployed, AWS will automatically provision the ECS service, ALB, and other necessary resources. You can access the service via the load balancer’s public endpoint.

Conclusion

In this guide, we’ve rapidly provisioned ECS infrastructure using AWS CDK and TypeScript. By leveraging the ApplicationLoadBalancedFargateService construct from aws-cdk-lib/aws-ecs-patterns, we were able to simplify the creation of a fully managed ECS service that includes an ALB, auto-scaling, and HTTPS support.

This approach significantly reduces the complexity of deploying containerized applications on AWS, allowing you to focus on developing your application instead of managing infrastructure.

For more detailed information, check out the official AWS CDK ECS Patterns Documentation.

GitHub Repository

You can find the complete code for this deployment in my GitHub repository.

Integrating Multiple EKS Clusters with ArgoCD for Simplifying Kubernetes Operations

Muhammad Ahmad Khan — Tue, 26 Mar 2024 19:49:52 +0000

In the rapidly evolving landscape of cloud-native technologies, Kubernetes has emerged as the de facto standard for container orchestration. As organizations scale their infrastructure, managing multiple Kubernetes clusters becomes inevitable. With this growth comes the challenge of ensuring consistency, reliability, and efficiency across all clusters. Enter ArgoCD, a powerful tool for continuous delivery and GitOps workflows in Kubernetes. In this blog post, we'll explore why integrating multiple clusters in ArgoCD is essential and how we can integrate multiple AWS EKS clusters in ArgoCD.

Why Integration is Required

Centralized Management: Managing multiple Kubernetes clusters manually can be daunting and error-prone. Integrating them with ArgoCD provides a centralized platform for managing and deploying applications across all clusters, streamlining operations and reducing complexity.
Consistency and Standardization: Different clusters may have varying configurations, making it difficult to maintain consistency in deployments. ArgoCD ensures that configurations and deployments are standardized across all clusters, promoting best practices and ensuring uniformity.
Scalability: As organizations grow, they often adopt a multi-cluster strategy to distribute workloads and improve fault tolerance. Integrating these clusters with ArgoCD enables seamless scaling of applications across clusters, allowing organizations to leverage resources efficiently.
Monitoring [Health Checks and Logging]: ArgoCD offers insights into the deployment status and health of applications across clusters through its user interface and API. By integrating multiple clusters, organizations gain centralized visibility and can monitor the status of applications across all clusters from a single dashboard.

How to Integrate Multiple AWS EKS Clusters in ArgoCD

Let's say we have AWS accounts as follows:

Account A with account id: 111111111111
Account B with account id: 222222222222
Account C with account id: 333333333333

Account A is where ArgoCD runs.

To authenticate and access the external cluster we need to add the configuration as follows:

In Account A:

Create an IAM role named argocd-manager.
Create a role policy named argocd-role-policy and attach it to a role named argocd-manager having the assume role policy given below

RolePolicyDocument

cat >argocd-role-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "*"
        }
    ]
}
EOF

AssumeRolePolicyDocument

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": [ "system:serviceaccount:argocd:argocd-server", "system:serviceaccount:argocd:argocd-application-controller" ]
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Now In Account B:

Create an IAM role named deployer having trust relationship as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::11111111111:role/argocd-manager"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Map this role in aws auth-config configmap Kubernetes object in Account B EKS cluster

kubectl edit -n kube-system configmap/aws-auth


# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::222222222222:role/my-role
      username: system:node:{{EC2PrivateDNSName}}
    - groups:
      - system:masters
      rolearn: arn:aws:iam::222222222222:role/deployer # deployer role arn
      username: deployer
  mapUsers: |
    - groups:
      - system:masters
      userarn: arn:aws:iam::222222222222:user/admin
      username: admin
    - groups:
      - system:masters      
      userarn: arn:aws:iam::222222222222:user/alpha-user
      username: my-user

Follow the same procedure in Account C as we have followed in Account B.

In Account A (where argocd is installed), add the following configuration in argocd helm chart values

Note: IAM role deployer must be created first in Account B or C

arn:aws:iam::222222222222:role/deployer
arn:aws:iam::333333333333:role/deployer

global:
  securityContext: # Set deployments securityContext/fsGroup to 999 so that the user of the docker image can use IAM Authenticator. We need this because the IAM Authenticator will try to mount a secret on /var/run/secrets/eks.amazonaws.com/serviceaccount/token. If the correct fsGroup (999 corresponds to the argocd user) isn’t set, this will fail.
    runAsGroup: 999
    fsGroup: 999

controller:
  serviceAccount:
    create: true
    name: argocd-application-controller
    annotations: {eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/argocd-manager} # Account A - IAM role service account
    automountServiceAccountToken: true

server:
  serviceAccount:
    create: true
    name: argocd-server
    annotations: {eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/argocd-manager} # Account A - IAM role service account
    automountServiceAccountToken: true

 configs:
  # -- Provide one or multiple [external cluster credentials]
  # @default -- `[]` (See [values.yaml])
  ## Ref:
  ## - https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
  ## - https://argo-cd.readthedocs.io/en/stable/operator-manual/security/#external-cluster-credentials
  ## - https://argo-cd.readthedocs.io/en/stable/user-guide/projects/#project-scoped-repositories-and-clusters
  clusterCredentials:
    - name: development
      server: https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.abc.region.eks.amazonaws.com # EKS cluster API server endpoint of Account B
      config:
        awsAuthConfig:
          clusterName: eks-development
          roleARN: arn:aws:iam::222222222222:role/deployer # Deployer role arn of Account B
        tlsClientConfig:
          # Base64 encoded PEM-encoded bytes (typically read from a client certificate file).
          caData: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx........==" # EKS cluster certificate authority
    - name: staging
      server: https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.abc.region.eks.amazonaws.com # EKS cluster API server endpoint of Account C
      config:
        awsAuthConfig:
          clusterName: eks-staging
          roleARN: arn:aws:iam::333333333333:role/deployer # Deployer role arn of Account C
        tlsClientConfig:
          # Base64 encoded PEM-encoded bytes (typically read from a client certificate file).
          caData: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx........==" # EKS cluster certificate authority

Obtain the EKS certificate of the respective cluster using AWS CLI:

aws eks describe-cluster \
        --region=${AWS_DEFAULT_REGION} \
        --name=${CLUSTER_NAME} \
        --output=text \
        --query 'cluster.{certificateAuthorityData: certificateAuthority.data}' | base64 -D

The important thing to note is that we need to set deployments securityContext/fsGroup to 999 so that the user of the docker image can use IAM Authenticator. We need this because the IAM Authenticator will try to mount a secret on /var/run/secrets/eks.amazonaws.com/serviceaccount/token. If the correct fsGroup (999 corresponds to the argocd user) isn’t set, this will fail.

Upgrading EKS Version with Zero Downtime Incorporating Custom Blue-Green Methodology

Muhammad Ahmad Khan — Sat, 23 Mar 2024 21:22:21 +0000

One of the most significant operational overheads of EKS is its version upgrade, which we have to tackle each quarter. Although AWS has announced extended support for older versions of EKS, the cost of extended support is something you do not want to incur. AWS maintains its standard support and extended support timeline in the following documentation: Kubernetes Release Calendar.

Keeping EKS clusters up to date with the latest versions is essential for security, performance, and access to new features. However, upgrading Kubernetes clusters can be challenging, especially in production environments where downtime is not an option. In this blog post, we'll explore how you can leverage the custom blue-green methodology to perform version upgrades on AWS EKS clusters seamlessly and with minimal disruption.

The blue-green deployment strategy is a technique used to reduce downtime and risk when deploying updates to applications or infrastructure. In the context of AWS EKS version upgrades, the traditional blue-green methodology involves creating a new cluster with the updated Kubernetes version (green), migrating workloads from the old cluster (blue) to the new one, and then decommissioning the old cluster. However, in our case, we will maintain a blue cluster only to serve traffic when we plan to upgrade our original cluster (green). This means that our original cluster will always remain green (before and after the EKS upgrade), and traffic will only shift from the green to the blue cluster while we perform upgrades on the original cluster. After the successful upgrade, traffic will shift back from the blue cluster to the original cluster (green).

Steps to Perform EKS Version Upgrade using Custom Blue-Green Methodology:

Prepare for Upgrade: Before initiating the upgrade process, thoroughly review the release notes for the new Kubernetes version to understand any changes or potential compatibility issues with your workloads. This includes checking deprecated resources and APIs, whether in your self-maintained application chart or in some open-source community chart. If you are using an open-source community chart, please ensure that the EKS version you are upgrading to has been tested with your installed chart.
Create a New Blue Cluster or Upgrade the Already Created Blue Cluster: Using AWS EKS, provision a new Kubernetes cluster with the desired version. Ensure that the cluster configuration, including node groups, networking, and IAM roles, aligns with your requirements. If you have created a new EKS cluster, make sure to configure DNS and Load Balancers by updating DNS records or adjusting load balancer settings to direct traffic to the new cluster gradually when workloads are migrated. If your cluster is large and you have many Kubernetes Ops tools installed in Kubernetes using Helm charts, it is a good idea to maintain this separate cluster for each upgrade cycle to save time. If you are already maintaining the cluster, then you have to prepare this cluster for upgrade by performing the following tasks:
- Upgrade the EKS control plane from the console.
- Upgrade each EKS Node group by setting the right update config as per the priority from the console and using Force Update.
- Upgrade add-ons from the console.
Deploy/Update Workloads: Utilize Kubernetes tools such as kubectl or Helm charts to deploy workloads onto the new blue cluster or update in the existing blue cluster. Monitor performance and functionality to ensure a smooth transition.
Validate and Test: Conduct thorough testing on this new/upgraded blue cluster to verify that all workloads and applications function as expected. Use automated testing tools and perform manual checks to validate performance, scalability, and reliability.
Cut Over Traffic: Once validation is complete, gradually shift production traffic from the old cluster to the new upgraded one using weighted routing. Monitor logs, pod scaling, and AWS Load Balancer metrics such as RequestCount as traffic increases. Monitor closely for any anomalies or performance issues during the transition.
Upgrade Original Cluster: Now perform the same process for the original cluster, including upgrading the EKS control plane, node groups, add-ons, as well as Kubernetes workloads. Verify if everything is working fine and transition traffic gradually back from the blue cluster to this original one after the full upgrade.
Decommission/Scale Down Blue Cluster: After confirming that the original cluster is handling all traffic effectively, decommission/scale down all node groups of the blue cluster to release resources and reduce operational overhead.

Best Practices for EKS Version Upgrades:

Automate wherever possible to streamline the upgrade process and reduce manual errors.
Implement monitoring and alerting to detect and respond to any issues promptly.
Maintain clear communication with stakeholders throughout the upgrade process to manage expectations and address concerns proactively.

AWS EKS provides a robust platform for managing Kubernetes clusters, and staying current with the latest versions is crucial for leveraging new features and ensuring security and performance. By adopting the blue-green methodology for version upgrades, organizations can minimize downtime, reduce risks, and maintain a seamless experience for end-users. By following best practices and leveraging automation and monitoring capabilities, you can confidently navigate EKS version upgrades and keep your Kubernetes infrastructure up to date with minimal disruption.

References: Kubernetes Deprecation Guide.

Distributing restricted static content through CloudFront using signed cookies.

Muhammad Ahmad Khan — Mon, 21 Nov 2022 14:47:55 +0000

Many times it's a business requirement that the content you share over the internet through your application including documents, business data, or media streams is intended to be consumed by selected users, for example, users who have paid a fee. Suppose that you have a similar situation and you are using an AWS CloudFront distribution (CDN) to distribute the static content including documents, images, and videos. But you want a restriction that this content is accessible through your app only to a set of selected users. Then we can use signed URLs and signed cookies.
CloudFront signed URLs and signed cookies provide the same basic functionality. They allow you to control who can access your content but both are beneficial in entirely different situations based on different requirements.
E.g.

Using signed URLs is beneficial when you want to restrict access to individual files, for example, an installation download for your application.
Using signed cookies is beneficial when you want to provide access to multiple restricted files, and don't want to change your current URLs.

In this blog, we will discuss about using signed cookies to grant access to your private content to the relevant user using CloudFront.
Signed URLs take precedence over signed cookies if you use both signed URLs and signed cookies to control access to your content.

How signed cookies works?
Here is an overview of how signed cookies work.

In your CloudFront distribution, specify one or more trusted key groups, which contain the public keys that CloudFront can use to verify the URL signature. You use the corresponding private keys to sign the URLs. Since you use private keys to sign the URLs you must store them somewhere and we will store them in AWS Secrets Manager. To create a KeyPair and to add a signer (trusted key group) in CloudFront distribution see the procedure here.
You develop your application to determine whether a user should have access to your content and if so, to send three Set-Cookie headers to the viewer. (Each Set-Cookie header can contain only one name-value pair, and a CloudFront signed cookie requires three name-value pairs.) You must send the Set-Cookie headers to the viewer before the viewer requests your private content. Typically, your CloudFront distribution will have at least two cache behaviors, one that doesn't require authentication and one that does. The error page for the secure portion of the site includes a redirector or a link to a login page. If you configure your distribution to cache files based on cookies, CloudFront doesn't cache separate files based on the attributes in signed cookies.
A user signs in to your website and either pays for content or meets some other requirement for access.
Your application returns the Set-Cookie headers in the response, and the viewer stores the name-value pairs.
The user requests a file. The user's browser or other viewer gets the name-value pairs from step 4 and adds them to the request in a Cookie header. This is the signed cookie.
CloudFront uses the public key to validate the signature in the signed cookie and to confirm that the cookie hasn't been tampered with. If the signature is invalid, the request is rejected. If the signature in the cookie is valid, CloudFront looks at the policy statement in the cookie (or constructs one if you're using a canned policy) to confirm that the request is still valid. If the request meets the requirements in the policy statement, CloudFront serves your content as it does for content that isn't restricted: it determines whether the file is already in the edge cache, forwards the request to the origin if necessary, and returns the file to the user.

There are two kinds of policies used for signed cookies

Canned policies
Custom policies

When you create a signed cookie, you write a policy statement in JSON format that specifies the restrictions on the signed cookie, for example, how long the cookie is valid. You can use canned policies or custom policies.
Canned policies allow you to specify an expiration date only. Custom policies allow more complex restrictions.

Canned policy example:

{
    "Statement": [
        {
            "Resource": "base URL or stream name",
            "Condition": {
                "DateLessThan": {
                    "AWS:EpochTime": ending date and time in Unix time format and UTC
                }
            }
        }
    ]
}

Custom policy example:

{
    "Statement": [
        {
            "Resource": "URL of the file",
            "Condition": {
                "DateLessThan": {
                    "AWS:EpochTime":required ending date and time in Unix time format and UTC
                },
                "DateGreaterThan": {
                    "AWS:EpochTime":optional beginning date and time in Unix time format and UTC
                },
                "IpAddress": {
                    "AWS:SourceIp": "optional IP address"
                }
            }
        }
    ]
}

Since we want to give access to all of the files located on a certain path of our CloudFront URL using wildcard * therefore we will use a custom policy.
For example, we want to give access to all of the files located on this URL: https://content.mysite.com/private-content/*
Here is the simple code in PHP for creating signed cookies using custom policy.

<?php

require 'vendor/autoload.php';

use Aws\Credentials\Credentials;
use Aws\SecretsManager\SecretsManagerClient;
use Aws\CloudFront\CloudFrontClient;
use Aws\Exception\AwsException;

// Check if user is login or check if user is paid the fee


// Initialize AWS credentials from IAM role
$credentials_uri = getenv('AWS_CONTAINER_CREDENTIALS_RELATIVE_URI');
$credentials_url = 'http://169.254.170.2' . $credentials_uri;
$get_credentials = file_get_contents($credentials_url);
$credentials_array = json_decode($get_credentials, true);
$credentials = new Credentials($credentials_array["AccessKeyId"], $credentials_array["SecretAccessKey"], $credentials_array["Token"], strtotime($credentials_array["Expiration"]));

// Create a Secrets Manager Client
$secretManagerClient = new SecretsManagerClient([
    'credentials' => $credentials,
    'version' => '2017-10-17',
    'region' => 'us-east-1',
]);

$secretName = 'private-content-access-key';

try {
    $secretResult = $secretManagerClient->getSecretValue([
        'SecretId' => $secretName,
    ]);

}
catch (AwsException $e) {
    // output error message if fails
    echo $e->getAwsErrorMessage();
    echo "\n";
}


function signCookie($cloudFrontClient, $policy, $privateKey, $keyPairId)
{
    try {
        $result = $cloudFrontClient->getSignedCookie([
            'policy' => $policy,
            'private_key' => $privateKey,
            'key_pair_id' => $keyPairId
        ]);

        return $result;

    }
    catch (AwsException $e) {
        return ['Error' => $e->getAwsErrorMessage()];
    }
}

function signACookie($privateKey)
{
    $resourcePath = 'https://content.mysite.com/private-content/*';
    $expires = time() + 3600; // 60 minutes (60 * 60 seconds) from now.
    $keyPairId = 'ABCDWXYZ';
    $policy =
    '{'.
        '"Statement":['.
            '{'.
                '"Resource":"'. $resourcePath . '",'.
                '"Condition":{'.
                    '"DateLessThan":{"AWS:EpochTime":' . $expires . '}'.
                '}'.
            '}'.
        ']' .
    '}';

    $cloudFrontClient = new CloudFrontClient([
        'credentials' => $credentials,
        'version' => '2014-11-06',
        'region' => 'us-east-1'
    ]);

    $result = signCookie(
        $cloudFrontClient,
        $policy,
        $privateKey,
        $keyPairId
    );

    $cookies = array();

    foreach ($result as $key => $value) {
        array_push(
            $cookies,
            array(
            'name' => $key,
            'value' => $value,
            'expires' => $expires,
            'path' => '/',
            'domain' => ".mysite.com",
            'secure' => true,
            'httpOnly' => true
        )
        );

    }

    foreach ($cookies as $cookie) {
        setcookie
            (
            $cookie['name'],
            $cookie['value'],
            $cookie['expires'],
            $cookie['path'],
            $cookie['domain'],
            $cookie['secure'],
            $cookie['httpOnly']
        );
    }
}

if (isset($secretResult['SecretString'])) {
    $privateKey = $secretResult['SecretString'];
    signACookie($privateKey);
}

// Give access to the content e.g https://content.mysite.com/private-content/doc.pdf

You can easily write similar code in Node.js (Javascript).
The most important function to consider from the CloudFront client is getSignedCookie() which takes three parameters: policy, private key, and key pair id (this is not key group Id).
This function returns a response with the following values.

CloudFront-Policy=base64 encoded version of the policy statement;
CloudFront-Signature=hashed and signed version of the policy statement;
CloudFront-Key-Pair-Id=public key ID for the CloudFront public key whose corresponding private key you're using to generate the signature;

Finally, we will set the cookies by using values returned in the response.

SQS depth based ECS task auto-scaling using step scaling.

Muhammad Ahmad Khan — Sun, 20 Nov 2022 21:20:10 +0000

As mentioned in a previous blog post here, we can easily apply scaling using step scaling on SQS queue depth.

Step scaling policies increase or decrease the current capacity of a scalable target based on a set of scaling adjustments, known as step adjustments. The adjustments vary based on the size of the alarm breach. All alarms that are breached are evaluated by Application Auto Scaling as it receives the alarm messages.

With step scaling, you choose scaling metrics and threshold values for the CloudWatch alarms that trigger the scaling process and thus it requires you to create CloudWatch alarms.

When you create a step scaling policy, you add one or more step adjustments that enable you to scale based on the size of the alarm breach. Each step adjustment specifies the following:

A lower bound for the metric value
An upper bound for the metric value
The amount by which to scale, based on the scaling adjustment type

Application Auto Scaling supports the following adjustment types for step scaling policies:

ChangeInCapacity—Increase or decrease the current capacity of the scalable target by the specified value. A positive value increases the capacity and a negative value decreases the capacity. For example: If the current capacity is 3 and the adjustment is 5, then Application Auto Scaling adds 5 to the capacity for a total of 8.

ExactCapacity—Change the current capacity of the scalable target to the specified value. Specify a positive value with this adjustment type. For example: If the current capacity is 3 and the adjustment is 5, then Application Auto Scaling changes the capacity to 5.

PercentChangeInCapacity—Increase or decrease the current capacity of the scalable target by the specified percentage. A positive value increases the capacity and a negative value decreases the capacity. For example: If the current capacity is 10 and the adjustment is 10 percent, then Application Auto Scaling adds 1 to the capacity for a total of 11.

Below is the CloudFormation template with the implementation of step scaling with SQS.

AWSTemplateFormatVersion: 2010-09-09
Description: Creates a fargate based auto-scaling environment that processes work from an SQS queue
Parameters:
  DockerImageUrl:
    Type: String
    Default: latest

  DockerContainerName:
    Type: String
    Default: consumer-service

  EnvironmentName:
    Type: String
    Default: dev

  Memory:
    Type: String
    Default: 8GB

  Cpu:
    Type: Number
    Default: 2048 # 2 vCPU

  ContainerPort:
    Type: Number
    Default: 3000

  HealthCheckPath:
    Type: String
    Default: http://localhost:3000/check

  FaragateScalingEnvSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/FARGATE_SCALING_ENV"

  QueueDepthScaleOutAlarmThresholdSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD"

  CpuUtilizationScaleInAlarmThresholdSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD"

  CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS"

  ComputeAutoScalingTargetMaxCapacitySSM:
    Type: AWS::SSM::Parameter::Value<String>
    Default: "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY"

Conditions:
  CreateNonProdResources: !Equals [!Ref FaragateScalingEnvSSM, 'non-prod']
  CreateProdResources: !Equals [!Ref FaragateScalingEnvSSM, 'prod']

Resources:
  SQSQueue:
    Type: 'AWS::SQS::Queue'
    # Properties:
    #   ReceiveMessageWaitTimeSeconds: 20
    #   VisibilityTimeout: 1200 # 20 minutes
    #   MessageRetentionPeriod: 1209600 # 14 Days

  QueueUrlParameter:
    Type: 'AWS::SSM::Parameter'
    Properties:
      Name: !Join
        - ''
        - - /
          - !Ref EnvironmentName
          - /services/
          - !Ref DockerContainerName
          - /SQS_QUEUE_URL
      Type: String
      Value: !Ref SQSQueue

  ComputeTaskLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Join
        - /
        - - /x-org
          - ecs
          - !Sub '${AWS::StackName}'
          - logs

  ComputeTaskRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: Required_Access
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'sqs:*'
                  - 'secretsmanager:*'
                  - 'ssm:*'
                  - 'logs:*'
                  - 'dynamodb:*'
                  - 's3:*'
                  - 'ecs:*'
                Resource: '*'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'

  ComputeTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    DependsOn: ComputeTaskLogGroup
    Properties:
      TaskRoleArn: !GetAtt ComputeTaskRole.Arn
      ExecutionRoleArn: !GetAtt ComputeTaskRole.Arn
      RequiresCompatibilities:
        - FARGATE
      NetworkMode: awsvpc
      Cpu: !Ref Cpu
      Memory: !Ref Memory
      ContainerDefinitions:
        - Name: !Sub '${AWS::StackName}'
          Image: !Ref DockerImageUrl
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: us-east-1
              awslogs-group: !Ref ComputeTaskLogGroup
              awslogs-stream-prefix: ecs
          HealthCheck:
            Command:
              - CMD-SHELL
              - !Sub 'curl -f ${HealthCheckPath} || exit 1'
            Interval: 30
            Retries: 3
            StartPeriod: 300
          PortMappings:
            - ContainerPort: !Ref ContainerPort
              Protocol: tcp
          Environment:
            - Name: EnvironmentName
              Value: !Ref EnvironmentName
            - Name: SQS_QUEUE_URL
              Value: !Ref SQSQueue

  ComputeCluster:
    Type: 'AWS::ECS::Cluster'
    # Properties:
    #   ClusterName: !Join ['-', [!Ref DockerContainerName, cluster]]

  NonProdComputeService:
    Type: 'AWS::ECS::Service'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      Cluster: !Ref ComputeCluster
      TaskDefinition: !Ref ComputeTaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      # Desired count should be 0; Otherwise the Task Scheduler will restart number of desired containers once they are stopped
      DesiredCount: 0
      # This may need to be adjusted if the container takes a while to start up
      # HealthCheckGracePeriodSeconds: 30
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          # Change it to DISABLED if you're using private subnets that have access to a NAT gateway
          AssignPublicIp: ENABLED
          Subnets:
            - !ImportValue ComputeSubnetA
            - !ImportValue ComputeSubnetB
            - !ImportValue ComputeSubnetC
          SecurityGroups:
            - !ImportValue ComputeSecurityGroup

  ProdComputeService:
    Type: 'AWS::ECS::Service'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      Cluster: !Ref ComputeCluster
      TaskDefinition: !Ref ComputeTaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      # This may need to be adjusted if the container takes a while to start up
      # HealthCheckGracePeriodSeconds: 30
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          # Change it to DISABLED if you're using private subnets that have access to a NAT gateway
          AssignPublicIp: ENABLED
          Subnets:
            - !ImportValue ComputeSubnetA
            - !ImportValue ComputeSubnetB
            - !ImportValue ComputeSubnetC
          SecurityGroups:
            - !ImportValue ComputeSecurityGroup

  ComputeAutoScalingRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
                - application-autoscaling.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: "/"
      Policies:
      - PolicyName: !Sub ${DockerContainerName}-ECSAutoScalingRole
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
            - ecs:UpdateService
            - ecs:DescribeServices
            - application-autoscaling:*
            - cloudwatch:DescribeAlarms
            - cloudwatch:GetMetricStatistics
            Resource: "*"
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole'

  NonProdComputeAutoScalingTarget:
    Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      MinCapacity: 0 # As desired task can be 0
      MaxCapacity: !Ref ComputeAutoScalingTargetMaxCapacitySSM
      ResourceId: !Join
          - '/'
          - - service
            - !Ref ComputeCluster
            - !GetAtt NonProdComputeService.Name
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      RoleARN: !GetAtt ComputeAutoScalingRole.Arn

  ProdComputeAutoScalingTarget:
    Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      MinCapacity: 1 # As desired task can be 1 but not 0
      MaxCapacity: !Ref ComputeAutoScalingTargetMaxCapacitySSM 
      ResourceId: !Join
          - '/'
          - - service
            - !Ref ComputeCluster
            - !GetAtt ProdComputeService.Name
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      RoleARN: !GetAtt ComputeAutoScalingRole.Arn

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdNoComputeAutoScalingPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdNoComputeAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ExactCapacity # Can use PercentChangeInCapacity but then need to come up with configuration including some estimated change in percent
        Cooldown: 60
        MetricAggregationType: Average # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average. 
        StepAdjustments: 
        - MetricIntervalLowerBound: !Ref AWS::NoValue
          MetricIntervalUpperBound: 0
          ScalingAdjustment: 0

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdInitialComputeAutoScalingPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdInitialComputeAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay 
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: 0
          MetricIntervalUpperBound: !Ref AWS::NoValue
          ScalingAdjustment: 1 # scaling up by 1 container when the alarm is greater than or equal to the Metric Threshold

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  NonProdComputeAutoScalingScaleOutPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateNonProdResources # only create if it is NonProd env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-NonProdComputeAutoScalingScaleOutPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref NonProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average. 
        StepAdjustments: 
        - MetricIntervalLowerBound: 0  # 0 means exactly equal to Metric Threshold which is 10 defined using SSM parameter
          MetricIntervalUpperBound: 15 # [Metrice Threshold + 15]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 15 # [Metrice Threshold + 15]
          MetricIntervalUpperBound: 25 # [Metrice Threshold + 25]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 25 # [Metrice Threshold + 25]
          MetricIntervalUpperBound: 35 # [Metrice Threshold + 35]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 35 # [Metrice Threshold + 35]
          MetricIntervalUpperBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  ProdComputeAutoScalingScaleInPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-ProdComputeAutoScalingScaleInPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ExactCapacity # Can use PercentChangeInCapacity but then need to come up with configuration including some estimated change in percent
        Cooldown: 60
        MetricAggregationType: Average # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: !Ref AWS::NoValue
          MetricIntervalUpperBound: 0
          ScalingAdjustment: 1

  # https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-step-scaling-policies.html
  ProdComputeAutoScalingScaleOutPolicy:
    Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
    Condition: CreateProdResources # only create if it is Prod env
    Properties:
      PolicyName: !Sub ${DockerContainerName}-ProdComputeAutoScalingScaleOutPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ProdComputeAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity # ChangeInCapacity —> Increase or decrease the current capacity of the scalable target by the specified value
        Cooldown: 60 # 1 min delay
        MetricAggregationType: Minimum # Valid values are Minimum, Maximum, and Average. If the aggregation type is null, the value is treated as Average.
        StepAdjustments: 
        - MetricIntervalLowerBound: 0  # 0 means exactly equal to Metric Threshold which is 10 defined using SSM parameter
          MetricIntervalUpperBound: 15 # [Metrice Threshold + 15]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 15 # [Metrice Threshold + 15]
          MetricIntervalUpperBound: 25 # [Metrice Threshold + 25]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 25 # [Metrice Threshold + 25]
          MetricIntervalUpperBound: 35 # [Metrice Threshold + 35]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 35 # [Metrice Threshold + 35]
          MetricIntervalUpperBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: 45 # [Metrice Threshold + 45]
          ScalingAdjustment: 1

# ######### https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html #########
#                                       (Total CPU units used by tasks in service) x 100
# Service CPU utilization =  ----------------------------------------------------------------------------
#                            (Total CPU units specified in task definition) x (number of tasks in service)

  NonProdCPUNoComputeAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdCPUNoComputeAlarm
      AlarmDescription: Alarm if container utilize low CPU based on specified threshold!
      Namespace: AWS/ECS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: CPUUtilization
      Dimensions:
        - Name: ServiceName
          Value:
            Fn::GetAtt:
            - NonProdComputeService
            - Name
        - Name: ClusterName
          Value:
            Ref: ComputeCluster
      Statistic: Average # Not using Sum since the metric is CPUUtilization
      Period: 60  # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: !Ref CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM # setting evaluation period 3 because when there is no task at all usually cpu starts with 0 (this way first evaluation period will already hit even before the task start doing anything) & 2 more as part of taking extra precautions! Change it to 2 if needed but not 1
      Threshold: !Ref CpuUtilizationScaleInAlarmThresholdSSM
      ComparisonOperator: LessThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdNoComputeAutoScalingPolicy

  NonProdInitialQueueDepthAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdInitialQueueDepthAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 60 # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: 1 # Threshold is 1 for initial depth
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdInitialComputeAutoScalingPolicy

  NonProdQueueDepthScaleOutAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateNonProdResources # only create alarm if it is NonProd env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-NonProdQueueDepthScaleOutAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 120 # 120 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: !Ref QueueDepthScaleOutAlarmThresholdSSM # change this as needed
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: NonProdComputeAutoScalingScaleOutPolicy

  ProdCPUScaleInAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateProdResources # only create alarm if it is Prod env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-ProdCPUScaleInAlarm
      AlarmDescription: Alarm if container utilize low cpu based on specified threshold!
      Namespace: AWS/ECS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: CPUUtilization
      Dimensions:
        - Name: ServiceName
          Value:
            Fn::GetAtt:
            - ProdComputeService
            - Name
        - Name: ClusterName
          Value:
            Ref: ComputeCluster
      Statistic: Average # Not using Sum since the metric is CPUUtilization
      Period: 60  # 60 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: !Ref CpuUtilizationNoComputeOrScaleInAlarmEvaluationPeriodsSSM # setting this 3 as part of taking extra precautions! Change it to 1 if needed
      Threshold: !Ref CpuUtilizationScaleInAlarmThresholdSSM 
      ComparisonOperator: LessThanOrEqualToThreshold
      AlarmActions:
        - Ref: ProdComputeAutoScalingScaleInPolicy

  ProdQueueDepthScaleOutAlarm:
    Type: AWS::CloudWatch::Alarm
    Condition: CreateProdResources # only create alarm if it is Prod env
    Properties:
      AlarmName: !Sub ${DockerContainerName}-ProdQueueDepthScaleOutAlarm
      AlarmDescription: Alarm if queue depth grows beyond specified threshold!
      Namespace: AWS/SQS # AWS::CloudWatch::Alarm.Period >= 60 for metrics in the AWS/ namespace
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value : !GetAtt SQSQueue.QueueName
      Statistic: Sum
      Period: 120 # 120 seconds ( Period must be 10, 30 or a multiple of 60 but 10 and 30 can not be used with namespaces with the following prefix: AWS/ )
      EvaluationPeriods: 1
      Threshold: !Ref QueueDepthScaleOutAlarmThresholdSSM # change this as needed
      ComparisonOperator: GreaterThanOrEqualToThreshold
      AlarmActions:
        - Ref: ProdComputeAutoScalingScaleOutPolicy

In the above template, I have created two sets of resources NonProd and Prod. In lower environments, the DesiredCount of ECS service is set as zero to save cost. Since CloudWatch Alarm takes at least one minute to respond to scaling events and in prod we do not want any delay that's why the DesiredCount is set as one in prod.
One more thing to note is that the scale-out is based on queue depth but scale-in is based on CPU utilization because the consumer-service consumes a message which can take several minutes to finish and I don't want Application Auto-Scaling to reduce tasks when the queue is empty and they are still processing something. In this regard, EC2 has something called
instance scale-in protection which allows you to have control over which queue workers are terminated when your Auto Scaling group scales in. It was not available for Fargate based ECS clusters but AWS just recently introduced a new feature for ECS called task scale-in protection. You can see the blog post here.

Here is the bash script for creating SSM parameters.

#!/usr/bin/env bash

while (($# > 1)); do
  case $1 in
  --profile) PROFILE="$2" ;;
  *) break ;;
  esac
  shift 2
done

echo "Deleting Parameters..."
aws ssm delete-parameter --profile $PROFILE --name "/$PROFILE/services/consumer-service/AWS_REGION"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/FARGATE_SCALING_ENV"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS"
aws ssm delete-parameter --profile $PROFILE --name "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY"

echo "Creating parameters..."
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/'$PROFILE'/services/consumer-service/AWS_REGION", "Value": "us-east-1"}'
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/FARGATE_SCALING_ENV", "Value": "non-prod"}' # valid values are: non-prod or prod
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/QUEUE_DEPTH_SCALE_OUT_ALARM_THRESHOLD", "Value": "10"}' # if you are increasing this then make sure you are also adjusting step scaling criteria
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/CPU_UTILIZATION_SCALE_IN_ALARM_THRESHOLD", "Value": "2"}' # 2% 
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/CPU_UTILIZATION_NO_COMPUTE_OR_SCALE_IN_ALARM_EVALUATION_PERIODS", "Value": "3"}' # do not set this as 1 in non-prod
aws ssm put-parameter --profile $PROFILE --overwrite --cli-input-json '{"Type": "String", "Name": "/config/ecs/consumer-service/AUTO_SCALING_TARGET_MAX_CAPACITY", "Value": "6"}' # put 1 if you want to disable autoscaling at all

ECS auto-scaling based on Amazon MQ for RabbitMQ’s queue depth.

Muhammad Ahmad Khan — Tue, 01 Nov 2022 15:15:40 +0000

There is a very common use case in the industry about scaling the application based on queue depth. In this regard, AWS itself has published the documentation which can be found here.

In this blog post we will discuss about ECS auto-scaling based on queue depth through the use of AWS application auto-scaling.

There are three solutions you can use with AWS application auto-scaling.

Target Tracking (Policy)
Step Scaling (Policy)
Scheduled scaling (Action)

Here we will discuss about first two only.

Target Tracking Scaling Policies:

Target tracking scaling policy increases or decreases the number of tasks that your service runs based on a target value for a specific CloudWatch metric (predefined or customized).
Target tracking scaling policy assumes that it should scale out your ECS tasks when the specified metric is above the target value and scale in when it is below the target value.
In short, in target tracking scaling policies, Amazon ECS creates and manages the CloudWatch alarms that trigger the scaling policy and calculates the scaling adjustment based on the CloudWatch metric and the target value that you specify. With target tracking, AWS controls the scaling adjustments automatically based on your target specified.

Step Scaling Policies:

Step scaling policy increases or decreases the number of tasks that your service runs by considering the set of scaling adjustments, known as step adjustments, which means you can set multiple actions to vary the scaling depending on the size of the alarm breach.
In short, step scaling policies increase or decrease current capacity based on a set of scaling adjustments (known as step adjustments) that you specify. With step scaling, you control the scaling adjustments.

When working with both policies, it must be noted that you can easily scale your application based on queue depth using step scaling but not with target scaling because target scaling will only allow these three predefined metrics in ECS.

ALBRequestCountPerTarget (load balancer metric)
ECSServiceAverageCPUUtilization
ECSServiceAverageMemoryUtilization

Further, since in target scaling AWS itself create CloudWatch Alarm and manage its scaling therefore you do not have the autonomy to set your desired metric for scaling without creating a custom metric. Thus you need to make a custom metric as described in AWS documentation.

In order to monitor the number of unconsumed message with SQS we can use metric named as ApproximateNumberOfMessagesVisible and if we want to monitor messages from Amazon MQ for RabbitMQ we can use metric named as MessageReadyCount.
One more problem with using a CloudWatch SQS metric like ApproximateNumberOfMessagesVisible for target tracking is that the number of messages in the queue might not change proportionally to the number of tasks within a service that processes messages from the queue. That's because the number of messages in your SQS queue does not solely define the number of tasks needed. The number of tasks in your ECS service can be driven by multiple factors, including how long it takes to process a message and the acceptable amount of latency (queue delay).
AWS proposes a solution to this problem by introducing a backlog per instance metric with the target value being the acceptable backlog per instance.

In AWS documentation, it has stated that:
“Backlog per instance: To calculate your backlog per instance, start with the ApproximateNumberOfMessages queue attribute to determine the length of the SQS queue (number of messages available for retrieval from the queue). Divide that number by the fleet's running capacity, which for an Auto Scaling group is the number of instances in the InService state, to get the backlog per instance.
Acceptable backlog per instance: To calculate your target value, first determine what your application can accept in terms of latency. Then, take the acceptable latency value and divide it by the average time that an EC2 instance takes to process a message.”

In other words, backlog per instance is the value obtained by dividing the number of messages yet to be consumed within the queue with the total number of tasks running at the moment.
Similarly, acceptable backlog per instance is the target value that we use in our scaling policy, and it is the acceptable latency value divided by the average time that an ECS task takes to process a message.
This acceptable backlog per instance (target value) varies from application to application and use case to use case.

Example from AWS documentation:
“As an example, let's say that you currently have an Auto Scaling group with 10 instances and the number of visible messages in the queue (ApproximateNumberOfMessages) is 1500. If the average processing time is 0.1 seconds for each message and the longest acceptable latency is 10 seconds, then the acceptable backlog per instance is 10 / 0.1, which equals 100 messages. This means that 100 is the target value for your target tracking policy. When the backlog per instance reaches the target value, a scale-out event will happen. Because the backlog per instance is already 150 messages (1500 messages / 10 instances), your group scales out, and it scales out by five instances to maintain proportion to the target value.”

Following the AWS documentation, I have created a lambda that creates a custom metric in a custom namespace named as “Queue Based Scaling Metrics”. But in this scenario, our application is using AmazonMQ for RabbitMQ rather than SQS.

See the simple flow of architecture in the below diagram.

In our lambda, we will calculate all the values for our custom metric including backlog per instance as well as acceptable backlog per instance as mentioned in AWS documentation.
For this purpose, we need to set two parameter values (acceptable_latency, time_process_per_message) as mentioned above and these parameters will vary from application to application. In my scenario, I have set these values as follows:

acceptable latency = 250
time process per message = 0.8

Here is the lambda code for all the calculations:

src/app.py

import boto3
import dateutil
from datetime import date, timedelta, datetime

def lambda_handler(event, context):
    cw_client = boto3.client('cloudwatch')
    ecs_client = boto3.client('ecs')
    cluster_name = event["cluster_name"]
    service_name = event['service_name']
    mq_cluster_name = event["mq_cluster_name"]
    mq_queue_name = event["mq_queue_name"]
    acceptable_latency = (event["acceptable_latency"])
    time_process_per_message = (event["time_process_per_message"])
    queue_attribute_calculation(cw_client, ecs_client, cluster_name, service_name, mq_cluster_name, mq_queue_name, acceptable_latency,
                                time_process_per_message)


def queue_attribute_calculation(cw_client, ecs_client, cluster_name, service_name, mq_cluster_name, mq_queue_name, acceptable_latency,
                                time_process_per_message):
    response = ecs_client.describe_services(cluster=cluster_name, services=[service_name])
    running_task_count = response['services'][0]['runningCount']
    print("Running Task: " + str(running_task_count))

    yesterday=date.today() - timedelta(days=2)
    tomorrow=date.today() + timedelta(days=1)
    response = cw_client.get_metric_data(
        MetricDataQueries=[
            {
                'Id': 'mq1',
                'MetricStat': {
                    'Metric': {
                        'Namespace': 'AWS/AmazonMQ',
                        'MetricName': 'MessageReadyCount',
                        'Dimensions': [
                            {
                                'Name': 'Broker',
                                'Value': mq_cluster_name
                            },
                            {
                                'Name': "VirtualHost",
                                'Value': "/"
                            },
                            {
                                'Name': "Queue",
                                'Value': mq_queue_name
                            }
                        ]
                    },
                    'Period': 1,
                    'Stat': 'Sum',
                    'Unit': 'Count'
                },
            },
        ],
        StartTime=datetime(yesterday.year, yesterday.month, yesterday.day),
        EndTime=datetime(tomorrow.year, tomorrow.month, tomorrow.day)
    )

    queue_size =  response['MetricDataResults'][0]['Values'][0]

    print("Running Task: " + str(running_task_count))
    print("Queue Message Count (per second): " + str(queue_size))

    """
    Backlog Per Capacity Unit = Queue Size (MessageReadyCount) / Running Capacity of ECS Task Count
    """
    try:
        backlog_per_capacity_unit = int(int(queue_size) / running_task_count)
    except ZeroDivisionError as err:
        print('Handling run-time error:', err)
        backlog_per_capacity_unit = 0
    print("Backlog Per Capacity Unit: " + str(backlog_per_capacity_unit))

    """
    Acceptable backlog per capacity unit = Acceptable Message Processing Latency (seconds) / Average time to Process a Message each Task (seconds)
    """
    acceptablebacklogpercapacityunit = float((int(acceptable_latency) / float(time_process_per_message)))
    print("Acceptable backlog per capacity unit: " + str(acceptablebacklogpercapacityunit))

    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'MessageReadyCount', int(queue_size),
                  'Queue Based Scaling Metrics')
    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'BackLogPerCapacityUnit', backlog_per_capacity_unit,
                  'Queue Based Scaling Metrics')
    putMetricToCW(cw_client, 'AmazonMQ', mq_cluster_name, 'Queue', mq_queue_name, 'AcceptableBackLogPerCapacityUnit', acceptablebacklogpercapacityunit,
                  'Queue Based Scaling Metrics')


def putMetricToCW(cw, dimension_name, dimension_value, dimension_sub_attribute_name, dimension_sub_attribute_value, metric_name, metric_value, namespace):
    cw.put_metric_data(
        Namespace=namespace,
        MetricData=[
            {
                'MetricName': metric_name,
                'Dimensions': [
                    {
                        'Name': dimension_name,
                        'Value': dimension_value
                    },
                    {
                        'Name': dimension_sub_attribute_name,
                        'Value': dimension_sub_attribute_value
                    }
                ],
                'Timestamp': datetime.now(dateutil.tz.tzlocal()),
                'Value': metric_value
            }
        ]
    )

Note that in order to get AmazonMQ for RabbitMQ metric (MessageReadyCount) I have used CloudWatch client SDK call because I was unable to find any API call to get this metric directly from AmazonMQ for RabbitMQ broker client SDK.

I have deployed this lambda using SAM template and here is the solution for it.
template.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Stack which creates a lambda function that is invoked by a CloudWatch Event Rule every minute to update custom metric datapoints through aws sdk calls.

Parameters:
  Env:
    Type: String
    Description: Values can be dev, qa, devops, test, staging and prod respectively
  ClusterNameParameter:
    Type: String
  ServiceNameParameter:
    Type: String
  MqClusterNameParameter:
    Type: String
  MqQueueNameParameter:
    Type: String
  AcceptableLatencyParameter:
    Type: Number
  TimeProcessPerMessageParameter:
    Type: Number
  SecurityGroupIdParameter:
    Type: String
  SubnetIdAParameter:
    Type: String
  SubnetIdBParameter:
    Type: String
  SubnetIdCParameter:
    Type: String

Resources:
  CustomMetricRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
        - 'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - 'lambda.amazonaws.com'
                - 'events.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: 'LambdaBasicExecutionPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: LambdaExecutionPolicy
                Effect: Allow
                Action:
                - ec2:Describe*
                - ec2:CreateNetworkInterface
                - ec2:DeleteNetworkInterface
                - ec2:DescribeNetworkInterfaces
                - cloudwatch:GetMetricData
                - cloudwatch:PutMetricData
                - "ecs:*"
                Resource: '*'
              - Sid: ExtraPolicy
                Effect: Allow
                Action:
                - ssm:GetParameters
                - ssm:GetParameter
                Resource: '*'
        - PolicyName: 'LogPolicy'
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'

  CustomMetricFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      FunctionName: !Sub create-custom-metric-for-scaling-${Env}-cluster # FunctionName uses Env variable to identify its belonging to the Env specific cluster
      Description: A function that is invoked by a CloudWatch Event every minute to update custom metric datapoints through aws sdk calls.
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.9
      Timeout: 900
      Role: !GetAtt CustomMetricRole.Arn
      Events:
        CloudWatchEventRule:
          Type: Schedule
          Properties:
            Input: !Sub '{"cluster_name": "${ClusterNameParameter}","service_name": "${ServiceNameParameter}","mq_cluster_name": "${MqClusterNameParameter}","mq_queue_name": "${MqQueueNameParameter}","acceptable_latency": "${AcceptableLatencyParameter}","time_process_per_message": "${TimeProcessPerMessageParameter}"}'
            # Read more about schedule expressions here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
            Schedule: cron(* * * * ? *) # Invoke function every minute
      VpcConfig:
        SecurityGroupIds:
          - !Ref SecurityGroupIdParameter
        SubnetIds:
          - !Ref SubnetIdAParameter
          - !Ref SubnetIdBParameter
          - !Ref SubnetIdCParameter

Here is the SAM template deployment command:
First, create the SAM template’s parameter file.

.sam-params-dev

Env=dev
ClusterNameParameter=dev-worker-cluster
ServiceNameParameter=dev-worker-service
MqClusterNameParameter=scan-dev
MqQueueNameParameter=scans_dev
AcceptableLatencyParameter=250
TimeProcessPerMessageParameter=0.8
SecurityGroupIdParameter=sg-123456789
SubnetIdAParameter=subnet-123456789
SubnetIdBParameter=subnet-123456789
SubnetIdCParameter=subnet-123456789

sam deploy --stack-name create-custom-metric --template-file template.yml --s3-bucket test-deployment-bucket --capabilities CAPABILITY_IAM --region us-east-1 --parameter-overrides $(cat .sam-params-dev)

Finally, we will use AWS CLI to put the auto-scaling policy with a target value on ECS.
First, create the JSON policy.

config-dev.json

{
    "TargetValue":312.5,
    "ScaleOutCooldown":1,
    "ScaleInCooldown":1,
    "CustomizedMetricSpecification":{
       "MetricName":"BackLogPerCapacityUnit",
       "Namespace":"Queue Based Scaling Metrics",
       "Dimensions":[
          {
             "Name":"AmazonMQ",
             "Value":"scan-dev"
          },
          {
            "Name":"Queue",
            "Value":"scans_dev"
         }
       ],
       "Statistic":"Average"
    }
 }

In above policy, the target value is Acceptable backlog per instance. This means it is:
acceptable latency/time process per message = 250/0.8 = 312.5 BackLogPerCapacityUnit is the custom metric that we have created through our lambda in Queue Based Scaling Metrics namespace.
scan-dev is the RabbitMQ cluster name and scans_dev is the queue name.

aws application-autoscaling register-scalable-target \ --service-namespace ecs \ --scalable-dimension ecs:service:DesiredCount \ --resource-id service/dev-worker-cluster/dev-worker-service \ --min-capacity 1 \ --max-capacity 10 \ --profile dev

aws application-autoscaling put-scaling-policy \ --policy-name ecs-scaling-based-on-message-consumption-rate \ --service-namespace ecs \ --resource-id service/dev-worker-cluster/dev-worker-service \ --scalable-dimension ecs:service:DesiredCount \ --policy-type TargetTrackingScaling \ --target-tracking-scaling-policy-configuration file://config-dev.json \ --profile dev

We can automate the creation of the application auto-scaling policy with Terraform and CloudFormation if required.
Note that we are just applying the autoscaling policy and AWS will create CloudWatch Alarm itself.

See the git repository here.

Forem: Muhammad Ahmad Khan

From Static to Runtime: Choosing Between CloudFront + S3, Amplify, and ECS for Frontends on AWS

Option 1 — CloudFront + S3

Why teams start here

Where the model starts to hurt

Option 2 — Amplify Hosting

The middle ground many teams should evaluate earlier

Why this is not just "CloudFront + S3 with nicer buttons"

Option 3 — ECS on Fargate

When the frontend becomes part of the application platform

The thing Next.js teams underestimate in containers

A simple decision matrix

What I would actually choose

Amazon ECS Deployment Strategies: Legacy CodeDeploy and the Native ECS Controller

The one distinction that clears up most confusion

Quick comparison: what changes in the traffic layer?

Full comparison diagram

Part 1: The legacy model — CodeDeploy blue/green for Amazon ECS

Legacy diagram 1 — CodeDeploy blue/green, all-at-once

Legacy diagram 2 — CodeDeploy blue/green, canary

Legacy diagram 3 — CodeDeploy blue/green, linear

Part 2: The modern model — native Amazon ECS service deployments

Native strategy 1 — Rolling

Native strategy 2 — Blue/Green

Native strategy 3 — Canary

Native strategy 4 — Linear

What actually changes between rolling and blue/green-style deployments?

What I would choose in practice

The most common mistakes to avoid

Where to go next

Monitoring & Alerting for AWS EKS Using Grafana, Prometheus & Alertmanager with SNS Integration

Prerequisites

Step 1: Install the kube-prometheus-stack Using Helm

1.1 Add the Prometheus Community Helm Repository

1.2 Deploy the kube-prometheus-stack

1.3 Access Prometheus and Alertmanager UIs

Step 2: Set Up Amazon SNS for Alertmanager

2.1 Create an SNS Topic

2.2 Create an Email Subscription for SNS

2.3 Create a New IAM Role for EKS Node Group to Assume and Publish to SNS

2.3.1 Create a Trust Relationship

2.3.2 Attach Permissions to Publish to SNS

2.4 Create a Resource-Based Policy for SNS

2.5 Update the IAM Role for the EKS Node Group

Step 3: Configure Alertmanager to Send Alerts to SNS

Step 4: Configure custom rules for custom alerts

Conclusion

Rapidly Deploy ECS Infrastructure on AWS with AWS CDK (TypeScript)

Prerequisites

Setting Up the Project

Using aws-cdk-lib/aws-ecs-patterns

Defining the ECS Fargate Infrastructure

Example: ECS Fargate Service with Application Load Balancer

Breakdown of the Code:

Deploying the Stack

Step 1: Install Dependencies

Step 2: Bootstrap the CDK Environment

Step 3: Synthesize the Stack

Step 4: Deploy the Stack

Step 4: Verify the Deployment

Conclusion

GitHub Repository

Integrating Multiple EKS Clusters with ArgoCD for Simplifying Kubernetes Operations

Why Integration is Required

How to Integrate Multiple AWS EKS Clusters in ArgoCD

Upgrading EKS Version with Zero Downtime Incorporating Custom Blue-Green Methodology

Distributing restricted static content through CloudFront using signed cookies.

SQS depth based ECS task auto-scaling using step scaling.

ECS auto-scaling based on Amazon MQ for RabbitMQ’s queue depth.

Target Tracking Scaling Policies:

Step Scaling Policies:

Step 1: Install the `kube-prometheus-stack` Using Helm

1.2 Deploy the `kube-prometheus-stack`

Using `aws-cdk-lib/aws-ecs-patterns`