Forem: SHAJAM

What can we learn from me-central-1 region outage

SHAJAM — Fri, 06 Mar 2026 04:23:16 +0000

Cloud outages always trigger the same conversation: "Is the cloud really reliable?" As someone who has spent years designing distributed systems and writing about cloud architecture, I see outages differently. They are case studies. They show us exactly where our assumptions about resilience break down.

The recent outage in the AWS Middle East (UAE) – me-central-1 region is a great reminder of a simple truth many architects intellectually know but don't always design for:

A cloud region is a failure domain.

Even when a provider advertises multiple AZs, a regional event can still cascade across services. If you build everything inside a single region, you are still accepting regional risk.

"Multi-AZ" Is Not the Same as "Highly Available"

Most production workloads proudly claim they are deployed across multiple Availability Zones. That is good practice — but it is not the same as regional resilience.

Availability Zones protect against:

Data centre failure
Power or networking issues within a zone
Localized infrastructure faults

They do not protect against:

Regional control plane failures
Regional networking issues
Identity or API failures affecting the whole region
Large-scale operational events

When the region itself has issues, every AZ can become unavailable at the same time.

Architectural takeaway

Design critical systems assuming:

Region failure = possible

That means evaluating whether your workload should support:

Multi-region failover
Active-active regional deployment
Regional evacuation playbooks

Control Planes Are Hidden Single Points of Failure

One thing outages repeatedly reveal is the difference between control plane and data plane resilience.

Even if compute instances are technically healthy, problems in the control plane can break systems in subtle ways:

Auto-scaling stops working
IAM authentication fails
Load balancers stop provisioning
Container orchestration cannot schedule workloads

Your application may be running, but operations around it are crippled.

Architectural takeaway

Design so your application can continue operating even when:

Scaling APIs fail
Infrastructure automation cannot run
New resources cannot be provisioned

This usually means:

Pre-provisioning capacity buffers
Avoiding dependency on real-time infrastructure changes
Ensuring applications degrade gracefully

Multi-Region Is Still the Gold Standard for Critical Systems

Cloud providers rarely experience full regional outages, but they do happen.

Organizations with true multi-region architectures typically see far smaller impacts during these events.

The three common patterns I see in mature systems are:

Active-Passive

Primary region serves traffic
Secondary region stays warm
Failover triggered by DNS or traffic routing

Pros:

Cheaper
Simpler

Cons:

Failover time may be minutes

Active-Active

Traffic is distributed across multiple regions simultaneously.

Pros:

No cold standby
Instant resilience
Better global latency

Cons:

Data consistency challenges
Higher cost
Operational complexity

Pilot Light

Minimal services run in secondary region, expanded during failover.

Pros:

Cost efficient

Cons:

Recovery time longer
Operational risk during scale-up

Regional Dependencies Are Often Hidden

Many outages expose something architects forget to model: implicit regional dependencies.

Examples include:

Identity services
DNS resolution
Secrets managers
Container registries
Monitoring pipelines

Your application may appear multi-region, but if authentication, secrets, or images live in one region, you have a hidden single point of failure.

Architectural takeaway

Audit dependencies in three layers:

Application dependencies
Platform dependencies
Operational dependencies

Your system is only as resilient as the weakest layer.

Monitoring and Observability Need Regional Awareness

Another common pattern during outages is monitoring blind spots.

Many teams run:

Logging
Metrics
Alerting
dashboards

—all in the same region as their application.

When the region fails, visibility disappears at the exact moment you need it most.

Architectural takeaway

For critical systems:

Send metrics to another region
Maintain external uptime checks
Keep incident tooling outside the impacted region

Runbooks Matter More Than Architecture

Architecture is important, but during outages execution matters more.

Organizations that handle incidents well usually have:

Clear regional failover procedures
Automated traffic switching
Regular disaster recovery drills
Defined decision authority

Without practice, even the best architecture can fail during an emergency.

Cost Optimization Often Competes with Resilience

One uncomfortable truth: many architectures stay single-region because multi-region costs more.

Extra regions mean:

Duplicate infrastructure
Data replication
Additional operational complexity

But outages like this remind us that resilience is a business decision, not purely a technical one.

The real question is:

What is the cost of downtime compared to the cost of redundancy?

For some systems the answer is obvious. For others, it requires honest discussion with stakeholders.

Conclusion

Cloud outages are not failures of cloud computing. They are reminders that distributed systems are still systems, and every system has failure modes.

The me-central-1 outage reinforces a few timeless lessons:

Regions are failure domains
Multi-AZ is not multi-region
Hidden dependencies break resilience
Observability must survive outages
Runbooks matter as much as architecture

The real measure of a cloud architecture is not whether it avoids outages — that’s impossible.

It's how gracefully it survives them.

If you're a cloud architect, moments like this are an opportunity to revisit your assumptions and ask one uncomfortable question:

"What happens if my region disappears right now?"

If the answer is "we're not sure", it might be time to redesign.

Advance Routing Policy in AWS Cloud WAN

SHAJAM — Fri, 06 Mar 2026 04:00:39 +0000

AWS Cloud WAN is a managed wide-area networking (WAN) service from AWS. It lets you build, manage, and monitor a unified global network that spans both cloud and on-premises environments. In practice, Cloud WAN lets you connect your data centres, branch offices, remote sites, and AWS cloud resources (e.g. VPCs) through a central control plane — instead of manually wiring together many VPCs, VPNs, Transit Gateways, and third-party SD-WANs.

To learn more about Cloud WAN, read my previous post on Connectivity using Cloud WAN.

Recently, AWS has announced advanced routing feature for Cloud WAN. Let's explore Segment routing policy in this post.

Let's assume, we have two segments -> DEVELOPMENT and PRODUCTION and we have the policy.

{
  "core-network-configuration": {
    "vpn-ecmp-support": true,
    "asn-ranges": [
      "64520-64524"
    ],
    "edge-locations": [
      {
        "location": "us-west-2",
        "asn": 64521
      },
      {
        "location": "us-east-1",
        "asn": 64522
      }
    ]
  },
  "version": "2025.11",
  "attachment-policies": [
    {
      "rule-number": 100,
      "action": {
        "association-method": "tag",
        "tag-value-of-key": "SEGMENT"
      },
      "conditions": [
        {
          "type": "tag-exists",
          "key": "SEGMENT"
        }
      ]
    }
  ],
  "segments": [
    {
      "name": "DEVELOPMENT",
      "require-attachment-acceptance": false,
      "edge-locations": [
        "us-west-2"
      ]
    },
    {
      "name": "PRODUCTION",
      "require-attachment-acceptance": true,
      "edge-locations": [
        "us-west-2", "us-east-1"
      ]
    }    
  ],
  "segment-actions": [
    {
      "mode": "attachment-route",
      "segment": "PRODUCTION",
      "action": "share",
      "share-with": [
        "DEVELOPMENT"
      ]
    }
  ]
}

This is essentially a cut down version of the policy I shared in the previous post. In this policy, PRODUCTION and DEVELOPMENT are shared, all routes are shared between the two segments.

Now, let us change the policy and allow only certain CIDR ranges from PRODUCTION to DEVELOPMENT and vice-versa. Specifically, we only want to allow

10.200.1.0/20 from PRODUCTION to DEVELOPMENT
10.200.2.0/20 from DEVELOPMENT to PRODUCTION

To achieve this, we can use the routing policy as per below.

"routing-policies": [
    {
      "routing-policy-name": "ProductionToDevelopment",
      "routing-policy-direction": "outbound",
      "routing-policy-number": 1,
      "routing-policy-rules": [
        {
          "rule-number": 1,
          "rule-definition": {
            "match-conditions": [
              {
                "type": "prefix-equals",
                "value": "prefix-in-cidr"
              }
            ],
            "condition-logic": "or",
            "action": {
              "type": "allow"
            }
          }
        },
        {
          "rule-number": 2,
          "rule-definition": {
            "match-conditions": [
              {
                "type": "prefix-in-cidr",
                "value": "0.0.0.0/0"
              }
            ],
            "condition-logic": "or",
            "action": {
              "type": "drop"
            }
          }
        }
      ]
    }

This policy allows 10.200.1.0/20 from PRODUCTION to DEVELOPMENT and denies all other traffic. You will need to apply a similar policy for DEVELOPMENT to PRODUCTION. It is important to note the route direction and it is set to outbound. It took me a bit of trialling to figure out the direction.

Now, lets say you have multiple direct connect gateways and and multiple default routes are appearing in the same segment, PRODUCTION. The PRODUCTION segment is extended across two regions - us-east-1 and us-west-2. That is, route 0.0.0.0/0 are appearing from the DX gateways.

By default, the same route might be preferred in PRODUCTION segment. We want to make sure, that PRODUCTION segment in us-east-1 get routes from the DX gateway in us-east-1 and vice versa.

Again, we can use routing policy and attachment policy to achieve this. Lets add a routing policy.

{
      "routing-policy-name": "AllowUsEast1",
      "routing-policy-description": "Allow us-east-1 DX G only",
      "routing-policy-direction": "inbound",
      "routing-policy-number": 21,
      "routing-policy-rules": [
        {
          "rule-number": 100,
          "rule-definition": {
            "match-conditions": [
              {
                "type": "prefix-equals",
                "value": "0.0.0.0/0"
              },
              {
                "type": "asn-in-as-path",
                "value": 64600
              }
            ],
            "condition-logic": "and",
            "action": {
              "type": "set-local-preference",
              "value": "300"
            }
          }
        }
      ]
    }

This is an inbound routing policy. It's checking for the default route 0.0.0.0/0 and ASN (ASN from DX gateway). When this matches, the local preference is set to 300. So, what happens is when the route matches prefix and ASN, a local preference is set. Cloud WAN prefers a higher local preference.

Now, you will need an attachment policy.

"attachment-routing-policy-rules": [
    {
      "rule-number": 1,
      "edge-locations": [
        "us-east-1"
      ],
      "conditions": [
        {
          "type": "routing-policy-label",
          "value": "DXDefault"
        }
      ],
      "action": {
        "associate-routing-policies": [
          "AllowUsEast1"
        ]
      }
    }
]

The attachment policy is set for the attachment of the DX gateway. The routing policy label needs to be applies to the attachment as well.

# CloudFormation snippet
Type: AWS::NetworkManager::DirectConnectGatewayAttachment
    Properties:
      CoreNetworkId: !Ref MyCoreNetworkId
      DirectConnectGatewayArn: !Ref MyDxGateway
      EdgeLocations: 
        - us-east-1
      RoutingPolicyLabel: DXDefault

You will need similar policies for us-west-2 location as well. Once this is applied, the route 0.0.0.0/0 will be propagated from the regional DX gateways.

AWS Interconnect: The New Era of Multicloud Connectivity

SHAJAM — Fri, 12 Dec 2025 22:02:00 +0000

Quick Summary: AWS Interconnect is a brand-new managed service that gives you private, high-speed connectivity between AWS and other cloud providers (starting with Google Cloud). This post breaks down what it is, why it matters, and how it changes multicloud networking.

🧠 Introduction: Cloud Networking Just Grew Up

Multicloud has always sounded cool:
Run workloads wherever they run best. Mix AWS, Google Cloud, maybe even Azure.
But in reality? Connecting clouds privately was a headache. VPNs, tunnels, carrier circuits, routing drama… no thanks.

AWS just launched AWS Interconnect, and it finally feels like someone fixed multicloud networking.

This blog breaks down:

What AWS Interconnect is
Why it's a big deal
Benefits for builders and architects
How it connects AWS ↔ GCP
A simple diagram you can reuse
Who should care (spoiler: probably you)

🚀 What Is AWS Interconnect?

AWS Interconnect is a newly released managed private connectivity service that links AWS VPCs to other cloud providers over dedicated, high-speed, encrypted connections.

No more dealing with:

Physical routers
Colocation cross-connects
Dozens of tickets
DIY BGP wrangling

AWS handles the physical infrastructure, routing, and resiliency — you just select your cloud, region, and bandwidth.

🌍 Why Now? A Quick Cloud Evolution Flashback

A few years ago, multicloud meant:

Cobbled-together IPsec tunnels
Unpredictable internet performance
Manual routing
Months to provision circuits
Lots of baby-sitting

Today, apps are more distributed, global, real-time, and hybrid than ever.
We needed a simpler way for clouds to talk to each other.
AWS Interconnect is that solution.

🎉 Benefits of AWS Interconnect

⚡ 1. Faster, More Predictable Traffic

Private backbones beat the public internet every time.
Low latency + high bandwidth = real multicloud apps.

🔐 2. Secure by Default

Everything runs over private physical links with encryption handled for you.

🔧 3. Fully Managed

Provision connections in minutes, not months.
AWS manages the physical layer so you don't have to.

🔗 4. True Multicloud Workflows

Run compute in one cloud, analytics in another, DR in a third — without network pain.

🔄 5. Cross-Cloud Interconnect (Starting With GCP)

Google Cloud and AWS are now offering coordinated APIs for seamless connectivity.
This used to be a unicorn.
Now it's real.

🔁 How AWS Interconnect Connects AWS ↔ Google Cloud

Here's a high-level view of how data flows:

                      +----------------------+
                      |      Google Cloud    |
                      |   VPC / Services     |
                      +----------▲-----------+
                                 │  Private
                                 │  High-Speed
                                 │  Interconnect
                                 ▼
              +----------------------------------------+
              |            AWS Interconnect            |
              |   Multicloud Private Connection        |
              |  (Managed, Encrypted, High-Speed)      |
              +----------------------------------------+
                                 ▲
                                 │  AWS Global
                                 │  Backbone
                                 │
      +----------------------+   +-----------------------+
      |  AWS Transit GW /   |   |   AWS Region VPCs     |
      |  Cloud WAN / VPC    |   |   Your Workloads      |
      +----------------------+   +-----------------------+

🧠 Real-World Use Cases

💾 Cross-Cloud Databases

Sync Aurora ↔ BigQuery privately.

🔄 Multicloud Disaster Recovery

Failover across clouds without internet dependency.

⚙️ Distributed Data Pipelines

Run ML in one cloud, analytics in another.

🛠️ Hybrid Applications

Route traffic between AWS and GCP seamlessly.

⚠️ Notes on Preview Status

AWS Interconnect is still in preview, meaning:

Bandwidth options are limited
Not recommended for production workloads yet
Some features may change before GA
Pricing may evolve

But for early adopters or multicloud architects, this is the most exciting thing AWS has released in years.

🏁 Final Thoughts

AWS Interconnect is a huge step forward in making multicloud actually work.
Not as a buzzword.
Not as a slide in a conference deck.
But for real applications and real workloads.

Private, fast, secure, managed connectivity between clouds is here — and it's only going to grow.

CloudFront Function vs Lambda@Edge - which one to use

SHAJAM — Fri, 12 Dec 2025 09:23:00 +0000

If you've ever played around with Amazon CloudFront and thought, "Hmm, how do I customize stuff at the edge?" you've probably bumped into CloudFront Functions and Lambda@Edge. And then, naturally, you've wondered: "Wait… aren't these basically the same thing?"

Short answer: kinda.
Real answer: not really.

Let's break it down like you're deciding between a skateboard (fast, lightweight) and a motorcycle (heavier, way more powerful).

CloudFront Functions — the Speedy Little Edge Worker

Think of CloudFront Functions as the edge-compute version of a sticky note: super lightweight, super fast, very cheap, but only meant for small, quick tasks.

What they're great at

Super fast viewer-request/response handling Your code runs before CloudFront even touches your request.
Crazy low latency It literally runs at CloudFront POPs — the closest possible place to your users.
Cheap Pennies. Even cheaper than Lambda@Edge.
Simple logic URL rewrites, redirects, header tweaks, basic authentication, A/B testing, that kind of stuff.

**What they can't do**

No access to the origin request/response events.
No network calls.
No file system.
No long-running or complex logic.
No heavy libraries — the runtime is tiny and minimal.

Basically: It's fast because it's limited.

Lambda@Edge — the Heavyweight With Superpowers

Lambda@Edge is the older, more powerful sibling. If CloudFront Functions are a scooter, Lambda@Edge is a touring bike with saddlebags and speakers and a cup holder.

What they're great at

Supports all four CloudFront trigger points Viewer request, viewer response, and origin request/response.
Can run longer and heavier code Want to modify HTML? Do image processing? Make API calls? You're good.
Supports more libraries Way more flexible environment than CloudFront Functions.
Origin-side personalization For example: choose an origin based on cookies or authorization rules.

**What they can't do well**

Not nearly as fast as CloudFront Functions (still fast, but not POP-level fast).
More expensive.
Deployments take longer — think minutes, not seconds.
More operational overhead.

Basically: It can do a lot more, but it costs you more.

⚔️ CloudFront Functions vs Lambda@Edge — The Showdown

Feature	CloudFront Functions	Lambda@Edge
Latency	Ultra-low (runs at POPs)	Low (runs at regional edge)
Cost	Very cheap	More expensive
Runtime limits	Tiny (ms-level)	Bigger (up to seconds)
Triggers	Viewer request/response only	All 4 (viewer + origin)
Use cases	Simple, fast logic	Heavy, complex logic
Supports network calls?	Nope	Yes
Deployment time	Seconds	Minutes

🧭 So which one should YOU use?

Here's the simple rule of thumb:

👉 If the logic is simple, fast, and doesn't need external calls → Use CloudFront Functions.

Examples:

Redirect /old → /new
Rewrite URLs
Add/remove headers
Basic cookie checks
Serve country-based variations
Block bots

👉 If the logic requires more power → Use Lambda@Edge.

Examples:

Fetching data from another API
Heavy header computation
Authentication with external services
Dynamic HTML rewriting
Modifying origin responses
Image transformation or content manipulation

🎯 TL;DR

CloudFront Functions: blazing fast, super cheap, lightweight. Perfect for easy viewer-side logic.
Lambda@Edge: powerful, flexible, and capable of handling complex workloads — but slower and pricier.

You really shouldn't choose based on which one is "newer" or "cooler."
Just ask yourself: Do I need more power or more speed?

Inter-Region Connectivity in AWS using CloudWAN

SHAJAM — Tue, 09 Dec 2025 04:37:28 +0000

Key Concepts & How It Works

Global Network
This is the top-level container — it can hold a core network and any associated attachments (VPCs, VPNs, etc.).

Core Network
The core network is the portion of your Cloud WAN that is managed by AWS — essentially the backbone/fabric that ties your network attachments (VPCs, VPNs, Transit Gateways, on-prem links etc.) together.

Core Network Edge
A Core Network Edge is a regional connection point provisioned and managed by AWS in each AWS Region that you include in your core network via your core-network policy. Under the hood, a CNE is similar to AWS Transit Gateway (TGW) — meaning it acts like a regional router that aggregates and routes traffic — but differs in that it is fully managed by AWS.

Core Network Policy
A policy document (in a declarative language) that states how your network should be configured:

which AWS Regions to include
how attachments map to segments
routing rules
traffic-isolation rules

Cloud WAN then implements the network configuration automatically.

Attachments
These are the resources you hook into your network — like VPCs, VPN connections, Transit Gateways, SD-WAN links, on-premises branch office links.

Network segmentation
Cloud WAN lets you define segments — isolated routing domains. So you can isolate e.g. production, development, branch offices, or sensitive resources even if they share the same overall network backbone.

Central dashboard & automation
You manage everything — global scope, attachments, routing — from a single dashboard (via AWS Network Manager) or using AWS CLI / APIs. Cloud WAN automates the heavy lifting so you don't manually configure every link.

Let's get started

Create the Global Network namespace

Name your Global Network. (AWS > Network Manager > Global network)
Add core network. Can have a max of 1 core network.
Set ASN setting. Set the range. (The value must be a range between 64512-65534 or 4200000000-4294967294.) This is used by AWS for managing traffic between VPCs.
Choose edge locations -> us-east-1, us-west-2...
Set default segment name. This will be available in all edge locations.

Once you create it, the default policy will be created.

{
    "version": "2021.12",
    "core-network-configuration": {
        "asn-ranges": [
            "64520-64524"
        ],
        "edge-locations": [
            { "location": "us-east-1" },
            { "location": "us-west-2" }
        ]
    },
    "segments": [
        {
            "name": "production",
            "description": "production environment"
        }
    ]
}

You can create additional segments and configure attachment policies.

Network Design

[src: https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/aws-cloud-wan.html]

In this diagram, there are multiple regions and multiple segments - Development, Shared and Production. Each region is connected using the CNEs. The segments have network connectivity defined by the core network policy.

Based on the diagram, I have created a sample network policy.

{
  "core-network-configuration": {
    "vpn-ecmp-support": true,
    "asn-ranges": [
      "64520-64524"
    ],
    "edge-locations": [
      {
        "location": "us-east-1",
        "asn": 64520
      },
      {
        "location": "us-west-2",
        "asn": 64521
      }
    ]
  },
  "version": "2021.12",
  "attachment-policies": [
    {
      "rule-number": 100,
      "action": {
        "association-method": "tag",
        "tag-value-of-key": "SEGMENT"
      },
      "conditions": [
        {
          "type": "tag-exists",
          "key": "SEGMENT"
        }
      ]
    }
  ],
  "segments": [
    {
      "name": "SHARED",
      "require-attachment-acceptance": false,
      "edge-locations": [
        "us-east-1",
        "us-west-2"
      ]
    },
    {
      "name": "DEVELOPMENT",
      "require-attachment-acceptance": false,
      "edge-locations": [
        "us-east-1",
        "us-west-2"
      ],
      "deny-filter": [
        "PRODUCTION"
      ]
    },
    {
      "name": "PRODUCTION",
      "require-attachment-acceptance": true,
      "edge-locations": [
        "us-east-1",
        "us-west-2"
      ]
    }    
  ],
  "segment-actions": [
    {
      "mode": "attachment-route",
      "segment": "SHARED",
      "action": "share",
      "share-with": [
        "DEVELOPMENT",
        "PRODUCTION"
      ]
    },
    {
      "mode": "attachment-route",
      "segment": "PRODUCTION",
      "action": "share",
      "share-with": [
        "DEVELOPMENT"
      ]
    }
  ]
}

Understanding the Policy

asn-ranges: You define the asn-ranges where you will later pick the asn for a region.

edge-location: Here you define the edge locations your network will work in.

attachment-policies: This defines the policy for attaching the segments. Here. I am attaching the attachments to the segments based on the tag automatically. The tag key is set to be SEGMENT.

segments: This is a list of segments with edge-location. Note, the DEVELOPMENT segment rule, there is a deny-filter policy to PRODUCTION which makes the traffic uni-directional.

segment-actions: By default, the sharing is bi-directional. That is, for SHARED segment, it is shared with DEVELOPMENT and PRODUCTION. You don't need to share DEVELOPMENT and PRODUCTION with SHARED as it is bi-directional.

More about Attachments

When you build attachments, you can create it for Direct Connect Gateway, VPNs and VPCs.

Direct Connect Gateway

With Direct Connect Gateway, this is a convenient way to connect your Direct Connect to multiple regions. Direct Connect has certain limitations on maximum number of gateways that you can create. With CloudWAN, you just create one, attach to the required edge locations (supports more than 1) and that's it. You never need to worry about Direct Connects.

VPC

With VPCs, we simply add an attachment to the VPC so we could use as part of Core Network as defined by the network policy. One of the coolest feature here is security group referencing which allows referring to security group in one VPC from another, thus making security rules much more streamlined. Previously, I was creating prefix lists for the security group rules, now, you no longer need to do so.

VPN

With Site-to-Site VPN attachment refers to attaching a VPN connection to the core network. In effect, you are bringing an external VPN into your Cloud WAN global network so that it can connect with other attachments.

Conclusion

CloudWAN is an amazing service (or shall I say VPC feature) if you want to centralise global connectivity and connect across the globe. Adding regions and VPCs to core network literally takes less than 30 minutes! This is simply unreal.

Cool Features in AWS CloudFormation

SHAJAM — Thu, 20 Nov 2025 00:25:17 +0000

AWS CloudFormation is an AWS tool that lets you create and manage cloud resources using code instead of clicking around the console. You describe everything you want - like EC2 instances, S3 buckets, IAM roles, VPCs — in a template file (YAML or JSON), and CloudFormation reads that template and automatically provisions, updates, or deletes those resources for you as a single stack. This makes your infrastructure repeatable, version-controlled, and easier to manage, because you can deploy the same setup to multiple environments (dev, test, prod) with minimal changes, roll back if something fails, and track all infrastructure changes just like you do with application code.

Recently, there has been number of improvements in CloudFormation making it much easier to manage infrastructure resources. In this post, I will talk about the improvements and how can you can take advantage of these.

Stack refactoring

With stack refactoring, you can rearrange the resources in your CloudFormation stacks without losing their existing configurations or data. This lets you move resources between stacks, break a large stack into several smaller ones, or merge multiple stacks into a single stack. This is helpful, because, you might have created a stack previously which has grown over time and has too many resources. Now, with stack refactoring, you can rearrange the resources into multiple stacks.

Stack refactoring in CloudFormation is a multi-step process: first you assess your existing stacks to find refactoring opportunities, then plan how to reorganize resources and decide which destination stacks (2–5, including nested stacks) they should move to. Next, you update templates (including moving resource definitions and optionally renaming logical IDs) and create the refactor by supplying stack names and templates. CloudFormation then validates dependencies and IDs, shows a preview if validation passes, or reports issues you must fix (such as providing logical ID mappings for conflicts). Finally, you execute the refactor and monitor the operation to ensure it completes successfully.

Refer Stack refactoring for details.

Validate stack deployments

With pre-deployment validation, you can spot and fix potential deployment problems before running your CloudFormation change sets. This capability checks your templates against common failure scenarios, allowing you to catch and address issues earlier in the development cycle.

Pre-deployment validation automatically checks your CloudFormation change sets before deployment by running syntax, name conflict, and S3 bucket emptiness validations, then highlighting exact issues in your template so you can fix them first and deploy with greater confidence.

When you create a changeset, if all is well, the Deployments validation will be a green light. You can also bake this into a CICD pipeline by using the cli commands for CloudFormation.

### describe events
aws cloudformation describe-events \
  --change-set-id "arn:aws:cloudformation:us-east-1:123456789012:changeSet/MyChangeSet/123456ab-cd12-98ab-6521-987qwe654asd"

You can then inspect the OperationEvents[i].OperationStatus to see if event contains FAILED or SUCCEEDED.

Drift-Aware ChangeSets

Drift-aware change sets are an improved type of CloudFormation change set that help you safely detect and handle stack drift. Stack drift happens when resources are modified outside of CloudFormation - for example, directly through the AWS console, CLI, or service SDKs—so they no longer match the template. Drift-aware change sets compare your templates with the real, current state of stack resources and help you align any drifted resources back with their template definitions. If you update a template so that a resource's definition matches its actual state, drift-aware change sets will clear the resource's drift status without making any changes to that resource.

Note that, all resources are not supported for drift-aware changesets. Refer AWS documentation to find the list of supported resources.

Practical ways to reduce costs of your AWS Lambda Functions

SHAJAM — Fri, 07 Nov 2025 04:28:27 +0000

Are you paying thousands of dollars for running Lambda functions in your AWS accounts. That's a common scenario where you start of Lambda functions as it is cheap but over time when you invoking the Lambda thousands of times, your bills just go high and you can't figure out how to reduce it. In this article, I will talk about steps you can take to reduce the cost.

Identify Source of Cost

Before we can reduce the cost, we need to understand where the cost is coming from. Let's go through a few metrics.

Invocations:
- How often are the Lambda functions being called?
- Is the architecture correct?
- Do you need to call it that often.
Duration:
- How long are the functions running for each invocations?
- Does it need to run for so long?
- You can optimise but updating the memory configuration which allow the Lambda to run faster.
Memory configuration:
- Have you allocated more than needed?

You can also enable AWS X-Ray or Lambda Power Tuning to visualize resource usage and duration.

Right-Size Memory and CPU

Lambda charges based on memory × duration. Increasing memory increases CPU proportionally — but beyond a certain point, you're just paying for idle CPU.

How to optimize:

Use AWS Lambda Power Tuning (by AWS Labs) — it runs tests across different memory configurations.
Find the sweet spot: the smallest memory size that keeps execution time fast enough.

For example: Sometimes downgrading memory from 512 MB to 256 MB doubles runtime but cuts cost 30–40%.

Reduce Invocation Frequency

Batch events together when possible. For example:
- Use SQS batching instead of processing one message per invocation.
- For Kinesis or DynamoDB Streams, increase the batch size.
Add filtering on triggers instead of Lambda (e.g., event filtering for S3 or DynamoDB Streams).
If Lambda is used for cron jobs, ensure they run only when needed.

Avoid "Always On" Patterns

If your function runs continuously or frequently:

Consider AWS Fargate or ECS for long-running processes. That is, run Fargate or ECS continuously instead of invoking Lambda functions.
For high-throughput APIs, consider ECS + ALB or API Gateway + ECR container.
For near-real-time streams, Kinesis Data Analytics or Glue streaming might be more cost-efficient.

Choose Cost Optimised Architecture

By default, most developers build Lambda functions using the Intel architecture, however, you can decrease the running cost by 20-30% by using the arm architecture.

To use arm, no code changes are needed if you use NodeJS or Python.
Minor .NET changes might be needed to use arm architecture.

Optimize Dependencies and Cold Starts

Cold starts can increase duration (and thus cost).

Use smaller deployment packages — avoid large dependencies. You can do so by using Lambda layers. This significantly reduces the size of your function's deployment package and allows for easier dependency management across multiple functions.
Prefer AWS SDK v3 modular imports.
Use Provisioned Concurrency only where latency is critical — otherwise it increases cost.

Use Savings Plan

Free Tier: 1M requests and 400,000 GB-seconds per month are free.
Compute Savings Plan: If Lambda is heavily used, a Savings Plan can cut costs by up to 17%–30%.
Regional pricing differences: Check if multi-region deployments make sense. For example, us-east-1 or eu-west-1 is much cheaper than eu-west-2 or ap-southeast-2.

Do/Don't Run Lambda on VPC

There is no explicit fee for attaching a Lambda to a VPC. However, network configuration side effects can increase costs in these ways:

VPC data transfer: Data transfer charges can incur when traffic is sent between AZs
NAT Gateway data processing fees: If Lambda needs internet, it will use NAT Gateway which in turn will incur processing fees.

Conclusion

As you can see, there are many opportunities for you to reduce the cost of Lambda functions. You might be on different stages in your cost management exercises and choose the step that is more suitable to you.

If you have additional ways to save Lambda's running cost, please suggest in the comments. If this article helped you to reduce cost, let me know.

Yet Another AWS AI Certification - AI Professional

SHAJAM — Thu, 30 Oct 2025 02:59:02 +0000

Introduction

Generative AI is rapidly becoming a business-critical capability. AWS launched a new specialised certification for AI domain as per AI certification. The certification is currently in beta stage.

Note, AWS is also decommissioning the Machine Learning Specialty certification.

What the certification is & who it's for

The AWS Certified Generative AI Developer – Professional is a Professional-level certification.
It is currently in beta (as of the announcement) – registration opens November 18 2025.
Exam overview: 204 minutes, ~85 questions (multiple choice / multiple response). - more questions than usual
Target candidate: developers with 2+ years of cloud experience, plus 1 year of hands-on experience implementing generative AI solutions. Also experience with AWS compute/storage/networking, security, deployment/infrastructure-as-code, monitoring/observability, cost optimisation.
It is aimed at those who are beyond proof-of-concept and can build and deploy production-ready generative AI solutions using AWS services (such as AWS Bedrock) and open-source tools.

Core knowledge & skills validated

You might emphasise what a "generative AI developer" in AWS world needs to know. From the AWS page we have hints; you'd likely want to infer/expand:

Understanding the AWS services and infrastructure for generative AI (e.g., AWS Bedrock, compute/storage/networking, identity/security).
Ability to design and implement generative AI models/solutions: selecting appropriate foundation models or open-source models, fine-tuning or customising them, integrating them with applications.
Deployment and operationalisation: infrastructure-as-code, cost optimisation, monitoring/observability, maintenance of models in production.
Security, governance, and responsible AI: managing identity and access, data privacy, model risk, bias mitigation.
Business value: building solutions that deliver measurable results, not just experiments. Identifying metrics, aligning with organisational goals, cost vs benefit.
Possibly multi-modal/advanced use-cases: text generation, image/video/audio, prompt engineering, RAG, pipelines, evaluation and iteration.

How to prepare (and prerequisites)

While no prior AWS certification is required, AWS suggests it might help to have other certifications such as: AWS Certified AI Practitioner, AWS Certified Solutions Architect – Associate, AWS Certified Machine Learning Engineer – Associate, AWS Certified Data Engineer – Associate.
Recommended to have hands-on experience: 2+ years with cloud and at least 1 year with generative AI solutions.
Preparation tips:
- Get hands-on with AWS generative AI services (e.g., AWS Bedrock, SageMaker JumpStart, etc)
- Study deployment/infrastructure aspects: network, security, observability, cost control
- Study model integration: prompt engineering, retrieval-augmented generation (RAG), evaluation metrics for generative output.
- Work through case studies: how generative AI is applied in businesses (content creation, summarisation, code generation, image/video generation, multimodal assistants)
- Use Official AWS resources: exam guides, sample questions, training courses, labs.
- Simulate environments: build a prototype end-to-end solution in AWS from data ingestion → model integration → production deployment → monitoring.
Mention that because it's a beta exam, early takers will be among the first to hold the certification.

Potential caveats / things to note

As with any certification: passing the exam doesn't automatically make you expert — practical experience still matters.
Being AWS-specific: It is focused on AWS services. So if you are working on other clouds/platforms you may still need to master those.
Because it's a beta exam, details may change as AWS finalises the content and certification.
The pace of generative AI is fast — new techniques emerge quickly — so continuous learning beyond the certification will be key.

Conclusion

The AWS Certified Generative AI Developer – Professional certification gives developers and organisations a structured way to validate generative-AI development expertise on AWS. It reflects the maturity of the generative-AI domain moving into production, and offers a pathway for career growth and organisational readiness.
For those looking to build generative AI solutions in the cloud, it's a timely credential to consider.

Lessons learned from AWS outage - what can you do?

SHAJAM — Thu, 23 Oct 2025 05:49:24 +0000

What was the AWS Outage all about

The outage stemmed from multiple services in the US-East-1 region experiencing increased error rates and latencies, impacting many AWS services simultaneously.

In particular:
The root cause appears to involve the Amazon DynamoDB API endpoint and DNS resolution issues in us-east-1 region.

The cascade effect: although the initial fault was in one internal subsystem, many other services and workloads suffered – because of inter-dependencies and the central role of us-east-1 region.

This is not unique: there has been past AWS regions/outages causing outages to numerous systems around the world.

Key take-aways:
Even with a major cloud provider like AWS, you do not have immunity from downtime. This is specially true for single region deployment with no disaster recovery (DR). A "single region" failure can affect large swathes of services if you rely heavily on that region.

In regulated sectors (like healthcare) the risk is higher because service interruptions can mean patient-safety, regulatory, compliance and reputational damage.

What is so special about us-east-1 region

The outage was related to us-east-1 region, but as you know, many applications had partial or full outages which are hosted in different regions. So, what is special about us-east-1 region?

us-east-1 region is the first AWS region and is like a global region. Many AWS global services are anchored or managed from there. AWS services like IAM, CloudFormation, S3, Route53 and CloudFront have internal dependencies where metadata or management planes are based in us-east-1. Thus, an outage in us-east-1 can cause a partial outage in an application in a different region.

What you can do to make systems resilient

You need to know how critical your application is because the design comes at a cost. You can create multi-cloud or multi-region architecture but these are not cheap. For example, if you have multi-cloud, you need to pay for AWS direct Connect and AWS Azure Express Route separately which is just the start. You still need to deploy to multiple cloud which is also costly. Development cost is also high as you need to deploy different cloud.

So, what about multi-region? Multi-region is cheaper and minimal additional development work is needed. Many AWS services like Aurora database and DynamoDB support multi-region by default. You can also create replica/standby capacity in secondary regions. However, like I mentioned previously, us-east-1 is like a global region where many services like IAM are managed from.

What do you need to do? - Service-dependency mapping

You need to create a Service-dependency mapping. You need to identify critical services for you business and setup a business continuity plan. For each services, you need to map dependencies (e.g., database, authentication service, file store, third-party APIs).

For each dependency, ask: "if this one fails, what happens?" and build mitigation (e.g., caching, queueing, offline degraded mode).

DNS, routing and fail-over:

The AWS event shows how DNS/resolution problems can cascade. Ensure you have resiliency in DNS (multiple providers, health checks). You can use Route 53 for your DNS; AWS provides 100% for this service.

Use health-checks and automated traffic shifting (via load balancers, Route 53 or equivalent) to redirect traffic away from failed zones/regions.

Data replication and backups:

Use cross-region replication (database replicas, file storage replication) so you have standby data in a different region. You need to ensure your backups are recent, tested, and you can restore rapidly into another region if needed.

Testing & exercises: Regularly simulate failure of a region/service (chaos testing) to ensure your fail-over works. Measure RTO (Recovery Time Objective) and RPO (Recovery Point Objective) for each critical business service.

What can you NOT Control

Sure, you can make multi-cloud, multi-region and what not but there are still certain things that you cannot control from business perspective. For example, lets say your application is multi-region, however, if you integrate with a vendor-hosted application, then you resiliency is dependent on that vendor-based product as well. You cannot control whether they use multi-cloud or multi-region and even multi-AZ deployments.

You are more or less relying on what the vendor tells you or is in the contract but it's never easy to calculate RTO when you rely on external services.

Why Multi-Region can be a NO-GO

I see many people posts in LinkedIn that you need to make you application multi-region but the reality is, it's not always easy for a number of reasons.

Cost: It's not cheap - you need budget allocated for this
Compliance: Critical business like healthcare and finance have strict data sovereignty policies. AWS does not have multiple regions in every country, so, multi-region is not always an option.
Complexity: Multi-region is complex. Managing infrastructure in multiple regions and then to keep it in sync in complex. You can it by infrastructure as code but it's still complex.
Automatic Failover: Don't forgot just having multi-region is not enough. You also need to safely failover to the secondary region. You need proper health checks and DNS failover to fail to the secondary region.

Summary

The AWS outage demonstrates that even dominant cloud providers and leading regions can suffer cascading failures spanning many services.

For critical businesses like healthcare or finance, the consequences can be more serious than just "website down" — they can affect patient care, compliance, revenue, and reputation.

To mitigate: adopt multi-region/fail-over architectures, decouple dependencies, maintain backups, exercise fail-over, monitor actively, and have a strong incident response plan.

The goal is not to assume "never will fail" (because failure is inevitable) but to plan for when it does so the impact is minimised.

Sign your AWS Lambda Code

SHAJAM — Fri, 17 Oct 2025 01:27:07 +0000

AWS Lambda is a serverless technology that lets you deploy your application without having to deploy servers. You essentially deploy tour code, configure basic settings like memory, ephemeral storage and IAM role and that's it, you are good to go.

What is Code Signing?

Code signing is a security process that applies a digital signature to software code or executables to prove the authenticity and integrity of your code. So, what does it mean?

You have deployed your code but how do you know the deployed code has not been tampered with? That's the problem Code Signing solves.

Authenticity: the code is deployed by a trusted source like yourself or your company.
Integrity: the code has not been modified since it was signed.
Non-repudiation: the signer like you cannot later deny having signed it.

What is AWS Signer?

AWS Signer is a fully managed code signing service from AWS.
It lets you digitally sign your software, deployment packages, and container images so that AWS services like Lambda or IoT can verify the code's integrity and authenticity before running it.

You can also use AWS Signer from CI/CD pipeline which is critical given most companies deploy code using CI/CD. This helps us confirm that only authorized, untampered code gets deployed.

How Signer Signs Lambda Code

To use AWS Signer, you need to create a signing profile as shown below.

To use AWS Signer, you need to upload the code to a S3 bucket. You can setup the bucket so that the signed and unsigned code are kept in different prefix. Once code is uploaded, you can run a sign job from AWS Signer. This will produce the signed code in the S3 bucket. After that, you can update the Lambda code with the signed code.

You also need to update signing profile setting in Lambda. Under Lambda dashboard, you can set the default configuration for signing. This is shown below.

You can choose one or more signing profiles. Note the Signature validation policy. You can set it to Warn or Enforce. For experimentation, you can start with Warn and eventually change it to Enforce. Once you enforce it, Lambda blocks the deployment request if the signature validation check fails. This ensures that only trusted and signed code is deployed to Lambda functions, providing a stricter security posture.

How to use Signing Profile in Lambda Function

Now, all the settings are done, you just need to start using the signing profile in your Lambda function. To do that, when you create a new Lambda function from console, you need to expand Additional configurations and select Code Signing under Security & governance. This is shown below.

Update Lambda Function to with Signed Code

Ok, we have the signing profiles ready. Now, we need to sign the code. So, as I mentioned before,

Upload code to a S3 bucket
Create a signing job from AWS Signer and choose the right signing profile
Update Lambda function to use the signed code

Note, once you have used signed code in Lambda, you can no longer view the code in AWS Lambda console. You can check the signing profile under Code properties in Lambda.

What Happens When a Signing Certificate Expires

Signer-managed certificates have a validity period that you can set. When a signing certificate expires:

Existing signed artifacts remain valid (they include timestamp metadata proving they were signed during a valid period).
You cannot sign new artifacts using the expired certificate.
You must create a new signing profile and sign again.

This is important for AWS Lambda code signing, since Lambda will reject expired signatures at deployment time.

Conclusion

To wrap up, AWS Lambda code signing brings cryptographic assurance to serverless deployments, ensuring that only verified, untampered code runs in your functions. It's a small implementation effort with a big payoff in security and trust.

What's special about AWS Service-Linked IAM Roles

SHAJAM — Wed, 08 Oct 2025 03:47:44 +0000

A service-linked role is a special type of IAM role that's directly associated with an AWS service. These roles are predefined by the service and include all the permissions the service needs to interact with other AWS services on your behalf. It is also a prerequisite for many of the services.
Service-linked roles make it easier to configure a service since you don’t need to manually assign permissions for it to perform actions on your behalf.

These predefined roles help define the permissions that an AWS service will need to function. These roles are only setup when you start using the service. You can view but not edit the permissions of these roles. You can, however, edit the description of the role in some cases.

For example, when you use ECS (container service), a service linked role is configured with ARN like arn:aws:iam::{AWS::AccountID}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS.

Note that, service linked roles and not same as service role. The service role is the IAM role that defines the permissions that the particular instance of the service or task can assume. For example, each Lambda function has its own IAM permissions requirement which is defined in service role. Service-linked role, on the other hand, is defined by AWS and defines what each of the instances of your service can assume by default.

Also, not all services require service linked roles. You can find the list of the services that need it at here.

Other important characteristics of service linked role are that

it is not impacted by service control policies (SCP)
it is not impacted by resource control policies (RCP)

How can you create the service linked role

You can create the service linked role by using AWS console, API or CLI. For example, if you want to create it from the console, you can go to IAM role, create Role, choose service, and it will show you if it needs / supports a service linked role. Refer to the screen shot below.

AWS IAM Users and Roles in the context of Zero Trust Security Framework

SHAJAM — Tue, 07 Oct 2025 20:33:55 +0000

Introduction

IAM Users and Roles in the context of Zero Trust Security Framework. The article discusses the principle - Never trust, always verify, and minimize exposure and how IAM users and roles plays with it.

The IAM users, roles, and policies section is where you establish how identity becomes the new perimeter in AWS. This is the core service that defines permissions for users and services and what they can do. For example, if you want a Lambda function to read and write data from a S3 bucket, you have to configure permissions. You configure it using IAM.

IAM User

IAM User are long-term identities created for people (e.g., developers, administrators) needing direct access to AWS resources. When you create the user, you can give it console login permission so the user can login to your AWS account. To make it secure, you can and should enforce MFA on the IAM user account console login.

You might also use IAM user to obtain the access keys. Access keys are long term tokens that you can use with your application or external applications or users. These tokens are long-term and therefore if leaked can be used for malicious activities.

Best practice using IAM User and Access Keys

As an alternative to access AWS console using IAM user, consider federated access via IAM Identity Centre (AWS SSO) or external IdPs. That way, user login is tied to federated access you don't need to manage the IAM user.
Never share IAM user credentials.
Enforce MFA (Multi-Factor Authentication) for all IAM users.
Rotate access keys regularly or use short-lived credentials (STS).

Zero Trust Tie-In: Long-lived credentials represent implicit trust — contrary to Zero Trust. Migrating to temporary credentials aligns with Zero Trust principles.

IAM Roles

IAM Roles are identities with permissions that can be assumed *temporarily * by trusted entities (users, applications, services, or other AWS accounts). For example, if you have an EC2 instance that needs to read or write data from S3, you can create an IAM role and associate it with the EC2 instance. Then, it can perform S3 operations.

You might wonder how is it different from access keys then. Well, the main difference is that the IAM role is providing temporary tokens issued via AWS STS - Security Token Service that is used for authorisation. With access keys, the keys are long term. Another difference is, with access keys, anyone or service can use the it to gain access to AWS. That is, you can use the access keys from your laptop, from EC2 or Lambda and you can still access the resource. Whereas, with IAM role, you define an IAM trust policy that defines what can assume the role. If the trust policy specifies EC2, then only EC2s can assume the role and using the same role with Lambda will fail.

Use Cases:

Service Roles: EC2 instance needs to read from S3.
Cross-Account Access: Account A allows Account B to assume a role to access its resources.
Federated Access: Users from an external IdP assume roles dynamically when signing in.

Thus, IAM roles provide a Zero-Trust benefit - temporary credentials and dynamic trust relationships reduce attack surface and eliminate static trust assumptions.

Summary

IAM users violate this principle because these rely on long-term credentials (passwords, keys) and does not really have trust relationships.

IAM roles support Zero Trust because

Access is dynamic and time-bound.
Role assumption requires continuous authentication (STS token issuance).
Least privilege can be enforced with fine-grained policies and session conditions.