Forem: Ricardo Cino

AWS Lambda Stubs for unit testing

Ricardo Cino — Mon, 03 Nov 2025 11:37:55 +0000

When working with AWS Lambda functions, unit testing can be a challenge due to the need to mocking all external dependencies, including AWS SDK clients. But also to provide the correct input when calling your Lambda handler functions. To make this easier, I created a small npm package that provides stubs for AWS Lambda handler functions, allowing you to easily provide a default event and customize the input with minimal changes.

Introducing aws-lambda-stubs

The aws-lambda-stubs package provides a set of stubs for AWS Lambda handler functions, making it easier to unit test your Lambda functions by providing default events and allowing you to customize the input as needed making your tests cleaner and more maintainable.

A simple example provided by the README of the package looks like this:

import { SQSEventStub } from "aws-lambda-stubs";
import { describe, expect, it } from "vitest";
import { handler } from "../src/sqs";

describe("sqs handler", () => {
  it("should log the received event", async () => {
    const mockEvent = SQSEventStub([
      {
        body: { message: "Hello, World!" },
      }
    ]);

    // assert on output
    expect(handler(mockEvent)).toEqual({});
  });
});

This clearly demonstrate how easy it is to create a mock SQS event with a custom message body. The output of above example would be a completely valid SQS event that you can pass to your Lambda handler function for testing looking like this:

{
  "Records": [
    {
      "messageId": "1",
      "receiptHandle": "MessageReceiptHandle",
      "body": "{\"key\":\"value\"}",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {},
      "md5OfBody": "ea7e1632014d79bb59dd5e08c6aaea39",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:012345678901:queue-name",
      "awsRegion": "us-east-1"
    }
  ]
}

Why did I build this?

After working on multiple projects, seeing numerous git repositories and seeing everyone making their own stubs for AWS Lambda events, or - non-reusing- stubs making unit tests much larger because of perparing empty test data, I decided to create a reusable package that can be used across multiple projects. This way, I can avoid duplicating code and make it easier for developers to write unit tests for their Lambda functions. Making this open source makes it available for everyone to use and contribute to.

What is supported?

The aws-lambda-stubs package aims to support all payloads given to AWS Lambda functions. Currently, it supports all the EventTypes provided by @types/aws-lambda.

Important while using

What still is very important while making unit tests is that you always test on the input that you expect your Lambda function to receive. The stubs provided by this package are just a starting point, and you should always customize the input to match your specific use case. This way, you can ensure that your unit tests are accurate and reliable.

In the case the defaults change inside the package, your tests might break, so always make sure to check the input you provide to your Lambda handler functions to ensure that they match your expectations.

Next steps

So while this package is already usable, there is still work to be done. My goal is to provide more and more stubs with valuable presets that can be used out of the box. If you want to contribute, please check the repository at https://github.com/cino/aws-lambda-stubs as I would love to make this not just mine, but a community-driven project.

InvalidSignature in Node with AWS SDK

Ricardo Cino — Sun, 28 Sep 2025 18:19:37 +0000

Sometimes you run into a weird issue that is hard to debug. This was one of those times. I was working on a Lambda function that was supposed to retrieve a secret from AWS Secrets Manager. However, I kept getting random InvalidSignatureException errors in production.

What is the issue?

The issue arose in our production environment, where the Lambda function would sometimes throw InvalidSignatureException errors when making calls to AWS services. This was perplexing because around 99% of the requests were successful, and the errors seemed to occur randomly.

Root cause

After extensive debugging and investigation, I discovered that the root cause of the issue was related to how the AWS SDK client was being initialized in our codebase. We had implemented a Repository pattern, and in the constructor of the repository, we were initializing the AWS SDK client to retrieve secrets from AWS Secrets Manager.

However, we were initializing the repository outside of the Lambda handler function, which meant that the same instance of the AWS SDK client was being reused across multiple invocations of the Lambda function.

This is due to the fact that we stored the SDK client in a property of the repository class, which was instantiated only once when the Lambda function was cold-started. As a result, the client would retain the same timestamp for signing requests, leading to signature mismatches and subsequent InvalidSignatureException errors.

Solution

While it's normally a good practice to initiate the repository outside of the handler, in this case, the solution was to move the initialization of the AWS SDK client inside the Lambda handler function. This ensures that a fresh client with a current timestamp is used for each invocation, preventing the InvalidSignatureException errors.

Imagine having a repository like this:

export class MyRepository {
    private readonly client: Promise<PrismaClient>;

    constructor() {
        this.client = getClient();
    }
}

Where getClient retrieves the Client instance to interact with the database, and within this function we would retrieve the secrets from AWS Secrets Manager where an AWS SDK Client is initialized. When doing this all outside of the handler, we would always re-use the same instance of the AWS SDK client, that would retain the same timestamp for signing requests.

So going from this:

import { MyRepository } from "./my-repository";

const repository = new MyRepository();

const handler = async (event) => {
  // implement
};

to this:

import { MyRepository } from "./my-repository";

const handler = async (event) => {
  const repository = new MyRepository();

  // implement
};

The downside

Because now the repository is initialized on every invocation, we lose the benefits of reusing the same instance across invocations, which causes us to retrieve the secrets from AWS Secrets Manager on every invocation. This can lead to increased latency and costs, especially if the Lambda function is invoked frequently.

However, we have implemented caching mechanisms within the repository to mitigate this issue. The repository caches the secrets after the first retrieval, so subsequent invocations can use the cached values instead of making repeated calls to AWS Secrets Manager.

This approach strikes a balance between ensuring valid signatures for AWS SDK requests and optimizing performance by reducing redundant calls to Secrets Manager.

An easy way to implement caching is to use AWS Powertools Parameters utility, which provides a simple way to cache parameters and secrets in memory and automatically refresh them after a specified duration. When not providing any additional configuration, it defaults to caching the secrets for 5 seconds:

import { getSecret } from '@aws-lambda-powertools/parameters/secrets';

export const handler = async (): Promise<void> => {
  // Retrieve a single secret
  const secret = await getSecret('my-secret');
  console.log(secret);
};

(example from AWS Powertools documentation)

Testing in a serverless environment

Ricardo Cino — Fri, 22 Aug 2025 13:38:13 +0000

Working in a pure serverless environment presents a distinct set of challenges when testing your software, particularly when multiple services interact with each other. In this post, I will explore some strategies for testing in a serverless environment.

Types of testing

When it comes to testing, there are different types of tests that you can implement in your software development lifecycle. Here are some of the most common types of tests:

Unit tests: These tests focus on individual components or functions of your codebase, ensuring that each part works as intended in isolation.
Integration tests: These tests evaluate how different components of your application work together, verifying that they interact correctly and produce the expected results.
Infrastructure tests: Tests to confirm that the infrastructure outputs the expected results and behaves as intended.
End-to-end tests: These tests simulate real user scenarios, testing the entire application stack from the frontend to the backend, including all external dependencies.

What we are going to focus on is primarily integration tests, which verify the interactions between different services in a serverless architecture.

Unit testing

Before we continue to integration tests it's important to mention that in my current environment everything is unit tested and we aim towards a high 90% test coverage on our codebase, obviously test coverage is not the end goal and you should properly think about what you are testing.

The reason we focus so much on unit testing is that these are the cheapest and fastest tests to execute. We test every path of our code with unit tests to ensure the results are as we expect them. We do this in the unit test to ensure we can build many tests, and run them locally fast while still building the integration tests to make sure they run in the cloud.

Integration testing

The fun part, how do you properly test your serverless application and why is it so important? The challenge with serverless is that, besides having your own codebase, there are many different parts of infrastructure that you need to consider, such as API gateways, lambdas, databases, queues, you name it. When you have all these different parts working together, there is one thing that unit tests will really not cover: the authorization between these components.

Can your lambda reach the database?
Does it have permissions to access the database?
Do you have the correct IAM Permission to send messages to SQS?

So how do we test this?

To make sure we can safely test our serverless application, we can do a number of things; You can use a tool like LocalStack, which I haven't done before but surely am looking at for a next adventure. Or, you can spin up a full environment for your change request and execute tests against your new environment.

What we like to do in our project is, on each pull request, we deploy a completely new environment prefixed with pr-{pr-number}. This does take a while to complete, based on all the resources that you'll be deploying, but it will be worth your while.

Once this is done, we start tests, which will:

Directly invoke our API's and expect a specific response. This will prove that the lambda behind our api can execute what is necessary and that the API Gateway is configured correctly.
Push messages directly into queues or event buses, and we'll wait until a result has appeared in a database or S3 bucket.

This gives us great confidence that the software that we deploy can be executed with the right permissions and configuration in place.

In comparison to the unit tests, in the integration tests, we do not test all paths, as these have been covered by the unit tests. What we do test is making sure that each external service (database/queue/other service needing specific configuration) is working as expected. In some cases, that is a single test; if there is a non-default path where a message goes to a specific queue or database, there will be a second test, and so on.

Infrastructure tests

A part from testing your code, you also want to make sure that your infrastructure is creating the resources as you expect. As such, you should include tests that validate the infrastructure itself. This can include checking that the correct number of resources have been created, that they are configured correctly, and that they have the right permissions (I know, this sounds double compared to integration tests, but think of it as layers of tests).

When working with AWS CDK you can do this in multiple manners, for one you can use Snapshot tests which capture the current state of your infrastructure and compare it to a state you previously stored inside the repository, this is useful for quick assertions on IF anything has changed and forces you to update the snapshot when you do.

Or you can use my favourite; fine-grained assertions, with this you can specifically check if the output of your CDK App matches your expectations. For example if the Lambda environment has a correct reference to a SSM Parameter, or if the Lambda function has the correct memory configured that is required to execute correctly.

Private testing

If you are a frequent reader of my blog, it must come as no surprise that I am a big advocate for building private architecture when you can. This, however, will give you an additional challenge if you are deploying all your resources in a private network and trying to run an end-to-end test in your CI/CD pipeline.

When working with GitHub Actions, for example, in a default workflow, your tests will be running in a GitHub runner, which would not have any access to your private network. To overcome this limitation, you can use self-hosted runners that are deployed within your private network. In our company, this is provided out-of-the-box by the Cloud Center of Excellence, and there is a great article about it written by a colleague at PostNL and fellow AWS Community Builder Matheus das Mercês.

Conclusion

As you can see, testing your software in a serverless environment can be challenging, but with the right strategies and tools, you can ensure that your application is working as expected. By implementing a combination of unit tests, integration tests, infrastructure tests, and end-to-end tests, you can build a robust testing strategy that will help you catch issues early and ensure that your application is reliable and scalable.

DynamoDB Streams with more than 24 hour retention

Ricardo Cino — Tue, 24 Jun 2025 22:00:00 +0000

That was kind of a misleading title, but I wanted to get your attention. The truth is that DynamoDB Streams have a maximum retention period of 24 hours and there is no way to extend that. When you do need more than 24 hours the default solution is to use Kinesis Data Streams, which can retain data for up to 365 days. While it would be easy to just move to Kinesis, this comes with extra cost which may not be justified for all use cases.

Why not Kinesis?

At the time of writing this I actually am using Kinesis Data Streams for a project, but the only reason we are using it is because we need to retain data for more than 24 hours in case of a failure in the processing lambda. At the time this was built time was of the essence, and we needed a solution that would work out of the box. Kinesis Data Streams is a fully managed service that integrates well with Lambda, and it was the quickest way to get started.

Time has passed and since we now have more time to think about the architecture, we are now considering alternatives to Kinesis Data Streams. The main reason is that Kinesis Data Streams is more expensive than DynamoDB Streams, and we are not using all the features that Kinesis provides. We only need to retain data for a few days, and we can achieve that with DynamoDB Streams by using a custom solution.

This in combination with a workflow where for every pull-request we spin up a new environment this will increase the cost drastically and when you don't necessarily need the performance of Kinesis Data Streams, you can save a lot of money by using DynamoDB Streams instead.

DynamoDb Streams Vs Kinesis Data Streams

DynamoDB Streams and Kinesis Data Streams are both services that allow you to process data in real-time, but they have
different use cases and features. Here are some key differences while using DynamoDB stream:

Retention Period: DynamoDB Streams have a maximum retention period of 24 hours, while Kinesis Data Streams can retain data for up to 365 days.
Cost: DynamoDB Streams are generally cheaper than Kinesis Data Streams, especially for small workloads. Kinesis Data Streams can become expensive as the number of shards increases, while DynamoDB Streams are priced based on the number of read requests.

The custom solution

This might look a bit odd with 2 queues but there is a logical reason for it, the first queue is there because when you put an onFailure destination on a DynamoDB Stream handler it will not place the DynamoDB Record change in the queue, but rather an object with the location in the stream of the record change. This means that if you try to process the message after 24 hours, it will not contain the actual data of the record change, but rather just the data on where to extract it. This message will look like:

{
  "requestContext": {
    "requestId": "12a8a0fa-f8d2-4d42-a6e3-72bae7d1e2f9",
    "functionArn": "arn:aws:lambda:<region>:<account>:function:stream-handler",
    "condition": "RetryAttemptsExhausted",
    "approximateInvokeCount": 4
  },
  "responseContext": {
    "statusCode": 200,
    "executedVersion": "$LATEST",
    "functionError": "Unhandled"
  },
  "version": "1.0",
  "timestamp": "2025-06-05T18:22:26.479Z",
  "DDBStreamBatchInfo": {
    "shardId": "shardId-10101010101010101010-10101010",
    "startSequenceNumber": "202328600001712385967261751",
    "endSequenceNumber": "202328600001712385967261751",
    "approximateArrivalOfFirstRecord": "2025-06-05T18:21:49Z",
    "approximateArrivalOfLastRecord": "2025-06-05T18:21:49Z",
    "batchSize": 1,
    "streamArn": "arn:aws:dynamodb:<region>:<account>:table/table/stream/2025-04-23T13:37:16.991"
  }
}

As you can imagine, if you would process this message after 24 hours it would not be very useful as the data at this location in the DynamoDB Stream would have been deleted. This is why we are using a second queue, the first queue is used to retrieve the actual data from the DynamoDB Stream, and the second queue is used to store the data for a longer period of time. In my case, I am using SQS as the second queue, which is not enabled by default, but you can enable it in the Lambda function configuration. This way, you can process the data from the first queue and store it in the second queue for later processing.

Whenever there is a real failure of our DynamoDB Stream handler it will give us some time to investigate the issue and resolve it before we enable the re-drive lambda and process the messages in the second queue. This way, we can ensure that we are not losing any data and that we can process it later when we have time to investigate the issue.

Idempotency

When you are using a solution like this you do want to make sure that your processing is idempotent. This means that if you process the same message multiple times, it will not have any side effects. In our case, we have used AWS Lambda Powertools for TypeScript (also available, with more features for other languages; Python, .NET and Java) to implement idempotency in our DynamoDB Stream handler in the critical points where we do not want to reprocess the same step unless it failed there. I can highly recommend using this library as it provides a lot of useful utilities for working with AWS Lambda, including idempotency, logging, and tracing.

Conclusion

Obviously this is not a standard solution, but it's a cost-effective way to retain data for more than 24 hours without using Kinesis Data Streams. It does require some extra work to set up, but it can save you money in the long run if you don't need the full capabilities of Kinesis. If you do need more than 24 hours of retention, Kinesis Data Streams is still a great option, but if you can work with the limitations of DynamoDB Streams, this custom solution can be a good alternative.

Private AppSync with custom dns

Ricardo Cino — Sat, 07 Jun 2025 13:20:08 +0000

In the last year I've been working a lot with AppSync and I have to say it didn't come without challenges. One of the biggest challenges was to create a private AppSync API with a custom domain. This is something that is not natively supported by AWS, but it is possible to achieve it using a combination of services.

While this is now possible for API Gateway, AppSync is a different beast. There is no native way to create a private AppSync API with a private custom domain. This is something that I had to figure out the hard way, and I want to share my findings with you.

This might become a bit of a long read, but if you are looking for a way to create a private AppSync API with a custom domain, you are in the right place. I will try to explain the steps I took to achieve this and the challenges I faced along the way.

What is AppSync?

AppSync is a fully managed service that makes it easy for developers to build scalable Graphql APIs on AWS. It allows you to create a flexible API for applications that require real-time data, such as mobile or web apps. With GraphQL support, AppSync helps you to define the structure of your data and how it can be queried.

With AppSync, you can easily connect to various data sources, including DynamoDB, Lambda, Elasticsearch, and more. It also provides built-in support for real-time subscriptions, offline data synchronization, and security features like authentication and authorization.

While there are other challenges with Authorization, which I will not cover in this article (but surely in a follow-up), I will focus on how to create a private AppSync API with a custom domain and all the challenges that come with it.

Target Architecture

(spoiler: We will not reach this architecture)

This would be the ideal situation i'd like to see, however this is not possible with the current state of AppSync, with the main reason being that it's currently not possible to add a custom domain to a private AppSync API. Which results in that when you do try to point

Private AppSync API

Making it private

Making AWS AppSync private itself is not a big deal, you just need to toggle the Use private API features option in the AppSync console. This will allow you to create a private AppSync API that is only accessible from within your VPC. This is done by creating a VPC endpoint for AppSync, which allows you to access the service without going through the public internet.

Disclaimer 1: You can not make a public AppSync API private, this needs to be configured at the start. This is a limitation of AppSync itself and not something that can be changed later on.

Disclaimer 2: When you make an AppSync API private, you will not be able to access it from the aws console and losing the ability to use the console to test your API. This can be overcome by using the CloudShell from your vpc or by deploying an EC2 instance inside your vpc.

Private AppSync API with custom domain

Now this is the part where it becomes challenging, because AppSync does not support private APIs with custom domains. This means that you will not be able to use the default AppSync endpoint, which is something that you will have to work around by using additional services.

Why a custom domain again?

While you could use the default AppSync endpoint, this is not recommended for production use. In case of a service replacement, you will have to update all your clients with the new endpoint. This is not a big deal if you are using a single client, but if you are using multiple clients (e.g. mobile and web), this can become a nightmare.

Even if you'd decide to switch to hosting your own GraphQL server you might be able to switch the back-end and not inform the client, if you keep supporting the exact same features as you are using. If you are using Subscriptions from AppSync it might become more challenging due to the implementation of AWS AppSync's authorization mechanism.

Creating custom DNS for your AppSync API

To make your AppSync API accessible from a custom domain, you will have to use the same approach as you would for API Gateway before they introduced native support . This means that you will have to create a Route 53 hosted zone for your custom domain and create a CNAME record that points to an Application Load Balancer that routes traffic to the AppSync VPC Endpoint.

However this is not enough, because where this will allow you to access the AppSync API from your custom domain, it will not allow you to only use the custom domain. Additionally you will have to send an HTTP header (X-AppSync-Domain) to the AppSync API that contains the custom domain name so the VPC Endpoint can route the traffic to the correct AppSync API. In that case you still need to share the AppSync API endpoint with the end user.

While it is technically possible to add the header to all your requests, this will only work for HTTP requests. If you are using WebSockets for subscriptions, you will not be able to add the header to the request, because the WebSocket protocol does not support custom headers in the web browser. When using a different client (e.g. mobile or server-side) you might be able to add the header, but this is not a solution that works for all clients.

The only way to avoid this limitation is to use a proxy that will route the traffic to the AppSync API. This can be done by using an Application Load Balancer with a custom domain and a target group that points a proxy (e.g. NGINX) that will route the traffic to the AppSync API. This way you can use the custom domain to access the AppSync API without exposing the AppSync API endpoint to the end user.

Disclaimer 3: Whenever you use a custom private domain the default libraries provided (AppSync SDK & Amplify Framework's AppSync, see this issue (especially the comment of sleepwithcoffee too.)) by AWS will not work. This is because the libraries are using the default AppSync endpoint and not the custom domain. This means that you will have to use a custom implementation of the AppSync client that supports custom domains.

To implement such proxy you can use NGINX, which is a popular web server that can be used as a reverse proxy. NGINX can be configured to route traffic to the AppSync API and add the required HTTP header (X-AppSync-Domain) to the request.

Example

  location /graphql {
    proxy_set_header X-AppSync-Domain ${identifier}.appsync-api.eu-west-1.amazonaws.com;
    proxy_pass https://${url};
    proxy_pass_request_headers on;
    proxy_ssl_session_reuse off;
    proxy_cache_bypass $http_upgrade;
    proxy_redirect off;
    ...
  }

  location /graphql/ws {
    proxy_set_header X-AppSync-Domain ${identifier}.appsync-realtime-api.eu-west-1.amazonaws.com;
    proxy_pass https://${url};
    proxy_pass_request_headers on;
    proxy_ssl_session_reuse off;
    proxy_cache_bypass $http_upgrade;
    proxy_redirect off;
    ...
  }

Final architecture

If we want to invoke our AppSync API from our custom DNS only the architecture will look like this:

Why is that? Well, the only way to access an AppSync API privately from your VPC is to use a VPC endpoint. This means that you will have to create a VPC endpoint for AppSync and route traffic from your custom domain to the VPC endpoint. This is done by creating an Application Load Balancer with the desired Custom DNS. Behind the ALB you will have 2 target groups, one for the AppSync VPC endpoint Private IP address to directly access the AppSync API and another one for the proxy that will be used to access the AppSync API from the browser.

With this setup, you can access the AppSync API from your custom domain without exposing the AppSync API endpoint to the end user. The proxy will handle the routing of the traffic to the AppSync API and will add the required HTTP header (X-AppSync-Domain) to the request.

Wishes

Like API Gateway, AppSync is a service that is constantly evolving. I hope that in the future we will have a native way to create private AppSync APIs with custom domains. This would make our lives a lot easier and would eliminate the need for workarounds like the one I described in this article.

Sources

I hope this post was helpful to you and if you have any questions or remarks feel free to reach out to me on Bluesky or LinkedIn.

Modifying DynamoDB TTL with CDK

Ricardo Cino — Sun, 23 Mar 2025 09:00:00 +0000

Ever tried to update the TTL attribute of a DynamoDB table using the AWS CDK and got a InvalidRequest in CDK or a ValidationException via the CLI? I did, and it took me a while to figure out why. In this post, I'll explain what happened and how to avoid the same issue in the future. This is a short post, but I think it's worth sharing because it can save you some time and frustration.

Introduction

Very recently I spent a few hours trying to figure out why I was getting a ValidationException when trying to update the TTL attribute of a DynamoDB table using the AWS CDK. The error message was not very helpful, and I couldn't find any documentation that explained the issue. After some digging (and asking around on Slack), I discovered that the problem was related to the fact that the TTL attribute was already set to a different value, and I was trying to change it to a new one which is not allowed in a single deployment.

Modifying DynamoDB TTL with CDK

The Error(s)

When you try to update the TTL attribute of a DynamoDB table using the AWS CDK, you might encounter the following error:

Invalid request provided: Cannot change time-to-live attribute name. To update this property, you must first disable TTL then enable TTL with the new attribute name.

For me this was a surprise and wasn't aware this limitation existed. Followed by this error I thought, let me try this by the CLI and see if I can update it manually before deploying the CDK Stack. So to clear up; I was trying to disable the TTL by CLI and than re-run the stack with the new TTL attribute.

Of course that didn't work either, because the state of the CloudFormation stack was still in the previous state. So when validating the stack it would still throw the same exact error.

Next up I thought, let's try to disable the TTL via the CLI and then re-enable it with the new attribute name. This is where I got the following error:

An error occurred (ValidationException) when calling the UpdateTimeToLive operation: Time to live has been modified multiple times within a fixed interval

Why was I confused

To me this was a experience of surprises. I looked at the documentation and it didn't mention anything about this limitation. I thought I could just change the TTL attribute name and be done with it. This was mainly due to the DynamoDB CloudFormation documentation about the TimeToLiveSpecification property, which doesn't mention anything about this limitation. It actually mentions you can modify this property without any interruption on this page.

And yes, there is a note below that saying there are limitations on DynamoDb and links to a new page, which does not mention the TimeToLiveSpecification property.

Now the second error I got was even more confusing. I was trying to disable the TTL and then re-enable it with a new attribute name, but I got an error saying that the TTL had been modified multiple times within a fixed interval. Again, this was something I hadn't yet spotted in any of the documentation which was a error on my side. I opened the documentation for UpdateTimeToLive and immediately scrolled down to the Errors section and this did not include the ValidationException.

However, at the top of the page that discusses the UpdateTimeToLive operation, it clearly states that:

TimeToLiveSpecification. It can take up to one hour for the change to fully process. Any additional UpdateTimeToLive calls for the same table during this one hour duration result in a ValidationException.

So, if you try to disable the TTL and then re-enable it with a new attribute name within that one hour duration, you'll get the ValidationException error.

Conclusion

In conclusion if you use any kind of Infrastructure as code tool, like the AWS CDK, you need to be aware of this limitation when modifying the TTL attribute of a DynamoDB table. You can't just change the attribute name and expect it to work. You'll need to:

Disable the TTL
Deploy the change
Wait for the change to be applied (up to one hour)
Enable the TTL with the new attribute name
Deploy the change
Wait for the change to be applied (up to one hour)

In hindsight, yes I could've spotted this in the documentation but I clearly didn't and as a result I lost multiple hours on researching but also, deploying.. and waiting for the changes to be applied. So I hope this post helps you avoid the same issue in the future.

Sources

Always set AWS CDK Defaults

Ricardo Cino — Mon, 25 Nov 2024 09:33:41 +0000

We are nearing the end of the year, the time to reflect on the past year and definitely share the things that went "wrong" or in this case the things that could have been done better. This is one of those things that I wish I knew earlier, and I hope it helps you too.

AWS CDK Defaults.. they change

As you're reading this you definitely know what AWS CDK is and you are most likely also using it. If you are not, I'd suggest you take a look at it, it's a great way to manage your AWS infrastructure in a programmatic way (nothing against the likes of Terraform and/or Pulumi, cdk is just my chosen evil (and mandated by the company)).

CDK is designed to be as user-friendly as possible, allowing you to create Amazon resources easily and quickly. To do so, it comes with a lot of defaults, which is great, but it can also be a bit risky.

Reason for me saying it are the following:

I was not aware of the defaults, and I was not aware that the defaults of CDK are NOT the same as AWS CloudFormation
Defaults change, and they can change without you knowing it

What happened?

In the projects I work with we always have a full CI/CD setup with a lot of tests. For each pull-request we open we deploy a full environment with AWS CDK and run a lot of tests against it. This is great, and really helps us to catch a lot of issues before they hit production.

While deploying all the resource is great, we also need to destroy the resources that we created and this is where the issue started. Before a certain change in CDK the default RemovalPolicy for Kinesis Stream was set to DESTROY, which is great for testing but not so great for production. After the change the default was set to RETAIN, which I support as for production you would want this.

However, because we were not setting the RemovalPolicy ourselves, we were relying on the default. This meant that after the change we were not able to delete the Kinesis Stream anymore, because the default was changed.

What surprised me about the change is that it was in a very large pull-request where a lot of changes were made, and it was not mentioned in the release notes. This is not a critique, but it's just something to be aware of.

How did I find out?

You don't want to hear this, but yes, through the Cost Explorer. While randomly checking our Dev environment where the pull-request environments are deployed to there was a surprise peak in Kinesis cost, not being sure why I started digging into the resources and found out that many Kinesis streams were still there after Pull-Requests were closed.

Learnings

For me this taught me to always override the defaults of CDK, even if they are the same as the AWS defaults. This way you are sure that you are in control of the resources you are creating and you are not relying on the defaults of CDK. Giving you no surprises when a sudden change happens.

Private API Gateway with DNS

Ricardo Cino — Thu, 21 Nov 2024 10:03:46 +0000

At PostNL we are building most of our applications with Serverless in mind, let me rephrase that, we build all our applications within our own landing zone with Serverless only. There is no option to deploy any kind of EC2 and if you need containers you'd be running them on Fargate only.

Given that, we are using quite a bunch of API Gateways in the projects I'm working on. While PostNL is also a big corporate company we have a strong focus on security and compliance, and that's why we are building our applications Private first. When there is no need to be public, it shouldn't be.

This is however, easier said then done when trying to do it with API Gateway and I'll show you why.

API Gateway

To start, let's have a look at Amazon API Gateway. When you create a gateway you'll be provided with an endpoint that is public by default and hosted under the Amazon domain (*.execute-api.{region}.amazonaws.com). This is not what we want, we want to have our API Gateway private and only accessible from within our VPC.

For Public API Gateways you can use a custom domain name and use Route 53 to point to the API Gateway endpoint. This is not possible for Private API Gateways, you can't use a custom domain name and you can't use Route 53 to point to the API Gateway endpoint.

When you deploy a simple API Gateway without any custom dns your architecture would look as simple as:

Now when you'd like to add your own custom DNS on a public API Gateway you can do so by creating a custom domain name and using Route 53 to point to the API Gateway endpoint (you can leverage the custom domain feature on api). This would look like:

At this point it still looks simple, but when you'd like to have a private API Gateway you can't use a custom domain name and you can't use Route 53 to point to the API Gateway endpoint. This is where it gets tricky.

Why a custom domain?

Even though it's not supported for Private API's you might wonder why would you need a custom domain on your API Gateway. To me there is a simple reason why I want this. Within a larger organization on AWS you often have other teams / aws accounts invoke your api's.

When you are in a situation like this often you have 2 options:

There is a central team that manages all the api's and they provide you with the endpoint to invoke the api. (which still means that you need to provide a private network connection to the central API team).
You allow another team to invoke your api's by providing them with the endpoint of the api gateway.

When you don't have custom dns on your API Gateway the only option to provide the API to a different account is to provide them with the API Gateway ID, which they'll use together with a VPC Endpoint in their account to reach your API Gateway. This is not a very user-friendly way to provide an API to another team.

A downside of this is that you lose the flexibility to replace or update the API Gateway without having to update all the consumers that are using the API Gateway. When you have a custom domain name you can point the domain to a new API Gateway and the consumers don't have to change anything.

When you have a custom domain name you can provide the other team with a domain name that they can use to invoke the API Gateway. This is a much more user-friendly way to provide an API to another team.

The Problem

As noted before it is not possible to use a custom domain name on a private API Gateway. This means that you can't use Route 53 to point to the API Gateway endpoint. The actual technical reason for me is still unknown but I do know that the API Gateway is not deployed inside your own VPC.

The only way to reach the API Gateway is by using a VPC Endpoint. This is a private connection between your VPC and the API Gateway. This is a great way to provide a private connection to the API Gateway but it doesn't provide you with a way to reach the API Gateway via a custom domain name.

The Solution

The solution to this problem is to use an Application Load Balancer (ALB) in front of the VPC Endpoint. You will need to retrieve the private ip addresses of the VPC Endpoint and point a target group towards these. The ALB is deployed in your VPC and is reachable via a custom domain name. This way you can reach the API Gateway via the custom domain name and fully private net working.

The beauty of this solution is that you can provide the custom domain name to other teams and they can invoke the API Gateway via the custom domain name without having to know the specifics of the API Gateway. Because the consumer is now not aware of what is behind the DNS you can replace it with whatever you want without having to update the consumers.

This is how the architecture would look like:

CDK Example

It's not a post from me without a CDK example, so here it is. In this example we are creating an API Gateway with a simple Mock Integration Response that is reachable via private DNS. The API Gateway is deployed in a VPC and is only reachable when invoked via the VPC Endpoint, to be able to use private DNS we are deploying an Application Load Balancer which is pointed towards the VPC Endpoint. Based upon the domain name that is pointed towards the Application Load Balancer and that is configured on the API Gateway we can reach the API Gateway via the private DNS.

const apiFqn = "api.cino.dev";

// Step 1: Retrieve vpc / hosted zones
const [vpc, publicHostedZone, privateHostedZone] = this.getNetwork();

// Step 2: Create API Gateway VPC Endpoint
const apiGatewayVpcEndpoint = this.createApiGatewayVpcEndpoint(vpc);

// Step 3: Create certificate for the API Gateway / ALB
const acmCertificate = this.createCertificate(apiFqn, publicHostedZone);

// Step 3: Create ALB
const alb = this.createApplicationLoadBalancer(vpc, apiGatewayVpcEndpoint);

// Step 4: Create API Gateway
this.createApiGateway(apiFqn, acmCertificate, apiGatewayVpcEndpoint);

// Step 5: Create ALB Listener for API Gateway VPC Endpoint
this.createApiGatewayVpcEndpointListener(
  acmCertificate,
  vpc,
  alb,
  apiGatewayVpcEndpoint
);

// Step 6 Create Route53 records pointing towards
// the Application Load Balancer
this.createDnsRecords(privateHostedZone, alb, apiFqn);

As you see we need the following resources:

VPC
Hosted Zones (Public & Private)
- Public Hosted Zone is used for the certificate
- Private Hosted Zone is used for the DNS records
API Gateway VPC Endpoint
Certificate for the API Gateway / ALB
Application Load Balancer
- Listener on port 443 for your domain pointing towards the VPC Endpoint Private IP Adresses
API Gateway
Custom Domain Name pointing towards the Application Load Balancer

For the full example you can visit the Github repository as I'll only show you the most important parts of the code, that said we immediately go to step 5 where we create the API Gateway VPC Endpoint Listener.

private createApiGatewayVpcEndpointListener(
  acmCertificate: Certificate,
  vpc: IVpc,
  alb: ApplicationLoadBalancer,
  apiGatewayVpcEndpoint: InterfaceVpcEndpoint
): void {
  const ipAddresses = this.getVpcEndpointIpAddresses(
    vpc,
    apiGatewayVpcEndpoint
  );

  const targets = ipAddresses.map((ip) => new IpTarget(ip));
  const apiGatewayTargetGroup = new ApplicationTargetGroup(
    this,
    `PrivateApiGatewayTargetGroup`,
    {
      vpc: vpc,
      targetType: TargetType.IP,
      targetGroupName: `priv-apigw-tg`,
      targets,
      protocol: ApplicationProtocol.HTTPS,
    }
  );

  apiGatewayTargetGroup.configureHealthCheck({
    path: `/test/hello`,
    healthyHttpCodes: "200,403",
    port: "443",
  });

  alb.addListener("PrivateApiGatewayListener", {
    port: 443,
    protocol: ApplicationProtocol.HTTPS,
    certificates: [acmCertificate],
    defaultTargetGroups: [apiGatewayTargetGroup],
  });
}

private getVpcEndpointIpAddresses(
  vpc: IVpc,
  vpcEndpoint: InterfaceVpcEndpoint
): string[] {
  const vpcEndpointProps = new AwsCustomResource(this, "VpcEndpointEnis", {
    onUpdate: {
      service: "EC2",
      action: "describeVpcEndpoints",
      parameters: {
        Filters: [
          {
            Name: "vpc-endpoint-id",
            Values: [vpcEndpoint.vpcEndpointId],
          },
        ],
      },
      physicalResourceId: PhysicalResourceId.of(Date.now().toString()),
    },
    policy: AwsCustomResourcePolicy.fromSdkCalls({
      resources: AwsCustomResourcePolicy.ANY_RESOURCE,
    }),
  });

  const vpcEndpointIps = new AwsCustomResource(
    this,
    "vpc-endpoint-ip-lookup",
    {
      onUpdate: {
        service: "EC2",
        action: "describeNetworkInterfaces",
        outputPaths: vpc.availabilityZones.map((_, index) => {
          return `NetworkInterfaces.${index}.PrivateIpAddress`;
        }),
        parameters: {
          NetworkInterfaceIds: vpc.availabilityZones.map((_, index) => {
            return vpcEndpointProps.getResponseField(
              `VpcEndpoints.0.NetworkInterfaceIds.${index}`
            );
          }),
        },
        physicalResourceId: PhysicalResourceId.of(Date.now().toString()),
      },
      policy: AwsCustomResourcePolicy.fromSdkCalls({
        resources: AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
    }
  );

  return vpc.availabilityZones.map((_, index) => {
    return vpcEndpointIps.getResponseField(
      `NetworkInterfaces.${index}.PrivateIpAddress`
    );
  });
}

The important part about this piece of code is we are creating a listener towards the private ip addresses of the created VPC Endpoint. This is done by creating a target group with the private ip addresses and creating a listener on the Application Load Balancer that points towards the target group.

However, the private ip addresses of the VPC Endpoint are not directly available in the CDK, so we need to use a Custom Resource to retrieve the private ip addresses of the VPC Endpoint. This is done by creating a Custom Resource that retrieves the VPC Endpoint ID and then creates another Custom Resource that retrieves the private ip addresses of the VPC Endpoint.

Full example can be found on GitHub.

Conclusion

In this post I've shown you how you can create a private API Gateway with DNS. This is not a straightforward process and requires some additional resources to make it work. However, when you have a need for a private API Gateway with DNS this is the way to go.

Please have a good look at the Github link provided as there are a lot more details to be found and it is a fully working example with a simple cdk deploy command. (when you provide your own route53 / hosted zone)

I hope this post was helpful to you and if you have any questions or remarks feel free to reach out to me on Bluesky or LinkedIn.

AWS CloudShell in your own vpc

Ricardo Cino — Thu, 21 Nov 2024 07:30:00 +0000

Until recently, I was completely unaware of AWS CloudShell, and I’m glad I finally decided to give it a try. CloudShell provides a shell environment right in your browser, and to my surprise, you can start an instance within your own VPC!

CloudShell in your own VPC

Why is this so cool? Well, I was working on a new example to show on my website I needed to access my private resources. While I have a full setup on my day-time job where I have a VPN connection to my VPC, I don't have this setup at my sandbox account. I was thinking about how I could access my private resources and then I remembered CloudShell.

When you start CloudShell you can select the VPC and subnet you want to use. This means that you can access your private resources without having to setup a VPN connection. This is great for me as I can now access my private resources without having to setup a VPN connection.

Cost

I was a bit worried about the cost of running an CloudShell instance in my VPC, but it turns out that it's free! You can run an instance in your VPC for free and will be terminated after 30 minutes of inactivity.

My use case

The real thing I was trying to do is resolve my Private DNS entries, by creating a Cloud Shell in my own VPC which was associated with the Private Hosted Zone I was able to resolve the Private DNS entries. This is great as I can now test my DNS entries without having to setup a VPN connection.

traceroute was not actually installed, you need to do that yourself ;)

Avoiding CloudFormation Stack Outputs

Ricardo Cino — Sun, 17 Nov 2024 17:47:15 +0000

Recently I’ve been working on a new project where we created many resources in a lot of different stacks. A feature of CloudFormation is that you can output values from your stack, which is great for referencing resources in other stacks. However, while there is a use-case for this, I’ve found that it’s better to avoid using these outputs and instead use SSM parameters.

Why avoid outputs?

There is a simple reason for me to avoid using outputs and that is dependencies. When you output a value and import it an a different stack you create a hard dependency between the two stacks. This means that you can't delete the stack that is being referenced without first deleting the stack that is referencing it.

Even better, you can't remove the stack output without first deleting the stack that is referencing it. This is a problem when you want to remove a resource from a stack, even when you are updating multiple stacks at the same time it will fail because of the dependency, in many cases it will try to update the stack which has the resource which is used for output first before the second stack where it is referenced. Even when the second stack wouldn't need the output anymore in that same deploy it will still cause a reference issue due to the order of the updates.

This is good right? Well, yes. There is an absolute case to be made where this gives you safety and stops you from deleting resources that are still in use. However, in my case:

We are deploying stacks often while making test environments and we want to be able to remove resources without having to delete all the stacks that are referencing it all-the-time.
When you actually have an issue in production, dealing with stack dependencies is not something you want to be doing at that time.
Destroying all your stacks can take a long time as they need to be deleted in the correct order.

Surprise outputs by AWS CDK

If you are using AWS CDK for your infrastructure as code you might be surprised by the many outputs that you are using! This is because when you reference a Construct in another stack it will automatically output the identifier of the resource. This is a nice feature but it can also be a surprise when you are not expecting it. Even though you might be thinking that you are not using outputs, you are.

For example when you deploy the following stacks, in this example we are deploying 2 resources in different stacks, there is a DynamoDB table in the DataStack and a single Lambda function in the AppStack. The Lambda function has been given the permissions to read the DynamoDB table by passing it into the next stack and calling table.grantReadData(lambda).

const { table } = new DataStack(app, "DataStack");
new AppStack(app, "AppStack", { table });

Full example: example-with-stack-outputs.ts

When building your infrastructure in this way you will see that the CDK will automatically output the identifier of the DynamoDB table in CloudFormation.

This value is imported by the AppStack during the synthesize phase and used to retrieve the DynamoDB Table. Now you have a hard dependency between the two stacks. When we now try to delete the AppStack we will get the following error:

const { table } = new DataStack(app, "DataStack");
// new AppStack(app, "AppStack", { table });

Delete canceled. Cannot delete export DataStack:ExportsOutputFnGetAttDataStackTableB5A722F5ArnBE341008 as it is in use by AppStack.

You'll see that it now automatically tries to delete the created Output as it is not longer in use by the AppStack, however because CDK now doesn't have a notion of the AppStack it will only try to perform an update on the DataStack which will fail because the AppStack is still referencing it.

This is just a small example of how you can get into trouble with outputs without even knowing it.

How to avoid this?

A pattern to avoid this is to store identifiers in SSM parameters. This way you can reference the SSM parameter in your other stacks and you can delete the resource without having to delete the stack that is referencing it. This does mean that when you delete the resource you will have to make sure that your other stacks don't depend on it anymore, giving you more risk of deleting resources that are still in use.

Personally for me this risk is worth it as you can still create dependencies between stacks during deployment like the example below. This way you are ensured that the resources are created in the correct order without having CloudFormation outputs and hard dependencies.

const dataStack = new DataStackWithSsmParameters(app, "DataStackWithSsmParameters");
const appStack = new AppStackWithSsmParameters(app, "AppStackWithSsmParameters");
appStack.addDependency(dataStack);

Full example: example-with-ssm-parameters.ts

Conclusion

If you are using CloudFormation (or CDK) you can avoid finding yourself in a situation where you can't update a stack because of a reference to another stack by using SSM parameters instead of outputs. This way you can delete resources without having to delete the stacks that are referencing them.

In all the projects I'm working on myself I will not be using any outputs anymore and will be using SSM parameters instead. I've been hurt too many times with the hard dependencies between stacks and the time it takes to delete all the stacks in the correct order.

At 1 of the projects I had to delete 10+ stacks which would take more than 1 hour in a pipeline due to the dependencies between the stacks. Removing these outputs and importing the values from SSM Parameters decreased the time to delete all the stacks to ~20 minutes. I admit this is still a long time but this is because there was a CloudFront and Fargate container in the codebase. (Cloudfront being the longest to delete)

✅ Pros	❌ Cons
Less conflicts when updating your stacks	Less "safety" out of the box from CloudFormation
Faster deletion of all your stacks in test environments	Needing to place dependencies between stacks yourself