Forem: Benoît Bouré

Private API Gateway as EventBridge API Destination

Benoît Bouré — Tue, 21 Jan 2025 08:00:00 +0000

In a previous post, I explained how to connect AWS Step Functions to a private API Gateway endpoint thanks to the new integration with AWS PrivateLink and Amazon VPC Lattice. In this issue, I’ll show you how to use the same integration to use a private API Gateway API as an EventBridge target using the CDK, removing the need for an intermediary Lambda function.

Overview

Source

The setup is similar to the one for Step Functions. A Resource Gateway is used as the entry point into the VPC. It is associated with a Resource Configuration, which defines the Private API Gateway resource, and the EventBridge Connection is configured to use the Resource Config as the final destination.

For more details about this setup, see my previous post about the Step Functions Integration.

CDK Stack Definition

We need to define the Resource Gateway and the Resource Definition.

    // Security Group for the Resource Gateway
    const rgSecurityGroup = new SecurityGroup(this, 'ResourceGatewaySG', {
      vpc: vpc,
      allowAllOutbound: false,
    });

    rgSecurityGroup.addEgressRule(
      Peer.ipv4(vpc.vpcCidrBlock),
      Port.tcp(443),
      'Allow HTTPS traffic from Resource Gateway',
    );

    // Resource Gateway
    const resourceGateway = new CfnResourceGateway(this, 'ResourceGateway', {
      name: 'private-api-access',
      ipAddressType: 'IPV4',
      vpcIdentifier: vpc.vpcId,
      subnetIds: vpc.isolatedSubnets.map((subnet) => subnet.subnetId),
      securityGroupIds: [rgSecurityGroup.securityGroupId],
    });

    // Resource Configuration
    const resourceConfig = new CfnResourceConfiguration(
      this,
      'ResourceConfig',
      {
        name: 'sf-private-api',
        portRanges: ['443'],
        resourceGatewayId: resourceGateway.ref,
        resourceConfigurationType: 'SINGLE',
      },
    );

    // Use the global DNS name of the API gateway's VPC endpoint
    // in the Resource Configuration
    resourceConfig.addPropertyOverride(
      'ResourceConfigurationDefinition.DnsResource',
      {
        DomainName: Fn.select(
          1,
          Fn.split(':', Fn.select(0, api.vpcEndpoint.vpcEndpointDnsEntries)),
        ),
        IpAddressType: 'IPV4',
      },
    );

    // Event Bus
    const eventBus = new EventBus(this, 'EventBus', {});

    // Connection to the API
    const connection = new Connection(this, 'ApiConnection', {
      authorization: Authorization.apiKey(
        'x-api-key',
        SecretValue.unsafePlainText('demo'),
      ),
    });

    // Setup the Connection with the Resouce Config
    (connection.node.children[0] as CfnConnection).addPropertyOverride(
      'InvocationConnectivityParameters',
      {
        ResourceParameters: {
          ResourceConfigurationArn: resourceConfig.attrArn,
        },
      },
    );

EventBridge is now able to connect to the private API Gateway. We can now create a rule and set the API as the target.

    const rule = new Rule(this, 'RequestAccountRule', {
      eventBus,
      eventPattern: {
        source: ['my-source'],
      },
    });

    const apiDestination = new ApiDestination(this, 'ApiDestination', {
      endpoint: `${api.api.url}/hello`,
      httpMethod: HttpMethod.POST,
      connection: connection,
    });

    rule.addTarget(
      new targets.ApiDestination(apiDestination, {
        event: RuleTargetInput.fromEventPath('$.detail'),
      }),
    );

Find the full code on GitHub.

Testing the Integration

Putting the following event on the bus.

{
  "DetailType": "somethingHappened",
  "Source": "my-source",
  "EventBusName":"EventBusVendingMachine308DEFEB",
  "Detail": {
    "foo": "bar"
  }
}

I can see that the Lambda function used as the handler of the endpoint is invoked with the following event.

{
    "resource": "/hello",
    "path": "/hello",
    "httpMethod": "POST",
    "headers": {
        "Accept-Encoding": "gzip, x-gzip, deflate, br",
        "Content-Type": "application/json; charset=utf-8",
        "Host": "899aggxh3a.execute-api.us-east-1.amazonaws.com",
        "Range": "bytes=0-1048575",
        "User-Agent": "Amazon/EventBridge/ApiDestinations",
        "x-amzn-cipher-suite": "ECDHE-RSA-AES128-GCM-SHA256",
        "x-amzn-tls-version": "TLSv1.2",
        "x-amzn-vpc-id": "vpc-0a1db1c1701e137ca",
        "x-amzn-vpce-config": "1",
        "x-amzn-vpce-id": "vpce-09fc3c0c5173d919b",
        "x-api-key": "demo",
        "X-Forwarded-For": "10.0.195.243"
    },
    "multiValueHeaders": {
        "Accept-Encoding": [
            "gzip, x-gzip, deflate, br"
        ],
        "Content-Type": [
            "application/json; charset=utf-8"
        ],
        "Host": [
            "899aggxh3a.execute-api.us-east-1.amazonaws.com"
        ],
        "Range": [
            "bytes=0-1048575"
        ],
        "User-Agent": [
            "Amazon/EventBridge/ApiDestinations"
        ],
        "x-amzn-cipher-suite": [
            "ECDHE-RSA-AES128-GCM-SHA256"
        ],
        "x-amzn-tls-version": [
            "TLSv1.2"
        ],
        "x-amzn-vpc-id": [
            "vpc-0a1db1c1701e137ca"
        ],
        "x-amzn-vpce-config": [
            "1"
        ],
        "x-amzn-vpce-id": [
            "vpce-09fc3c0c5173d919b"
        ],
        "x-api-key": [
            "demo"
        ],
        "X-Forwarded-For": [
            "10.0.195.243"
        ]
    },
    "queryStringParameters": null,
    "multiValueQueryStringParameters": null,
    "pathParameters": null,
    "stageVariables": null,
    "requestContext": {
        "resourceId": "yjzggg",
        "resourcePath": "/hello",
        "httpMethod": "POST",
        "extendedRequestId": "Efl1aFY5oAMFoYA=",
        "requestTime": "16/Jan/2025:18:29:54 +0000",
        "path": "/prod/hello",
        "accountId": "438465158289",
        "protocol": "HTTP/1.1",
        "stage": "prod",
        "domainPrefix": "899aggxh3a",
        "requestTimeEpoch": 1737052194204,
        "requestId": "3a6754ae-5488-429c-9ec6-1837a4c21727",
        "identity": {
            "cognitoIdentityPoolId": null,
            "cognitoIdentityId": null,
            "vpceId": "vpce-09fc3c0c5173d919b",
            "apiKey": "demo",
            "principalOrgId": null,
            "cognitoAuthenticationType": null,
            "userArn": null,
            "userAgent": "Amazon/EventBridge/ApiDestinations",
            "accountId": null,
            "caller": null,
            "sourceIp": "10.0.195.243",
            "accessKey": null,
            "vpcId": "vpc-0a1db1c1701e137ca",
            "cognitoAuthenticationProvider": null,
            "user": null
        },
        "domainName": "899aggxh3a.execute-api.us-east-1.amazonaws.com",
        "deploymentId": "74v610",
        "apiId": "899aggxh3a"
    },
    "body": "{\"foo\":\"bar\"}",
    "isBase64Encoded": false
}

Conclusion

The new VPC Lattice and AWS Private Link integration allows developers to invoke Private APIs directly without needing a Lambda function. This reduces code, maintenance, and latency.

Invoking Private API Gateway Endpoints From Step Functions

Benoît Bouré — Tue, 14 Jan 2025 10:25:55 +0000

At Re:Invent 2024, AWS announced EventBridge and Step Functions integration with private APIs. Thanks to this new feature, customers can now directly invoke APIs that are inside a private VPC from EventBridge (with API destinations), or Step Functions (HTTP Tasks). Before, users had to use Lambda functions inside the VPC as a proxy to their private APIs.

In a previous post, I explained how to invoke HTTP endpoints from Step Functions with the CDK. In this issue, I will cover calling a private API Gateway endpoint using the new integration.

Overview

First, let’s examine how this new integration works and how the different components interact with each other. At the end of this post, I’ll show you how to deploy this setup with the CDK.

Here is a diagram that describes the architecture.

Source

The new private API integration is powered by VPC Lattice and AWS PrivateLink. VPC Lattice has two new features that make connecting Step Functions to a VPC possible: Resource Gateways and Resource Configurations

Resource Gateway

A resource gateway is a point of entry into the VPC where your resources reside. It can span one or more availability zones through the VPC subnets.

To access a private API Gateway from Step Functions, we need a Resource Gateway that lives in the same VPC and subnets as the VPC endpoint that is attached to the API.

Resource Configuration

Once we have a Resource Gateway for our VPC, we can create and attach Resource Configurations to it. Resource Configurations represent resources that are accessible through the gateway, and how they are accessed.

In the case of API Gateway, a resource configuration consists of the VPC endpoint’s regional DNS name. We can also specify a port or range of ports that are accessible, which in our case is just 443 (for HTTPS).

EventBridge Connection

To call HTTP endpoints, Step Functions uses EventBridge Connections. The connection defines the authorization method, and the credentials to access the endpoint. Connections now have a new capability that allows integration with private APIs through a VPC Lattice Resource Configuration.

For API Gateway, the Resource Configuration is the one that defines the API Gateway.

And that’s all we need. Everything else works the same as calling a public HTTP endpoint.

Definition With the CDK

As explained earlier, we need a Resource Gateway that serves as the point of ingress into our VPC. We also create a security group that only allows egress to port 443, which is all we need for this use case:

const rgSecurityGroup = new SecurityGroup(this, 'ResourceGatewaySG', {
  vpc: vpc,
  allowAllOutbound: false,
});

rgSecurityGroup.addEgressRule(
  Peer.ipv4(vpc.vpcCidrBlock),
  Port.tcp(443),
  'Allow HTTPS traffic from Resource Gateway',
);

// Resource Gateway
const resourceGateway = new CfnResourceGateway(this, 'ResourceGateway', {
  name: 'private-api-access',
  ipAddressType: 'IPV4',
  vpcIdentifier: vpc.vpcId, 
  subnetIds: vpc.isolatedSubnets.map((subnet) => subnet.subnetId), // all isolated subnets
  securityGroupIds: [rgSecurityGroup.securityGroupId],
});

We also need a Resource Config that describes the API Gateway’s VPC endpoint.

// Resource Configuration
const resourceConfig = new CfnResourceConfiguration(
  this,
  'ResourceConfig',
  {
    name: 'sf-private-api',
    portRanges: ['443'],
    resourceGatewayId: resourceGateway.ref,
    resourceConfigurationType: 'SINGLE',
  },
);

resourceConfig.addPropertyOverride(
  'ResourceConfigurationDefinition.DnsResource',
  {
    DomainName: Fn.select(
      1,
      Fn.split(':', Fn.select(0, api.vpcEndpoint.vpcEndpointDnsEntries)),
    ),
    IpAddressType: 'IPV4',
  },
);

At the time of writing, the CfnResourceConfiguration L1 construct does not support DnsResource for ResourceConfigurationDefinition, so I’m using an override. For DomainName, we need the regional public DNS name of the VPC endpoint of the API Gateway, which is the first item of the DnsEntries CloudFormation returned value. It’s prefixed with the hosted zone id, so I’m using intrinsic functions to extract the value.

We can now use the configuration in our Step Functions definition:

const connection = new Connection(this, 'ApiConnection', {
  authorization: Authorization.apiKey(
    'x-api-key',
    SecretValue.unsafePlainText('demo'),
  ),
});

(connection.node.children[0] as CfnConnection).addPropertyOverride(
  'InvocationConnectivityParameters',
  {
    ResourceParameters: {
      ResourceConfigurationArn: resourceConfig.attrArn,
    },
  },
);

const http = new HttpInvoke(this, 'Http', {
  apiRoot: api.url, // url of the API Gateway
  apiEndpoint: TaskInput.fromText(`hello`),
  method: TaskInput.fromText('GET'),
  connection: connection,
});

The Connection construct does not support InvocationConnectivityParameters yet, so I’m also using an override here as well.

And here you have it! You can find this code in full on GitHub.

Conclusion

With this new integration with Amazon VPC Lattice and AWS PrivateLink, teams now have the ability to invoke private API Gateway endpoints directly from AWS Step Functions or Amazon EventBridge. This eliminates the need for a Lambda Function, which in turn reduces the amount of code required and decreases overhead. It's a significant step forward for teams looking to optimize and simplify their cloud infrastructure.

I attended re:Invent for the first time in 2023 - Here is what I learned

Benoît Bouré — Tue, 05 Dec 2023 21:54:34 +0000

This year was the one. I finally got the chance to attend re:Invent after several years of big FOMO 🙂. In this blog post, I will share my experience with you, the lessons I learned, and my tips and tricks so that you too (and hopefully my future self) can get the best out of this amazing event in the following years.

In this article, I won't go through the usual stuff like staying hydrated, getting comfortable shoes, and so on. Those topics have been covered many times already. Instead, I will try to cover other aspects I have never been told but I wish I knew.

Timing

Time flies at re:Invent, and it's easy to lose track of time (It's Vegas after all!). Here are a few things you should know for you to better plan your days.

Account for Jet Lag

If you're coming from a distant timezone, be sure you take jet lag into consideration. Coming from Europe, it took me two days to fully recover. The first two days upon arrival, I woke up at around 2 am without being able to go back to sleep (YMMV, maybe it's just me being old 😆). Those days have been very tiring. If you can afford it, try to arrive early enough to fully adjust your biological clock before the beginning of the event.

Reserved Sessions

As you have probably read elsewhere already, reserving sessions is important to make sure that you get a spot at talks you are interested in. But what nobody tells you is that there is a cut-off time for reserved seats. Usually, it's about 15 minutes before the start of the session. I once arrived about 10 min before the start, but it was already too late. They had me go to the end of the walk-up line even though I had reserved. I'd recommend arriving about 20-30 min early. They usually let people with reserved seats in about 10-15 min before the beginning.

If you did not get a chance to reserve and want to walk up, you should show up at least 30 min before the start. There are usually plenty of overflow seats but queues are FIFO. You want to be among the firsts in line.

Also, make sure you have reserved every session you want to attend. For some reason I can't explain yet, I had a bunch of reservable sessions on my calendar that I did not reserve properly and I had to go through the walk-up line. Take the time to plan your sessions well enough in advance.

Extra tip: I've noticed some sessions still had available seats even right before they started. Still, people were queuing in the walkup line. It's worth always double-checking on the AWS events app, just in case. Even if a session is full, save it in your favorites and check it from time to time. As people plan their days, they might unreserve some sessions and you might be able to grab them.

Breakfast, Lunch, and Snacks

Time windows for food are short, in my opinion, and they are also early-ish. Breakfast usually starts at 6 am and ends at 8.30 or 9 am. Lunch starts at 11 am and ends at 1.30 pm.

Those schedules are usually also very strict. I once arrived at lunch near the end, and at 1.31 pm, they were already starting to take everything away (I am not exaggerating). Result: I was deprived of dessert 😞. Don't worry, once you've grabbed food and seated, you can finish your meal at ease.

The same goes for the snacks and coffee you can find everywhere in the venues. They are usually up until 11 am. Same story: I once wanted to pick something up but needed to go to the restroom first. When I can came out, it was all gone 🙂.

What about dinner? Dinner is not included in the event, so where should you eat? Well, there are free sponsored parties almost every day at re:Invent. They usually offer free food and drinks. Just make sure you reserve if needed. Otherwise, the Strip is full of restaurants of all kinds (Ask Paul, he knows all the good plans).

💡 Tip for AWS Community Builders: CBs have a private lounge where you can go get some rest, meet other builders, etc. (this year it was at the Buddy V Ristorante). I have not spent too much time there, but I've been told there was food all day long. So it might be a good backup solution if you need (free) food at any time.

Location and Moving Around

The city and campus are huge. Here are my tips to help you move around.

Choose Your Hotel Wisely

If you have the chance to choose the hotel you will be staying at, or can influence your company to do so, select a hotel that is on or near the campus. It will be much easier to get to and from all the different venues. My hotel was located north of the Strip, a good 30-minute walk from the nearest venue (but conveniently just next to the re:Play ground, so it was not all bad 🙂). It made it a bit harder to get to and from the campus. If you count on getting breakfast at the venues, you might need to leave a bit earlier than expected if your accommodation is located farther away.

Use the DEUCE Bus

If you can't get a hotel close enough, or if you need to move up or down the Strip to go to places that are not covered by the shuttles, I highly recommend you use the DEUCE Bus. It runs every 10 to 15 minutes in both directions (North and South bounds) and covers most venues and important locations. It's very easy to use and much more affordable than Uber. I also found it to be much more convenient, as you can just hop on from the street (at dedicated stops), as opposed to rideshares for which you need to go to the pickup location, which is not always easy to find or reach. Flagging taxis is also illegal on the Strip.

For the bus, a 3-day pass costs $20. You can also get 24-hour passes for $8. Single rides are $4. You can buy them from the rideRTC app. It will give you a QR code you can scan when boarding the bus, next to the driver.

⚠️: This year, due to construction works, I observed high delays at some times, but hopefully, this won't be the case next year. The bus can also be quite packed at times and I've seen people being denied boarding.

Shuttles

As you know, free shuttles are going from one venue to the other. They are very convenient and I never had to wait. However, because of traffic jams, you never know how long it can take to reach your destination. If you want to make it to a session, make sure you leave early enough. Again, this year, it might have been because of the construction work, but you never know.

Expo and Sessions

Record Sessions (that are not already recorded)

I wish I had thought about it earlier, but around the middle of the event, I started recording sessions. It's much easier than taking notes. You can listen back to it later if you want. I used my Android phone recording app which conveniently also creates a transcript.

Note that Keynotes and Breakout sessions are already video recorded and uploaded to the re:Invent YouTube channel a few days later, so you don't need to worry about those.

Pick The Right Sessions

There are lots of sessions to see and you won't be able to see them all. I'd recommend that you choose those that are a bit out of your zone of comfort. For example, I put too much importance into serverless-related sessions, where I already have expertise. In the end, I did not learn a lot new and it almost felt like I lost my time. I should have focussed more on AI/ML where I know almost nothing, for example.

Also, as I mentioned in the previous section, some sessions are recorded. So, unless you want to meet and greet the speaker or have questions, I would favor non-recorded ones. You can always catch up later on YouTube.

Swags

I know it's tempting to grab every single T-shirt and pair of socks out there at the Expo. However, think about it: When will there be an opportunity to wear them? I usually don't. In the best-case scenario, I'd use them as PJs. So, I tried to refrain from the temptation to take everything I could find. If it's going to end up in a drawer, it's not worth it, and not sustainable. Be responsible. If you think you'll give it a good use, then sure, have fun!

Also, note that there are swag donation boxes next to the Expo entrance. If you can't refuse a swag that is given to you, or change your mind later, you can deposit them there. I am not sure what they do with them but they will probably go to some charity.

Sponsors: Bring Polyglots!

This one is more for sponsors that have a booth at the Expo. Serverless Guru had a booth for the first time this year. The whole team helped talk to people visiting us and interested in our services. What I loved the most was meeting people from all around the world. Surprisingly, some people did not or barely spoke English. Fortunately, we had an amazing international team speaking several languages: French, Spanish, Portuguese, Bosnian, and even Japanese. That really helped make an impact! 💪

Even if you don't have to attend people at a booth, it might be worth learning a few words in different languages to help start some conversations 😉.

Networking, Partying and Fun

It's Never "One Beer" 🍺.

Let's be honest, one of the best things about re:Invent is being able to meet all your virtual friends in person. Therefore it's very tempting to go have a few drinks together. There is usually a party almost every day (believe in serverless, Re:Play, etc). Go out for "one beer", and the next thing you know is that you wake up the next morning with a hangover. Days are busy at re:Invent. You want to be in good shape and full of energy for the next day to be able to attend all the sessions, visit the expo, etc.

It does not mean you can't party and have fun. What has worked for me is to take it easy on alcohol, and even if I had alcohol, alternate with water or softs. You'll thank me later 😉.

Pro tip: It does not have to be "boring". There are always some delicious alcohol-free cocktail alternatives.

You Can't Meet With Everyone

There are so many people at re:Invent that you probably won't be able to meet with all the people you wish. I missed so many people. I was even able to see some of them from a distance without being able to interact with them.

If there are some people you absolutely want to meet, I'd recommend you use the appropriate tools for that and plan in advance. e.g. use Peer Talk, or just send them a DM/email and ask them to meet for 10 min somewhere you agree.

Bring a Taste of Home

Some people I met greeted me with some sweets they brought from their home country. I thought it was such a great idea! It helps break the ice, know new cultures, and start conversations. I'll make sure I do it next year for sure!

Re:Play

You should know that the party takes place outside, at the Las Vegas Festival Ground. It can be very chilly and windy. Make sure you bring a jacket.

The different music tents can be very loud too. Please protect your hearing. You can find earplugs at the First Aid tent. I swear it won't ruin anything and you will still enjoy the music.

Last tip: The party starts at 7.30 pm and ends at midnight, but the bar closes at 11 pm. Make sure to grab that last drink before then 🍸.

After the Event

Do Something Fun

If you have time, explore and enjoy Vegas. Do something fun with your teammates! It might be a long time before you see each other again. My team and I went for some kart racing 🏎. They kicked my ass, but it was a lot of fun! We ended the day with a nice dinner.

Take some Rest

Re:Invent is an intense week. Take some time to get some rest when you're back home. If possible, avoid getting back to work immediately. Spend time with your family and chill.

Conclusion

Re:Invent is one of the most exciting events I attended. I am so grateful to Serverless Guru for making this unique opportunity possible 🙏. I hope I can be back next year, meet more of you, and put all these lessons learned into practice. If you happen to be there, let me know!

Everything You Should Know About the AppSync JavaScript Pipeline Resolvers

Benoît Bouré — Sun, 27 Nov 2022 20:09:27 +0000

In November 2022, AWS announced JavaScript support for AppSync Resolvers. It was a long-awaited feature that everyone was excited about.

In this article, I will try to shed some light on this new feature, what JS resolvers are, how they can improve developer experience, what possibilities they unlock; but also their limitations.

What JS Resolvers Are Not

Before we start, it is important to make some clarification about what JS resolvers are not. There might be some misconceptions that will arise, mostly because they allow writing resolvers in a way that many developers already write their Lambda functions with (i.e. JavaScript). We should clear that out of the way from the beginning: JS resolvers are not a replacement for Lambda function resolvers, they will never be. You won't be able to do whatever you want; they come with a set of limitations that we will explore later. They are, however, a good alternative to Lambda in some cases and should simplify the choice between both options.

What Are JS Resolvers, Then?

They are a new way for AppSync to communicate with your data sources. This is what we have always known as “mapping templates”; except that instead of writing them in the VTL language, you can now use (a subset of) JavaScript, which brings tons of benefits (see below). You still need to attach your JS resolvers to a data source, and the only thing they should do is generate a request for that data source and format the response it returns.

From the doc:

The request handler takes a context object as an argument and returns a request payload in the form of a JSON object used to call your data source. The response handler receives a payload back from the data source with the result of the executed request

What benefits do they bring?

Now that we made clear what JS resolvers are, let's talk about their benefits; because there are! (other than "I don't like VTL 🤓")

Better Developer Exprience

Being able to write AppSync handlers in JavaScript obviously makes it easier for everyone. JS is a language that most developers are familiar with, and it has so many benefits over VTL such as better IDE support or code linting. If you want type support, you can also write them in TypeScript as long as you transpile them to JavaScript later.

Room for extension

JavaScript makes it possible to write reusable code by creating custom functions, utilities, or libraries. This was not (easily) possible with VTL. This should make it easier to write better code faster.

No cold start and no extra cost

JS resolvers, unlike Lambda functions, will not suffer cold start, and there is no penalty for choosing JS over VTL in terms of performance. They are also available at no extra cost. Being able to write resolver handlers as JS will potentially allow many to get rid of unnecessary Lamba functions; reducing latency and cost at the same time.

Limitations

If JS resolvers have some benefits, you should be aware of their limitations. These limitations might evolve over time, but some of them will likely never go away. Keep in mind that if JS resolvers don't allow you to do some things, it might as well be to protect you from using bad practices.

Only Pipeline resolvers are supported

For the time being, only Pipeline resolvers are supported. Unit resolvers still require the usage of VTL mapping templates.

Hint: 💡 If you still want to use JS with a unit resolver, you can always write a pipeline resolver that has only one function.

Unsupported JS features

It is primordial to remember that JS resolvers use a subset of ECMAScript 6. It means that all features are not available.
For example, you can't use: for loops (for-in and for-of are supported), try, catch, finally, continue, do-while or while loops.

throw is also not supported. If you want to "throw" an error, you should use the util.error() util, which is basically the equivalent to the $ctx.util.error() that you already know.

Arrow functions are not supported either.

No Async/Await

async/await are not supported, for obvious reasons. JS resolvers should be completely synchronous and return as fast as possible.

No network access

You won't be able to do any network request in a JS resolver (e.g. you can't use fetch/axios).

💡 Hint: If you need to do that you probably want to use an HTTP resolver instead.

Size limitation and import

If you follow the above rules, you are free to do everything you want. While you can't use the import statement (except with @aws-appsync/utils), you should be able to include custom libraries and code if you bundle them in a single file before you upload them to AppSync. (e.g. using esbuild).

There is a big catch though: all the code in the bundle MUST be a valid ES6 module, follow all the above AppSync JS rules, and the final package must not exceed 32KB.

Unfortunately, this means that you will likely never be able to include almost any npm package out there at the moment. But, it opens the door to creating dedicated AppSync-compatible libraries that strictly follow the requirements. A quick POC shows that this is possible.

// Detect dark theme var iframe = document.getElementById('tweet-1594686086000828423-763'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1594686086000828423&theme=dark" }

Please refer to the documentation for a full list of supported and non-supported features.

What Else Has Changed?

The AppSync utils

In VTL, we were used to invoking functions like $util.dynamodb.toDynamoDBJson(). Those utilities still make sense and are useful in JS. They now live in a new npm package: @aws-appsync/utils.

This new package also contains an API for interacting with AppSync: extensions.

CloudWatch logs

Those who, like me, were used to diving into the CloudWatch AppSync logs will notice new log types: BeforeRequestFunctionEvaluation, RequestFunctionEvaluation, ResponseFunctionEvaluation, AfterResponseFunctionEvaluation.

These log records are equivalent to the ones that you used to see when debugging VTL mapping templates (namely BeforeMapping, AfterMapping, RequestMapping, and ResponseMapping), but they are specific to JS resolvers. They look roughly the same as their VTL counterpart.

{
  "logType": "RequestFunctionEvaluation",
  "fieldName": "getPost",
  "resolverArn": "arn:aws:appsync:us-east-1:12345678912:apis/abcdefghijklmnop/types/Query/resolvers/getPost",
  "functionName": "getPost",
  "fieldInError": false,
  "evaluationResult": {
    "operation": "GetItem",
    "key": {
      "id": {
        "S": "b03a2a76-1f2c-4150-9b3c-aab88a04f49e"
      }
    }
  },
  "parentType": "Query",
  "path": [
    "getPost"
  ],
  "requestId": "83e25c50-8d97-4939-8475-440c0ed8b36d",
  "context": {
    "arguments": {
      "id": "b03a2a76-1f2c-4150-9b3c-aab88a04f49e"
    },
    "prev": {
      "result": {}
    },
    "stash": {},
    "outErrors": []
  },
  "errors": [],
  "graphQLAPIId": "ykwu6mw2mrc3ho4jczeg6p42pa",
  "functionArn": "arn:aws:appsync:us-east-1:12345678912:apis/abcdefghijklmnop/functions/uvwyyz"
}

You can also write custom logs by using the console.log() function. You will see them appear in CloudWatch.

83e25c50-8d97-4939-8475-440c0ed8b36d INFO - code.js:5:3: "Hello from JS Resolver!"

Conclusion

JS resolvers will greatly improve the developer experience. They should help developers adopt direct integration with data sources (e.g. DynamoDB) more often, while Lambda functions are still useful for more advanced and complex use cases.

There is still a lot to explore, though. This is only the beginning. I am sure the community will come up with great ways to use them and extend them 🚀

Credits: Cover image generated by dall·e

Securely Access Your AWS Resources From Github Actions

Benoît Bouré — Mon, 27 Dec 2021 15:15:33 +0000

Security is a very important topic for all cloud engineers. Making sure that your infrastructure and data are kept out of reach of malicious people is one of the most serious things to get right. In AWS, we are used to dealing with IAM roles and permissions that make our resources accessible to users or to other resources. However, sometimes you need to grant access from outside your organization.

One example is when you want to deploy your infrastructure from a CI/CD pipeline, like Github Actions. How do you allow your workflow to gain access to your AWS account?

One approach is to create a dedicated IAM user, store its credentials in the Github secrets store, and allow the workflow to use them. Easy, enough! Secrets are encrypted by Github, so it is secure, right?

Not really... The problem is that those credentials are meant to be long-lived. It means that if anyone is able to get hold of them for whatever reason (eg: a leak in a workflow logs, someone gaining access to a GitHub action runner, etc), they will be able to access all your resources (at least those that the credentials are allowed to control). Sure, you could rotate them from time to time, but you'd have to do that manually. This is probably not something you want to spend time doing, and let's face it, you probably won't!

Luckily, there is a better solution. If you are using Github Actions, you can allow Github to grab temporary, short-lived, credentials that it can use during the execution of the workflow. After that, the credentials will expire and no one will ever be able to use them again.

In this post, I will guide you through the steps to set this up. Don't worry, it's actually easier than you think!

Here is a schema representing what we are going to accomplish

Setting up your AWS account

💡 TL;DR; I created a CloudFormation quick-create link that you can use to automate the following steps. See at the bottom of this article. If you want to know how it works, and what CloudFormation is going to do, keep reading this section.

Create an OpenID Connect Identity provider

The first step is to create an OpenID Connect (OIDC) identity provider in your AWS Account. This will allow Github to identify itself.

Got to the IAM console -> Identity providers
Click Add new provider
Select OpenID Connect
Provider Url: https://token.actions.githubusercontent.com (Don't forget to click Get Thumbprint)
Audience: sts.amazonaws.com
Add tags if you want to and click Add Provider

💡 You will need to do this step only once per AWS account.

Create a role

You now need to create a role that Github will be able to assume in order to access the resources it needs to control.

Go back to IAM and select Roles
Create a new Role
Chose Web Identity, select the Identity provider you created in the previous step, and its audience.
Click Next:Permissions

You now need to give the role the appropriate permissions (Policies). These are the ones that Github needs in order to do whatever it has to do. This will vary based on your use case, so I will leave that up to you. Keep in mind that you should stick to the principle of least privileges.

When that is done, give your role a name and click Create Role.

There is now an additional step to do. You need to edit the trust policy of the role to reduce its scope to your repository only. Make sure you don't skip this part, it is very important. Without that, any repository on GitHub will be able to assume your role and access your resources. (Unfortunately, there does not seem to be a way to do that at creation time).

Go back to IAM Roles and select the created Role. Choose Trust Relationships and Edit Trust Relationship.

Under Condition, add the following segment:

"StringLike": {
  "token.actions.githubusercontent.com:sub": "repo:[your-org]/[your-repo]:*"
}

Replace the organization and repo names to match yours, and click Update Trust Policy.

✍️ Note: You can take this even further and reduce the scope, by using git references, to a branch or tag only, for example.
eg: repo:[your-org]/[your-repo]:ref:refs/heads/master

The final result will look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::1234567890:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:[your-org]/[your-repo]:*"
        }
      }
    }
  ]
}

This concludes the required configurations on your AWS account. Take note of the role ARN, you'll need it later.

💡 You can create different roles per account and use a different one for each use case. For example, one per application, per usage (configurations, deployment, integration tests), etc. You can play with that to reduce the scope of each session even more.

Configure Github action workflow

Your Github workflow requires additional permissions in order to be able to use OIDC. Add the following at the top of your workflow's YML file. You can also add it at the job level to reduce the scope if needed.

permissions:
  id-token: write # required to use OIDC authentication
  contents: read # required to checkout the code from the repo

You can now use the configure-aws-credentials Github action in the job that needs to assume the role. Add this step to generate credentials before doing any call to AWS:

- name: configure aws credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    role-to-assume: arn:aws:iam::1234567890:role/your-role-arn
    role-duration-seconds: 900 # the ttl of the session, in seconds.
    aws-region: us-east-1 # use your region here.
# You can now execute commands that use the credentials👇
- name: Serverless deploy
  run: sls deploy --stage dev

The configure AWS credentials step will use the OIDC integration to assume the given role, generate short-lived credentials, and make them available to the current job.

💡 If you want to take security even further, you can also keep your role's ARN used in role-to-assume in a Github secret.

Automate

The guys at configure-aws-credentials shared a CloudFormation template that you can use to automate the AWS configuration steps.

I took it one step further; I hosted that template and created a deployment link for you.

Click here to deploy it into your account.

Fill in the parameters:

GitHubOrg: your organization name, or your Github username
RepositoryName: the repository that needs access to your AWS account
OIDCProviderArn: your existing OIDC provider's ARN, if you have one already. If you don't, leave it empty and one will be created for you. (Remember that you only need one per account).

✍️ Note: The created role will not have any Policy attached to it. You will still need to attach the ones that your workflow needs to it after that.

Conclusion

As you can see, securing your account doesn't have to be hard. The part that might require a little more effort is to define the right Policies if you want to follow the principle of least privileges (which you should!).

For more content like this, follow me on Twitter @Benoit_Boure.

How to Observe EventBridge Events with AppSync Subscriptions

Benoît Bouré — Thu, 07 Oct 2021 08:18:28 +0000

I recently came across David Boyne's blog post: How to Observe EventBridge Events with Postman and WebSockets. What a great idea! But, then I thought:

I can do the same with AppSync Subscriptions!

I had to try! Here is what I achieved:

Building the basic AppSync API

The idea was simple. I needed the following components:

An AppSync API
A Mutation that receives events from EventBridge
A Subscription that is attached to the aforementioned Mutation
An EventBridge rule that sends events to the AppSync Mutation (target)

I also wanted to be able to filter events I was interested in. Here, I thought about two options:

Filter the events in the EventBridge rule.
Send all events to AppSync and use AppSync to filter them, thanks to subscription arguments

I went with the second approach. It would give me more flexibility to filter the events at query time instead of having to re-deploy each time I wanted a new filter.

Here is the GraphQL Schema I created:

type Mutation {
  sendEvent(event: EventBridgeMessageInput!): EventBridgeMessage
}

type Subscription {
  subscribe(
    source: String
    detailType: String
    account: String
    resources: [String!]
  ): EventBridgeMessage
    @aws_subscribe(mutations: ["sendEvent"])
}

type EventBridgeMessage {
  id: ID!
  version: String!
  detailType: String!
  source: String!
  account: String!
  time: AWSDateTime!
  region: String!
  resources: [String!]
  detail: AWSJSON!
}

input EventBridgeMessageInput {
  id: ID!
  version: String!
  detailType: String!
  source: String!
  account: String!
  time: AWSDateTime!
  region: String!
  resources: [String!]
  detail: AWSJSON!
}

I also needed to setup the Mutation. I used a NONE data source for that and a simple mapping template that just returns the received payload.

All done! Now, by executing the sendEvent Mutation, it gets delivered to the subscription! 🙌

All that was left to do was to configure EventBrige and set the Mutation as a target.

First attempt: API Destinations

My first attempt was to use API Destinations. I followed this awesome tutorial and defined my Input Path and Input Transformer rules which looked like this:

InputPathsMap:
  version: $.version
  id: $.id
  detailType: $.detail-type
  source: $.source
  account: $.account
  time: $.time
  region: $.region
  resources: $.resources
  detail: $.detail
InputTemplate: |
  {
    "query": "mutation SendEvent($event: EventInput!) { sendEvent(event: $event) { version id detailType source account time region resources detail } }",
    "operationName": "SendEvent",
    "variables": {
      "event": {
        "version": "<version>",
        "id": "<id>",
        "detailType": "<detailType>",
        "source": "<source>",
        "account": "<account>",
        "time": "<time>",
        "region": "<region>",
        "resources": "<resources>",
        "detail": <detail>
      }
    }

Unfortunately, that didn't work! 😞

The problem is that in EventBridge, the detail attribute is an arbitrary JSON object which could have any shape. This is the reason I used an AWSJSON type in my GraphQL schema (I wanted to receive any event). The problem is that AppSync expects the JSON to be stringified!

After some investigation, I could not find any way for EventBridge to stringify JSONs. So, that was a dead end.

AWS Lambda to the rescue!

If EventBridge cannot do it, Lambda surely can! So, I wrote a simple lambda that receives the event, reformats it and calls the AppSync endpoint. I then just configured the Lambda as an EventBridge target. (See the code here).

✍️ Note: I also added an IAM authentication method to the AppSync API that Lambda can use to call the Mutation (in addition to the API key used by the subscription).

All set! Now, running the following subscription:

subscription MySubscription {
  subscribe {
    resources
    region
    source
    version
    detailType
    detail
  }
}

And sending an event into Event Bridge

aws events put-events --entries '[{"DetailType": "my.detail.type", "Source": "my.source", "Detail": "{\"foo\": \"bar\"}"}]'

Response:

{
  "data": {
    "subscribe": {
      "resources": [],
      "region": "us-east-1",
      "source": "my.source",
      "version": "0",
      "detailType": "my.detail.type",
      "detail": "{\"foo\":\"bar\"}"
    }
  }
}

It works! 🎉

The power of AppSync subscriptions

One of the great features of AppSync subscriptions is that you can specify which changes you are interested in at query time. You can do that by adding arguments to the subscription endpoint. Whatever value you pass in the input, you will only receive changes that match the Mutation's response fields values.

So, I can now do queries such as

## Will match events with detail-type = "my.detail" only
subscription {
  subscribe(detailType: "my.detail") {
    id
    detailType
    detail
  }
}

## Will match events with source = "my.source" only
subscription {
  subscribe(source: "my.source") {
    id
    detailType
    detail
  }
}

## Will match events with detail-type = "my.detail" AND source = "my.source"
subscription {
  subscribe(detailType: "my.detail", source: "my.source") {
    id
    detailType
    detail
  }
}

Isn't that great? I can now listen to exactly the events I am interested in 🔥

Limitations & gotchas

Unfortunately, this technique has some limitations. It cannot filter events based on the content of the detail field. This is because the data comes stringified.

Also, filters only work when the values exactly match. You cannot use advanced filters such as prefix, anything-but, etc. These are filters supported by eventBridge, not by AppSync.

Note that any advanced filter can still be achieved through filters at the EventBridge rule level, of course!

Conclusion

In this post, I showed you how we can observe EventBridge events through AppSync subscriptions and how we can even filter them at query time. Although its usage is somewhat limited, it can probably still be very helpful when you only need to filter on the detailType or source values, for example. You can easily use it to debug/test your application.

Find the full code of this implementation on Github

A big thanks to David Boyne for the inspiration!

5 Ways to Prevent Accidentally Deleting Your CloudFormation Resources

Benoît Bouré — Mon, 22 Mar 2021 19:51:25 +0000

CloudFormation is an AWS service that allows you to maintain Infrastructure as Code (IaC). Whether you are using it natively (with JSON or YML) or through a third-party service such as the Serverless Framework, AWS CDK or SAM, it is a great way to make your infrastructure reproducible across various stages. It also makes the deployment process easily automatable through CI/CD pipelines. In other words, it makes managing your infrastructure less prone to human errors.

Although automating things sounds like a good idea, one of the downsides of CloudFormation is that it is hard to understand what is going on under the hood and what exactly is going to happen to your stack during the process, turning every single deployment into a potential 7 minutes of terror story. Imagine that an entire resource gets deleted and all its data with it. Make just one mistake and you will only find out when it's too late. What if it is a production database? It happens more than you think. 😱😱😱

To avoid this kind of disasters, I will show you 5 ways to protect your resources from deletion with CloudFormation.

1. Review the Changeset

The first technique is to understand which actions will effectively be executed during the update before they happen. CloudFormation offers a tool that lets you pre-visualize all the modifications that would be applied by a change in your template.

To use it, follow these simple steps:

go to your CloudFormation console and select the stack that you want to update
click the Stack actions button and then select Create change set for current stack.
Choose Replace current template and upload your new template, or enter an S3 path to the file.
From there, just follow the guide in order to create the changeset

It might take a few seconds for the changeset to be generated. Once it is done, the console will show you a detailed summary of what actions would be executed if you decided to proceed with the update.

Example:

As you can see, you can easily spot what resources will be Modified, or Removed and if they require replacement. Once you are confident enough that this is what you intend to do, you can hit the Execute button with a certain peace of mind 🧘

This method is useful when you want to visually confirm a change that you are unsure about. However, it is not always convenient. Let's explore other solutions.

2. Retain Specific Resources

With the DeletionPolicy attribute, you can control what CloudFormation should do with a resource in the event of it being removed from the template, or if the stack is deleted altogether. The default value is Delete which is probably not what you want in some cases. By changing the value to Retain, you are telling CloudFormation to keep the resource instead.

Example:

Resources:
  MyTable:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain
    Properties:
      TableName: mytable

One thing to notice here is that this method will not make your deployment fail. CloudFormation will execute all your changes. The difference is that any instruction to delete a resource with a Retain policy will be ignored and the resource will be "detached" from the stack instead. This also means that if you try to add the resource back to the stack, any subsequent deployment might fail because CloudFormation will try to re-create the resource that already exists (e.g: the DynamoDB table already exists with that name). If that happens, you can check this guide for Importing Existing Resources into a CloudFormation Stack.

⚠️ Attention!

This capability doesn't apply to resources whose physical instance is replaced during stack update operations. For example, if you edit a resource's properties such that CloudFormation replaces that resource during a stack update.

This extract from the official documentation is very important. What it means is that if you change a property of a resource that requires replacement (e.g.: changing a DynamoDB table's name), the deletion policy will not apply, and it would still be deleted and re-created. Before you change a property, you should pay attention to the Update requires section of the CloudFormation documentation for that resource's attribute.

With certain types of resources, like EC2 volumes or RDS instances, you can also use Snapshot. In that case, the asset would still be deleted but a backup would be executed first.
You can read more about this strategy by reading the official documentation.

3. Define a Stack Policy

A more advanced way of protecting your resources is through Stack Policies. With Stack Policies, you can constraint what actions are allowed to be executed or not according to specific rules that you define. When you add a policy, all resources are protected by default. You need to explicitly Allow the changes on the resources that you want to update. You can think of it as an IAM policy, but the difference here is that it only applies during stack updates.

Example:

The following policy allows any change on all resources, except for the resource whose id is MyDynamoDBTable. By explicitly denying Update:Delete and Update:Replace, the resource is protected against deletion and replacement. On the other hand, modifications are still allowed (e.g.: Add a Global Secondary Index).

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["Update:*"],
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Delete", "Update:Replace"],
      "Principal": "*",
      "Resource": ["LogicalResourceId/MyDynamoDBTable"]
    }
  ]
}

To learn how to write custom stack policies, refer to the documentation

4. Enable Stack Termination Protection

If all you worry about is someone (or a process) tearing down a whole stack by mistake, what you need is Stack termination protection. When enabled, CloudFormation will reject any attempt of deleting the stack.

To enable termination protection:

Go to CloudFormation and select the stack that you want to protect.
Chose Stack actions followed by Edit termination protection
Chose Enabled and hit Save

5. Place Sensitive Resources in Different Stacks

Last but not least, if you are too paranoid about deleting precious resources and all the data they contain, the best thing you can do is isolate them into their own stack. Place each one of them in a dedicated template and touch them only if and when you need to. By doing so, you will not risk destroying them while deploying other stacks that change more often.

Which One Should You Use?

Each solution has its own pros and cons. They also behave differently in different situations. To help you better understand the differences, I created a simple cheat sheet.

Will my resource be protected if	It is removed from the stack	It requires replacement	The stack is deleted
Changeset review	Manual (1)	Manual (1)	No
DeletionPolicy	Yes	No	Yes
Stack Policy	Yes (2)	Yes (2)	No
Stack Termination Protection	No	No	Yes
Resource Isolation	No (3)	No (3)	No (3)

(1) You will need to manually review and approve the changes.

(2) Provided you configure the policy properly

(3) On its own, resource isolation will not protect any resource. You'll need to combine it with other solutions

As you can see, there is no one-fits-all solution (none of the rows has all Yeses). You will need to use more than one if you want full protection.

Conclusion

I just showed you 5 ways to avoid accidental deletion of CloudFormation resources:

Review the changeset is good if you want to sporadically review changes manually before applying some important changes.
The DeletionPolicy attribute will save your data in the event of a resource removal or stack deletion, but it won't help against resource replacement.
Stack Policies will save you from accidentally removing a resource from the stack and changes that force a replacement. On the other hand, it won't be of any help if the stack is deleted altogether.
Stack termination protection will only prevent accidental deletion of the stack.
Placing sensitive resources in isolation will help against some human mistakes, but on its own, it will not protect your data.

Use the one that best fits your needs and your particular use-cases. If you need complete protection, you can combine them together and benefit from several safety nets at the same time.

Hopefully, these measures will help you and your team sleep better at night 😴.

If you would like to read more content like this, follow me on Twitter and subscribe to my brand new newsletter on Hashnode.

How to Store Large Attribute Values in DynamoDB

Benoît Bouré — Mon, 15 Mar 2021 07:22:18 +0000

DynamoDB is a fully managed NoSQL database that delivers single-digit millisecond performance at any scale. In order to keep up with its promises, there are a couple of constraints and good practices that you need to follow. One of them is to keep your items as small as possible. This is true not only for performance but also for cost. With DynamoDB, you pay per amount of data that you read or write as well as for storage. Reducing your data size is important if you want to reduce your monthly bill.

On top of that, DynamoDB also comes with some hard-limits including:

Any item cannot exceed 400 KB in size.
Query and Scan operations are limited to 1 MB of data scanned (After that, you will be forced to paginate).

If you handle large amounts of data, you can hit those limitations very quickly.

For example, imagine that you are building a blog application (like Hashnode). You might store posts and comments in a DynamoDB table. These kinds of items contain free text that can be quite long and grow very fast. A blog post can easily reach 10 to 20 KB or more. When you know that half an RCU allows you to read 4 KB of data (providing that you are doing eventually consistent reads), we are talking about 2 to 3 RCUs for every read, if not more if you have several other attributes!

When dealing with such large data, AWS recommends compressing them and storing them as Binary attributes. In this blog post, I will show you how to compress long text strings with gzip and how to store them into DynamoDB. We will then inspect the read and write units consumed and compare them with the corresponding uncompressed version.

1. Writes

In this demo, I'll be using node.js but you should easily be able to apply these techniques to your favourite programming language.

For the purpose of this test, we'll write a simple script that will first generate a dummy blog post using lorem-ipsum. We will then save it to DynamoDB twice: once as raw text (uncompressed) and once compressed with gzip. We will also make use of the ReturnConsumedCapacity property of DynamoDB so that it returns the consumed capacity (WCU) for both operations.

//write.js
const AWS = require('aws-sdk');
const { loremIpsum } = require('lorem-ipsum');
const { gzipSync } = require('zlib');

const content = loremIpsum({
    count: 20,
    units: "paragraph",
    format: "plain",
    paragraphLowerBound: 5,
    paragraphUpperBound: 15,
    sentenceLowerBound: 5,
    sentenceUpperBound: 15,
    suffix: "\n\n\n",
});

// output some stats about the text's lenght.
console.log(`Generated a text with ${content.length} characters and ${content.split(' ').length} words`);

// compress the content
const compressed = gzipSync(content);

// more stats about the content
console.log(`total size (uncompressed): ~${Math.round(content.length/1024)} KB`);
console.log(`total size (compressed): ~${Math.round(compressed.length/1024)} KB`);

// config DynamoDB
AWS.config.update({ region: 'eu-west-1' });
const dynamoDbClient = new AWS.DynamoDB();

dynamoDbClient.putItem({
    "TableName": "blog",
    "ReturnConsumedCapacity": "TOTAL",
    "Item": {
        "author": {
            "S": "bboure"
        },
        "slug": {
            "S": "raw-blog-post"
        },
        "title": {
            "S": "My blog post"
        },
        "content": {
            "S": content,
        }
    }
}).promise().then(result => {
    console.log('Write capacity for raw post', result.ConsumedCapacity );
});


dynamoDbClient.putItem({
    "TableName": "blog",
    "ReturnConsumedCapacity": "TOTAL",
    "Item": {
        "author": {
            "S": "bboure"
        },
        "slug": {
            "S": "compressed-blog-post"
        },
        "title": {
            "S": "My blog post"
        },
        "content": {
            "B": compressed,
        }
    }
}).promise().then(result => {
    console.log('Write capacity for compressed post', result.ConsumedCapacity );
});

In the script above, we generate a text of 20 paragraphs. Each paragraph will have between 5 and 15 sentences, and each sentence will be 5 to 15 words long. That is enough to generate a text of around 2000 words. Then, we compress the text and save both versions into DynamDB.

Let's run the script:

$ node write.js
Generated a text with 12973 characters and 1943 words
total size (uncompressed): ~13 KB
total size (compressed): ~4 KB
Write capacity for compressed post { TableName: 'blog', CapacityUnits: 4 }
Write capacity for raw post { TableName: 'blog', CapacityUnits: 14 }

As you can see, the raw text was around 13 KB and consumed 14 WCUs, while the compressed one only 4 KB for 4 RCUs. That looks right since 1 WCU computes for 1 KB of data.

By compressing the data we just saved ourselves 10 WCUs. That's a 70% gain! Not only that, but we also reduced the item size by 70%! Since DynamoDB also charges us for storage, that can make a huge difference on our AWS bill! 🎉

2. Reads

Now that we saved our blog post in DynamoDB we want to read it back. Let's create a new script that will read the items back and see how many RCUs they are consuming.

//read.js
const AWS = require('aws-sdk');
const { gunzipSync } = require('zlib');

AWS.config.update({ region: 'eu-west-1' });
const dynamoDbClient = new AWS.DynamoDB();

dynamoDbClient.getItem({
    "TableName": "blog",
    "ReturnConsumedCapacity": "TOTAL",
    "Key": {
        "author": {
            "S": "bboure"
        },
        "slug": {
            "S": "raw-blog-post"
        },
    }
}).promise().then(result => {
    console.log('Read capacity for raw post', result.ConsumedCapacity );
});


dynamoDbClient.getItem({
    "TableName": "blog",
    "ReturnConsumedCapacity": "TOTAL",
    "Key": {
        "author": {
            "S": "bboure"
        },
        "slug": {
            "S": "compressed-blog-post"
        },
    }
}).promise().then(result => {
    console.log('Read capacity for compressed post', result.ConsumedCapacity );
    // uncompress post content
    const content = gunzipSync(result.Item.content.B).toString();
    console.log(`Original text with ${content.length} characters and ${content.split(' ').length} words`);
});

Let's run it:

$ node read.js
Read capacity for compressed post { TableName: 'blog', CapacityUnits: 0.5 }
Original text with 12973 characters and 1943 words
Read capacity for raw post { TableName: 'blog', CapacityUnits: 2 }

At read time, we only consumed 0.5 RCUs against 2 for the uncompressed version. That's 4 times less! And as you can see, it is just as easy to uncompress the data back into its original form.

3. Secondary indexes

Before we call it a day, there is one last test I'd like to make. Sometimes, you want to add secondary indexes to your table. In our blog example, we could add a GSI that will index blog posts by author and sort them by timestamp. One could argue that you should probably avoid projecting the entire blog content in all your indexes (and I would definitely agree with that), but sometimes, you might not have a choice; and for the sake of completeness, we'll try it out.

Let's create another script that will test just that. I'm not going to copy the full script again here. Instead, just know that I added a timestamp attribute and I created a GSI index that projects all the attributes (Index name: timestamp, PK: author, SK: timestamp).

$ node write.js
Generated a text with 13673 characters and 1986 words
total size (uncompressed): ~13 KB
total size (compressed): ~4 KB
Write capacity for compressed post {
  TableName: 'blog',
  CapacityUnits: 12,
  Table: { CapacityUnits: 4 },
  GlobalSecondaryIndexes: { timestamp: { CapacityUnits: 8 } }
}
Write capacity for raw post {
  TableName: 'blog',
  CapacityUnits: 42,
  Table: { CapacityUnits: 14 },
  GlobalSecondaryIndexes: { timestamp: { CapacityUnits: 28 } }
}

As you can see, GSIs can be greedy in capacity units. That is because every write you make must be replicated to all your indexes. Our secondary index alone consumed 8 WCUs for the compressed post and a whopping 28 WCUs for the uncompressed version! Add the 4 and 14 WCUs that correspond to the table to that and we are at 12 vs 42 WCUs!

Note: To be honest, I was expecting the GSI to consume the same amount of WCU as the table index (i.e.: 4 and 14). For some reason that I still don't understand, that amount is doubled. I could not find any information about why that happens. If you happen to know, please don't hesitate to drop a comment below. 🙏

Here, even though the saving in terms of percentage is about the same (70%), the difference of capacity units starts to increase. We consumed 30 WCUs less with compressed content! Over time, this can quickly make a difference.

Note that here, there would be no difference in terms of RCUs when reading the data back. DynamoDB will read from the index that you provide in the query, and that index only.

Conclusion

We just learned that by compressing large contents before saving them in DynamoDB, you can expect saving up to 70% in WCU, RCU and storage cost. This is significant enough to take the time and make the extra effort of compressing/decompressing the data as you read/write it.

If you'd like to read more content like this, follow me on Twitter