Forem: Jason Butz

New DynamoDB Key Feature & Why It Matters

Jason Butz — Mon, 24 Nov 2025 22:00:00 +0000

AWS announced multi-attribute composite keys for DynamoDB global secondary indexes on November 19th. That's a lot of words, but what does it actually mean? It means it's easier to pull data out of DynamoDB in ways you hadn't planned initially, without a bunch of extra work. To fully explain why, I need to explain keys in DynamoDB. I'll assume you know that DynamoDB is a NoSQL database, that the "database" is called a table, and the table is a collection of items, which you can think of as records or rows.

Primary Key, Partition Key, Sort Key

You'll hear many names used for different keys in DynamoDB. It all makes sense, but it can be confusing. Every item in your table has a primary key, as you are probably used to with relational databases such as MySQL, PostgreSQL, Microsoft SQL Server, Oracle, and many others. That primary key is a unique value you can use to retrieve a particular item. In DynamoDB, the primary key can consist of either a single attribute or two attributes.

Your primary key always has a partition key (PK), sometimes called a hash key or hash attribute. The partition key is used by DynamoDB to distribute your items across partitions, enabling the performance for which DynamoDB is known. There are a whole lot of best practices for choosing a partition key, but you don't need to know them right now.

Your primary key might include a sort key (SK), sometimes called a range key or range attribute. If your primary key has a sort key, then it is known as a composite primary key. With a composite primary key, multiple items can have the same partition key. The combination of the partition key and sort key is what must be unique. The sort key determines how values returned are sorted when you query for the items with a given partition key.

Below is a screenshot using sample data AWS provides that shows an example table with only a partition key. In the example, the partition key is the user's username, LoginAlias, as shown in the screenshot. This allows you to easily retrieve a user's information if you know their login alias.

The following screenshot shows the same data, with the partition key as the user's first name and the sort key as their last name. This allows you to easily find users by name, for example, all users named Jane.

Get, Query, Scan

Before I get into key structures, I have to cover the three primary operations for reading data from DynamoDB. You can either use a GetItem action to read a single item, use the Query API to retrieve multiple items based on primary key values, or use the Scan operation.

For GetItem, you provide the entire primary key, and DynamoDB returns your item. It's very straightforward and very fast.

For a query, you must know the partition key value, but you can perform more complex comparisons for the sort key value, as well as for other attributes on the items. Using the screenshot above as an example, you could query for all users with the first name "Jane" and a last name that begins with "R", or for all users with the first name "Jane" who have "software" as a skill. Query operations are generally very fast, thanks to knowing exactly which partition to access for the data.

Scan operations don't require you to know the primary key, but they are much slower and far less efficient than other operations. You can apply filter expressions to limit the data returned, but this happens after the data is read. A scan operation reads out 1MB of data from your table, then applies any filter expression. If that doesn't return your results, you will need to paginate and scan the next batch of data. In effect, you are reading your entire table to get the data you want. You should avoid scans whenever possible.

After reviewing these operations, you can see why we want to focus on GetItem and query operations, but both require us to know the partition key. Now you can see why key structure matters. There is one more topic to cover before we really focus on key structures: indexes.

Secondary Indexes

DynamoDB has a feature called secondary indexes. They are effectively copies of your table that you read from the same way, but organized differently. There are two types: global secondary indexes (GSI) and local secondary indexes (LSI). A local secondary index uses the same partition key as your base table, but you can configure a different sort key. A global secondary index can have a different partition key and sort key from your base table.

There are additional considerations with secondary indexes, but those are outside the scope of this discussion.

Key Structures

You've seen how important the partition key is for your data. A standard line you used to hear from AWS about DynamoDB performance was the importance of "well-structured queries". If you don't know your partition key, your query is not well-structured. GSIs let you define new partition keys and offer a lot of flexibility, but you still need to know the partition key for each GSI.

The importance of your partition key is why you need to understand your data and data access patterns before developing your DynamoDB table. If you will always know your user's username once they log in, then it may be a good partition key to use. If you have a multi-tenant application and will always know the tenant ID associated with a user, then that may be a good partition key.

Sort keys are where things get interesting. Using someone's last name as a sort key works for a basic example, but real applications are rarely that simple. Perhaps you are tracking details and audit records for different pieces of equipment. That's two different kinds of data about your equipment, in the same table. Your equipment ID could serve as your partition key, and you could use "details" as your sort key for the equipment information. For your audit items, you can use version numbers or timestamps as part of your sort key to make them sortable, something like audit_v1 or audit_v20251121T121314Z. This lets you keep track of your past audits while also being able to retrieve the most recent one by sorting values. You do need to be careful of possible issues when sorting numbers as strings. For example, audit_v10 will come after audit_v1 but before audit_v2.

With complex use cases, you may end up using a composite sort key that isn't a single value but instead multiple values concatenated. AWS's documentation uses the following as an example for a table listing geographical data.

[country]#[region]#[state]#[county]#[city]#[neighborhood]

This kind of structure lets you use begins_with, between, and greater than/less than operators to retrieve related groups of data. You can use your partition key and query for all data related to King County, Washington (where Amazon's HQ is) by querying for a sort key beginning with usa#northwest#wa#king# (USA for the country, Northwest for the region, WA meaning Washington state, and King meaning King County). Note that I included the # at the end of the value, which prevents us from picking up extra values due to partial matches.

By combining these ideas, you can have multiple formats for sort keys in the same table. I had one application with a single DynamoDB table containing 12-15 different item types. We had numerous organizations in the same table, with their organization ID as the partition key and sort keys such as organization, user#[user guid]#detail, user#[user guid]#permissions, and more.

The downside to these composite sort keys is that you have to build them yourself when you create your items, setting them as an attribute. AWS doesn't do it for you. Now imagine you're adding a new GSI and want a new composite sort key. Now you have to update every item in your table with the correct value for that sort key. It becomes very challenging; hence, you generally want an in-depth understanding of how your data will be accessed before defining the key structures for your DynamoDB tables.

Enter Multi-Attribute Key Schemas

This is where the AWS announcement comes in. Multi-attribute key schemas are a new feature of GSIs. Instead of needing to manually concatenate and backfill values, the DynamoDB service can do that for you. Instead of writing a process to iterate over your table and update every item with your new attribute for a new partition key and sort key in a new GSI, you tell AWS which attributes you want used in the keys and in what order, and AWS handles it. AWS doesn't actually add a new attribute, but instead lets you retrieve items based on multiple attributes.

AWS uses TOURNAMENT#WINTER2024#REGION#NA-EAST as an example composite key, representing scores from a game tournament. Instead, you specify up to four attributes for your partition key and up to four for your sort key. In this example, tournamentId and region are the partition key. When reading data from the GSI, instead of referencing an attribute you defined with a composite value, you specify the attributes as part of your query. This keeps your data looking cleaner and saves you a headache.

This means instead of defining an item like the code below, with composite keys as attributes.

const item = {
  matchId: 'match-001',
  tournamentId: 'WINTER2024',
  region: 'NA-EAST',
  round: 'SEMIFINALS',
  bracket: 'UPPER',
  player1Id: '101',
  // Synthetic keys needed for GSI
  GSI_PK: `TOURNAMENT#${tournamentId}#REGION#${region}`,
  GSI_SK: `${round}#${bracket}#${matchId}`,
};

You define the attributes on your GSI and use your item as-is, without extra concatenation.

const item = {
  matchId: 'match-001',
  tournamentId: 'WINTER2024',
  region: 'NA-EAST',
  round: 'SEMIFINALS',
  bracket: 'UPPER',
  player1Id: '101',
  matchDate: '2024-01-18',
};

You are limited to four attributes in the partition key and four attributes in the sort key. This applies only to global secondary indexes (GSIs). It is not available for your base table or local secondary indexes (LSIs).

If you want to learn more about what this means for key structures, AWS added a page to their documentation covering multi-attribute key patterns.

Wrapping Up

I'm excited about multi-attribute key schemas because they make DynamoDB easier to use and more flexible without requiring complex backfill work. I've always had to caution people about DynamoDB because of the importance of well-designed key structures. They're still important, but adding new GSIs with different key structures is easier now, and that's a significant improvement.

Is flat-rate CloudFront worth it?

Jason Butz — Wed, 19 Nov 2025 16:00:00 +0000

AWS just released flat-rate plans for CloudFront and related services, wrapping together WAF, Route53, CloudFront Function, logging, and more. Sounds tempting, especially with a free tier. You can read more about AWS's announcement on their blog. Is it really worth it?

AWS has announced four pricing tiers: Free, Pro ($15/month), Business ($200/month), and Premium ($1,000/month). Of course, each has different capabilities and limits included.

I had thought about creating a big table to compare what's available, but it quickly got too complicated. You can check out AWS's non-marketing comparison of features by pricing tiers in their documentation. I do want to call out a few things that are included and excluded from some of the lower tiers. I'm not going to focus as much on the Business and Premium tiers because, of course, they have the best features, and at those price points, you've already shown some willingness to get them. It's the lower tiers where you miss out on some valuable features that aren't expensive, and those are the tiers where small creators and companies are going to focus.

One interesting thing included in the free tier is WAF rules. In the free tier, you get 5; in the pro tier, you get 25. Those rules usually cost $1 per month. That's already making the pro price tag appealing. Include the WordPress/PHP/SQL DB security rules in the pro and above tiers, and that looks like a good deal. But what are you giving up?

On the free and pro tier plans, you don't have access to private VPC origins. Instead, your origins, the systems CloudFront is connecting to, must be in a public VPC or generally accessible to the internet. If you want access to your site or application via the CDN, you probably don't want other public paths available to the system. You can use security groups and Origin Access Control (OAC) to limit access to only CloudFront, but not having the system publicly addressable is better. Additionally, you'll probably also have to pay for public IPv4 addresses from AWS. It's not a massive cost, but it can add up over time.

You also can't use custom origin request or response header policies in the free or pro tier plans. The default request policies are fine for most cases, but the response header policies matter for security headers such as Content Security Policy (CSP) and Cross-Origin Resource Sharing (CORS). Most modern websites and applications should have CSP and CORS headers configured.

One CloudFront feature I was surprised to see available only in the premium tier is CloudFront origin groups, which provide origin failover to enable high availability. You configure multiple origins that can return the same content into a group with primary and secondary origins. When the primary origin returns specific errors, CloudFront retrieves the content from the secondary origin. I remember the us-east-1 outage in December 2021; my client experienced a major P1 incident, with most of their applications returning errors. The issue most users encountered was that S3 couldn't return the assets needed to load the web frontends, though other issues users didn't see also existed. CloudFront origin groups can fix that. I even wrote a blog post about addressing that for my site (Building a Resilient Static Website on AWS). It's not that complicated and is within reach of smaller businesses and creators who want to ensure there is at least a friendly error message for users visiting their site during an AWS outage.

AWS Hero Yan Cui wrote a post on LinkedIn discussing this new flat-rate pricing, including highlighting concerns with it. He pointed out that the marketing material references "serverless edge compute," but that is misleading. Only CloudFront functions are included in the flat-rate pricing plans; Lambda@Edge functions are not. That's a significant difference in capabilities. If I'm reading the documentation right, you can't even use Lambda@Edge functions with these flat-rate plans. He also calls out the language around not incurring overage charges, noting that, instead, performance may be reduced, such as throttling or requiring you to change your pricing structure. Yan extrapolates on that last point, plainly calling out the trade-off: If you're below your quota, you may be wasting money, but if you're above your quota, you risk throttling and/or losing your pricing structure.

What's more important to you? Stable pricing or availability and elasticity? One of the main selling points of the major cloud providers is availability and elasticity. If price were the only concern, you could probably host your site or application elsewhere much more cheaply. Traditional hosting and data center companies still exist, and you can still host applications on a server running in your home or business. Both will have a much more consistent pricing structure, but will lack the availability and elasticity of the cloud.

I understand AWS wanting to offer consistent pricing if that's what customers are asking for, but this whole thing stinks of Azure's tendency to lock features behind tiers across its services. I hope that's not the direction AWS is heading. I appreciate the true building-block approach that AWS provides.

If these flat-rate plans provide what you and your use case need, and the consistent pricing is a big selling point for you, then by all means use them. AWS capabilities are solid regardless of how you pay for them, but if you need flexibility and availability and can handle some bill fluctuation, stick with pay-as-you-go pricing for CloudFront.

Debugging ECR Private Pull Through Cache

Jason Butz — Wed, 12 Nov 2025 09:17:00 +0000

I completely missed that AWS added pull through cache rules for private ECR (Elastic Container Repository) repositories earlier this year. If you're an organization making use of publicly available container images this goes a long way towards helping you avoid throttling issues, accelerating deployments, and letting you have some confidence that the container versions you depend on will remain available.

One of my coworkers was having trouble with the pull through cache, his application that was running in ECS wasn't pulling the most recent container image for the latest tag like he expected. Even worse, it couldn't pull down images if he used a different tag. I don't know how he got the initial version of the image into ECR, but I wouldn't have expected his issues to start up after things seemed to be working.

🕐 TL;DR

Make sure your ECS task execution role has the ecr:BatchImportUpstreamImage and ecr:CreateRepository actions on repositories matching your pull through cache prefix. Once you have it working, refine the policy to the least access necessary.

When he tried different to use new tags in his task definition he got errors in the deployment logs like the one below:

service Sample-service was unable to place a task. Reason: CannotPullContainerError: pull image manifest has been retried 7 time(s): failed to resolve ref 012345678910.dkr.ecr.us-east-1.amazonaws.com/docker-hub/sample/sample:4.13.5: 012345678910.dkr.ecr.us-east-1.amazonaws.com/docker-hub/sample/sample:4.13.5: not found.

The image didn't exist in the ECR repository, so it makes sense the service wasn't able to find it, but the whole point of a pull through cache is that the image gets pulled. I decided I needed to test that the pull through cache could work at all.

I set up my local environment with the right AWS credentials and then used the View Push Commands button for the repository to get the command I needed to set up access to the ECR repository. That let me try pulling the image locally through the cache.

# Setup ECR Repository Access
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 012345678910.dkr.ecr.us-east-1.amazonaws.com
# Pull the container image
docker pull 012345678910.dkr.ecr.us-east-1.amazonaws.com/docker-hub/sample/sample:4.13.5

That worked, and the new images for that tag showed up in ECR. So the cache was working, but something was preventing it working within ECS. Sounds like permissions issues.

Since the container images are run as ECS tasks, I pulled up the ECS task IAM execution role. It has the permissions necessary to pull the container images, but does it have the permissions for the pull through cache? It did not!

There are two permissions potentially needed for pulling through images for the cache:

ecr:BatchImportUpstreamImage to actually pull down the image and update your private ECR repository
ecr:CreateRepository to create a new repository if you specify a brand new image

Since my coworker wasn't planning to pull down entirely new container images, he only needs the first permission. I updated the IAM role with the policy below and had things working!

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": "ecr:BatchImportUpstreamImage",
    "Resource": "arn:aws:ecr:us-east-1:012345678910:repository/docker-hub/*"
  }]
}

That's it! We had things working. We just needed to give the ECS task execution role the additional permissions needed for the repositories matching his pull through cache prefix. He is pulling multiple container images, so the wildcard isn't too excessive. It it were a single image the policy could be updated for that single repository.

We're still not sure why the once per 24 hour update alluded to in the AWS documentation isn't working, but that's a mystery for another day.

JavaScript Lambda Runtime Benchmarking

Jason Butz — Thu, 22 May 2025 12:30:00 +0000

Last month, I tried out using Bun as a custom JavaScript Lambda runtime and was surprised by the performance difference between Bun and AWS's managed Node.js runtime. The Node runtime had a shorter cold start time, and the invocations had a shorter duration. The cold start times weren't surprising given AWS's caching and performance optimizations, but the difference in the invocation durations was surprising.

Since then, I have been working on a better benchmark to compare performance between AWS's Node.js runtime and the custom Bun and Deno runtimes.

The Setup & Comparison

I originally generated a JWT as a simple test of the runtime's compute performance. For this new comparison, I continued to use hashing but instead generated 50 SHA3-512 hashes per invocation. I felt that generating a single hash was too small a test, and generating 50 hashes would result in a better comparison.

For Bun and Deno, I wrote the handler in TypeScript, taking advantage of the runtime's capabilities. For the Node.js handler, I used plain JavaScript. If the runtime had a hashing function built in, I used that, hoping for the best performance. For Bun, I used the CryptoHasher function. For Deno and Node.js, I used the Node.js crypto package and the createHash function. The source code for the handler functions is below.

I set up the Bun custom runtime using Lambda Layers, like I had previously. The Deno custom runtime is now container-based; it used Lambda Layers when I last used it. I used the Dockerfile that is available in Deno's documentation and only modified the filename for the handler.

//
// bun-handler.ts
//
const HASH_NUMBER = 50;

export default {
  async fetch(): Promise<Response> {
    const output = [];

    for (let i = 0; i < HASH_NUMBER; i++) {
      const hasher = new Bun.CryptoHasher('sha3-512');
      hasher.update(
        `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
      );
      output.push(hasher.digest('hex'));
    }

    return new Response(JSON.stringify(output));
  },
};

//
// deno-handler.ts
//
import { createHash } from 'node:crypto';

const HASH_NUMBER = 50;

Deno.serve(() => {
  const output = [];

  for (let i = 0; i < HASH_NUMBER; i++) {
    const hasher = createHash('sha3-512');
    hasher.update(
      `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
    );
    output.push(hasher.digest('hex'));
  }

  return new Response(JSON.stringify(output));
});

//
// node-handler.js
//
const { createHash } = require('node:crypto');

const HASH_NUMBER = 50;

export async function handler() {
  const output = [];

  for (let i = 0; i < HASH_NUMBER; i++) {
    const hasher = createHash('sha3-512');
    hasher.update(
      `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
    );
    output.push(hasher.digest('hex'));
  }

  return output;
}

I built an AWS CDK project to deploy the different Lambda functions, the infrastructure necessary to trigger the functions, and the dashboard for the aggregated results. I've created a GitHub repository with everything if you want to try out the experiment or see how I used the custom runtimes.

I used an event-based approach to invoke the Lambda functions and run the hashing code. Each Lambda was triggered based on messages being added to an SQS queue. I spaced out when messages would become available in the queue using the delay property to avoid having a surge of cold starts and parallel invocations. I wanted to be sure I got invocations that used an already running instance of the function. After all, having a readily available function and everything cached is often when you have your best performance.

Every couple of hours, a scheduled Lambda function added 30 messages to each SQS queue with a steadily increasing delay until the max delay of 5 minutes was reached.

I enabled JSON-formatted logs for the functions; I found I was able to parse out the metrics I needed easily in CloudWatch. I relied on the metrics reported by Lambda in the platform.report log entry for this analysis.

The Results

I used the duration and initialization duration measurements provided by the Lambda service for these results. Both numbers are in milliseconds. Remember that 1000 milliseconds (ms) is 1 second. The duration is how long it took the function to process the event. The initialization duration is how long it took the function's Init phase to complete. This happens when a new execution environment is being set up. Technically, this may not include the time required to download the source code, but that is difficult to measure. I'm simplifying things and considering the initialization duration and the cold start duration equivalent for this comparison.

I had 1326 invocations per runtime, totaling 3978 invocations overall. Each runtime experienced around 70 cold starts.

ℹ️ Percentiles are values below which a given percentage of all the values in the dataset exist. So if 15.190 is our value for the 50th percentile, then 50% of our values are below 15.190. You can learn more on Wikipedia and Amazon CloudWatch's documentation. In the tables below percentiles are written as "p10", "p25", etc. Where "p10" means "the 10th percentile", "p25" means "the 25th percentile", and so on.

Duration

Runtime	Invocations	Avg	p10	p25	p50	p75	p90
Bun	1326	50.513	2.610	6.855	15.190	37.531	68.230
Deno	1326	13.708	2.167	2.359	6.692	15.699	19.836
Node	1326	21.290	1.951	2.156	8.052	20.831	56.711

When it comes to the invocation durations, Deno had the best averages, followed by the Node.js runtime. On average, Bun took more than twice the time to generate the hashes.

Looking across the percentiles, Deno and Node were pretty close all the way out to the 75th percentile. Even at the 75th percentile, Deno was about 5ms faster, which you probably wouldn't notice. Looking at Bun, up until about the 50th percentile, it wasn't drastically slower. You're not going to notice 4ms, or even 10ms. Let's be honest, you're probably not going to notice a 50ms difference. To put that in perspective, a human blink takes between 100ms and 400ms. But, with AWS, you're billed for that time, and it can add up at the end of your billing cycle.

Initialization Duration

Runtime	Cold Starts	Avg	p10	p25	p50	p75	p90
Bun	70	547.651	500.075	512.706	527.048	546.935	603.223
Deno	72	267.474	184.607	191.996	203.221	218.661	297.237
Node	71	152.014	145.555	147.338	151.067	154.884	159.869

Looking at the initialization durations, the Node runtime was the fastest on average. I don't find this surprising, since I imagine AWS has put a lot of effort into minimizing cold start times. Bun was the slowest on average; I expect that is because it uses Lambda Layers, where Deno uses a container.

Across the percentiles, there is a very small gap in the values for the Node runtime. To me, this indicates AWS has ensured these initialization durations for their Node runtime are fairly consistent. The difference between the 10th percentile and the 90th percentile is about 14ms. That's astoundingly consistent. The difference for Deno is about 30ms, and for Bun is 56ms. The initialization duration for the Bun runtime is consistently more than half a second. For Deno, it's under a quarter of a second more than 75% of the time.

In Summary

AWS's Node.js runtime for Lambda is very performant, and I am sure AWS has put a lot of work into ensuring that is the case. It has relatively consistent initialization durations and can generally be quite fast.

Deno's container-based custom runtime is currently more performant than Bun's Lambda Layer-based approach, for both initialization and invocation. That said, Bun's approach is easier to work with and deploy. Deno's approach means you may end up building a container for every Lambda function, or you end up with larger Lambda functions where you use a fraction of the files included. I'm impressed by how performant the Node.js runtime is, when Bun and Deno are meant to be more performant JavaScript runtimes.

These runtimes all perform well, especially if your application rarely has cold starts. If you need to gain every last bit of performance out of each Lambda invocation, your choice may matter, but if you're building a fairly basic API, I think you could choose any of these runtimes. Focus on the features you need and the developer experience you want.

JavaScript Lambda Functions Using a Bun Custom Runtime

Jason Butz — Wed, 23 Apr 2025 12:30:00 +0000

I've previously tried out Lambda functions with a custom runtime using Deno, and it had great security and convenience benefits. But Deno isn't the only alternative to the Node.js runtime. Bun is a more recent entrant to the space, but it has an impressive number of features, including not requiring TypeScript to be transpiled, and it makes a lot of claims around speed. Bun also has everything for a custom Lambda runtime buried in its GitHub repository.

Custom Lambda Runtimes

Custom runtimes are distributed as Lambda Layers, and the OS-only Lambda runtimes are used as their base. The OS-only runtimes are Amazon Linux 2 and Amazon Linux 2023 and are the foundation on which the rest of the runtime builds. The custom runtimes have certain APIs and processes that they have to implement to work with the Lambda service and be able to respond to events.

I visualize and think of Lambda runtimes and layers like Docker or other OCI-compatible containers. I wouldn't be surprised if that's actually how things work, considering Lambda supports using container images, but I have nothing to prove that is how the Lambda service works.

Containers are made up of different layers. If you've ever pulled down a container image from Docker Hub, GitHub, or elsewhere, you probably saw multiple progress bars. Each of those was for a layer of that image. Each layer is generally dependent on the layer before it, and when loaded on top of each other, they make the complete container image. Having multiple layers and the dependencies between them means you don't have to rebuild the entire container image if you make a change. Instead, only the layer you changed and any layers that depend on it need to be rebuilt.

The Lambda OS-only runtime is effectively the base image for a container, which has the Lambda layers applied on top of it, with the source code you provide as the last layer applied.

This is an oversimplification of container images, and not something you need to know if you want to use a custom runtime, but I've found this visualization useful when troubleshooting complex Lambda situations.

Building & Deploying the Bun Layer

Bun's readme for the bun-lambda package provides the commands necessary to build and deploy the Lambda layer to your AWS account. You need Bun and the AWS CLI installed and your terminal configured with access to the AWS account where you want the layer deployed.

The commands default to building and deploying a layer for arm64 Lambdas and call the layer "bun". Since Lambda still defaults to the x86_64 architecture, I suggest a slightly different set of commands. The architecture for the layer does matter and determines which version of the bun runtime is downloaded, so I recommend deploying layers for both the x86_64 and arm64 architectures with a naming convention similar to the AWS Parameters and Secrets Lambda Extensions.

Instead of running the publish layer script as detailed in the README, run the commands below. I ran into issues with the publish script correctly detecting the configured AWS region. I suggest manually entering the AWS region (or regions) you want the layer published to.

bun run publish-layer --layer bun \
    --arch x64 \
    --region us-east-1 us-east-2
bun run publish-layer --layer bun-arm64 \
    --arch aarch64 \
    --region us-east-1 us-east-2

After running these commands, I had four Lambda layers in my AWS account, and the script output the ARNS for each of them:

arn:aws:lambda:us-east-1:000000000000:layer:bun:1
arn:aws:lambda:us-east-1:000000000000:layer:bun-arm64:1
arn:aws:lambda:us-east-2:000000000000:layer:bun:1
arn:aws:lambda:us-east-2:000000000000:layer:bun-arm64:1

Creating the Lambda Function

The process of creating the Lambda function is mostly the same as you're used to, but with two important differences:

You have to add the bun or bun-arm64 Lambda layer to the Lambda function, depending on which architecture you chose for your Lambda
You have to return a Response object from your function handler instead of a JavaScript object.

One of the quirks of the Bun runtime is the requirement to use the Response class as the return value from your handler functions unless you're working with websockets, which I'll get to in a moment.

Below is a basic function handler that returns a JSON string with a message:

export async function handler(request, server) {
    return new Response(JSON.stringify({message:"Hello from Lambda!"}));
}

The Bun Server Format

The Bun runtime has built-in functionality for a high-performance web server, including WebSocket support. You can leverage this format to build your Lambda function, which lets you easily test your code locally without setting up containers or additional tools.

Below is the code sample Bun provides for a Lambda that will respond to Amazon API Gateway requests:

export default {
  async fetch(request: Request): Promise<Response> {
    console.log(request.headers.get("x-amzn-function-arn"));
    // ...
    return new Response("Hello from Lambda!", {
      status: 200,
      headers: {
        "Content-Type": "text/plain",
      },
    });
  },
};

The Bun custom Lambda runtime does not support the routes functionality of the Bun HTTP server added in Bun v1.2.3. It only supports the catch-all fetch function and the websocket function.

I haven't dug into the WebSocket functionality; it seems like it is intended for use with API Gateway WebSocket APIs, but that isn't something I have had the occasion to use.

It may seem like the Bun custom runtime is focused on supporting integrations with API Gateway, but it does support other event sources. This is an example of what a basic handler might look like for SQS, SNS, S3, EventBridge, and other services:

export default {
  async fetch(request: Request): Promise<Response> {
    const event = await request.json();
    // ...
    return new Response();
  },
};

Testing Locally

If you use the Bun server format, you can run your handler function locally using Bun and your browser or an API testing tool.

> bun run handler.ts
Started development server: http://localhost:3000

An HTTP server gets started on localhost, and you can use whatever tools you want to make requests to it. If you've ever used the Lambda testing capabilities of the SAM CLI, this is similar, except the Docker container used by the SAM CLI makes things a little more similar to the AWS environment.

With the Bun server, it doesn't matter what kind of request you make, GET request, POST, or otherwise, it will execute the code. If you're really trying to mimic the AWS environment, you will need to put together the right HTTP body to make the request your function receives look like a request in AWS.

It's handy to test something locally, but it's important to remember this isn't a full testing harness and isn't mocking over any AWS services. You either need to create those or use a tool like LocalStack.

Is it Fast?

In a very unscientific test, I created a Bun Lambda function that creates a JWT using the function ARN and the current ISO 8601 timestamp as the payload. It's not much computational work, but the Lambda function does everything and doesn't rely on outside systems. I invoked the function several times, waited a while, and repeated the process. This helped ensure I had a couple of cold starts to get initialization numbers.

I had three cold starts in my dataset, and the initialization of the Lambda averaged 629.48ms. That initialization time is the extra time added by the cold start. Including the initialization time, the Lambdas averaged 75.66ms with 46 invocations. Removing that initialization time from consideration and focusing strictly on the handler's execution time, the average was 34.61ms.

I adjusted the code to work in a normal Node.js 22 Lambda and followed the same procedure. The average execution time was 1.99ms, and the cold starts averaged 134.65ms. I know the Lambda team has a lot of optimizations in place for the base runtimes, and I don't know how much of this difference is due to those optimizations or to differences in Node and Bun.

I may not be able to get super precise numbers, but I'm considering doing a more rigorous comparison between Node.js, Deno, and Bun Lambdas in the future.

Wrapping it Up

Regardless of the numbers, the Bun Lambda is still fast, and with an easier way to test it locally, the trade-off may be worthwhile. Bun has excellent Node.js compatibility and has a lot of features that are great for the developer experience. I've used it for some automation scripts and doing local data transformations, and I'd be open to using it in production applications at this point, even in a Lambda.

How Big Should Your Engineering Team Be?

Jason Butz — Thu, 17 Apr 2025 13:00:00 +0000

Have you ever been on a team where it seemed like it took forever to get anything done? You had constant status meetings and could never seem to get a handle on what was being done. It's not an unusual situation, and it can have a variety of causes. How big was the team? Was it six or seven people? Maybe more? That could have been your problem.

I was recently asked to propose a solution to reimagine an existing system. The system should use all the same data and database tables, but the rest of the application could be reimagined. The goal was to be able to deliver an MVP as soon as possible. I had to include the technical aspects and what the team of engineers should look like. The stakeholder wanted the MVP delivered in a month, maybe six weeks, and assured me that there were plenty of engineers available, so whatever it took to get the MVP delivered was fine. He told me he had 10 to 15 engineers he could assign to this easily if we could deliver the MVP in under 6 weeks.

I proposed a team of five engineers, and he was surprised at how few I wanted.

The Mythical Man-Month

The Mythical Man-Month: Essays on Software Engineering by Fred Brooks was published in 1975 and contains many ideas and lessons that still apply today. The idea that I reference most, and which was at play in my recommendation for a small team, was the mythical man-month.

There is an analogy or idiom I've heard that illustrates the mythical man-month concept well, even if it doesn't go into the reasoning from the book: "Nine women can't make a baby in a month." A pregnancy is about 9 months long, and the baby is growing and developing, which is not something you can divide among multiple people. The pregnancy will take roughly the same length of time, no matter what.

Unlike a pregnancy, developing software can be divided between multiple people, but can't be perfectly divided into discrete, self-contained tasks without dependencies. There will be dependencies between tasks and the people doing the work. People will have to communicate and coordinate their work.
Communication and task coordination are major limiting factors for development teams. When you have two people, there is only one line of communication between them. If you have a team of three, there are three lines of communication. If you have a team of 10 people, there are 45 lines of communication.

The formula for the lines of communication between people in a group is n(n-1)/2, where n is the number of people. If you really want to get into the math, this creates a polynomial growth curve.

The Work Matters

Regardless of the number of lines of communication, the kind of work you're doing matters when it comes to team size and scaling, and what can get done. When you can eliminate dependencies between people and tasks, you find opportunities to be more efficient.
If your large team owns multiple microservices, for example, dividing the team between the microservices for the current sprint helps reduce the number of lines of communication. The same idea applies if you have 100 engineers migrating 20 microservices from Java to Go, for example. If you assign five people to each microservice, you have drastically reduced the number of lines of communication by removing dependencies outside those groups of five. In both these examples, you effectively created smaller teams.

Suppose your team needs to create a lot of data models in your codebase, for example. In that case, you can often effectively split that across a large team if you properly plan for dependencies and relationships. The fewer dependencies and relationships, the easier the work will be with a large number of engineers.

That's the key: Reducing the dependencies between work can enable you to leverage a larger team effectively. It's not always a great idea, but it's an option if you have no choice.

Can I Speed Up My Project with More People?

Probably not. One of the key points in The Mythical Man-Month about the mythical man-month is what has been coined as Brooks' law: "Adding manpower to a late software project makes it later." Getting new people up to speed on the work will require coordination and communication, which will be affected by that growth curve from earlier.

If the work is well enough defined, there are few enough dependencies, and the new people need very little context, you might be able to add people and speed up the project. But that is a lot of "maybes" and "ifs."

Small Teams Are Fast

Several years ago, I led a team that fluctuated in size from four to seven engineers. When the team had seven engineers, it took a lot of mental effort to keep everything coordinated and moving forward; I felt like I was losing my mind. When we were down to four or five people, we had a few sprints where it was just me and one or two other engineers due to vacations and holidays. During a two-week sprint with four or five engineers, we completed 25 - 30 story points. When it was just me and two other engineers, we completed over 30 story points. Our last sprint that year was just me and one other engineer, and we completed just under 50 story points! This was the same kind of work we had been doing, but we reduced the necessary communication and coordination and produced amazing results! Sometimes, shrinking your teams is the answer.

Amazon and other companies are well known for using the "two pizza team" concept, the idea being that you should be able to feed the entire team with two large pizzas. According to this concept, Amazon says teams should be less than 10 people. I think teams need to be eight people or fewer, including leads, BAs, POs, or whatever. If someone is allocated 50% or more to the team, they count. For me, the sweet spot is around 4 to 5 engineers per team, including the lead.

Scaling Software Engineering is Different

There are so many industries and disciplines where adding resources enables scaling. That's even the case with most systems we deploy, but not with our engineering teams. Many people think of software development as solitary work, like being a machine on an assembly line, but that's not the case. Software development is knowledge work, and there may be similar concepts and approaches across different projects and products. Still, they aren't the same, and there needs to be a lot of coordination.

Keep your teams small, and communication and coordination are easier. Look for places to divide work and eliminate dependencies. Don't try to speed up a project by adding a bunch of developers to it; it won't work.

From PartyRock to Bedrock: AI-Powered Automation at Work

Jason Butz — Fri, 31 Jan 2025 17:00:00 +0000

There's been an explosion of tools claiming to make your job easier and touting GenAI, and I'm sure many of them can be very helpful. However, it can be hard to get support at many companies to integrate large new suites of tools into a process. It's often easier to add tools to your tasks. That's where the idea for this post came from.

Last year, I told my boss and co-workers about AWS's PartyRock tool. We'd been experimenting with tools like ChatGPT and HuggingChat and found some use cases, but we'd have to paste the prompt every time. It didn't make things easy to reuse and share. We had a lot of success with PartyRock, and now we're starting to use what was built with PartyRock to automate processes.

What is PartyRock?

PartyRock is a GenerativeAI-based tool that builds AI-powered applications using GenAI. The quickest way to get started is to write a prompt explaining what you want the application to do, and then PartyRock creates that application. There are some significant limitations to that. It won't turn your start-up idea into a working SaaS offering but can help with specific and limited tasks.

Here are some application examples from PartyRock's site:

Create a meeting summary and list of action items when given meeting notes.
Compare a job description and your resume and highlight gaps in your resume.
Get grammar and writing feedback using PartyRock like an editor.

A prompt for PartyRock will tend to be complex, but they have many examples to help get you started. Once you've generated your application, you can also fine-tune it by adjusting the prompts and other behaviors.

PartyRock is currently free to use, but if you're going to use it for your job, you should probably disable sharing data with AWS. You should also review PartyRock's privacy policy, terms of service, and other legal stuff, along with your company's AI-use policies, to make sure what you're doing isn't going to cause problems, especially if you put anything sensitive into PartyRock.

Go to "Backstage" in PartyRock's menu to find the option to disable sharing data with AWS.

What we did with PartyRock

We've tried out several different ideas with PartyRock, and I'm not going to go through all of them. Some didn't work out well, and some border on proprietary. One example I don't think anyone will mind my sharing is a tool to provide an initial review of RFPs.

RFPs, or requests for proposals, are a big part of public-sector work. When working with governments and public institutions, like public schools, not public companies, there are generally rules about ensuring adequate competition between vendors that wish to provide services to the organization. That generally means documents are created that highlight the work that needs to be done, the expectations and requirements for the work, how the grading of a proposal works, and all kinds of other information. These documents are called RFPs and are what vendors use to create a proposal to try and win the work. Sometimes, these RFPs are a handful of pages long. Sometimes, they are hundreds of pages long. Reviewing these documents to see if they match your services can be a long and tedious process, and the format and quality of RFP documents are inconsistent. You can't just scroll to the page that says "services requested." Sometimes, that information is called out clearly. Sometimes, special codes indicate categories of services requested, and sometimes, you have to guess what the author was trying to request.

Since these documents are generally required to be publicly accessible by law, there isn't too much concern about putting them into a GenAI tool, especially since we don't get RFPs for classified projects in the part of the company I work in. Only RFPs that are already accessible online.

We used PartyRock to inspect these documents and determine the services being requested, how they match up against the services we offer, estimated budgets and timelines, and all kinds of other information that can quickly help us determine whether we want to read the entire RFP document. It's a huge timesaver! With PartyRock, we drop the PDF document onto the page, and PartyRock extracts and inspects the text. If we get a PDF from scanned documents, it doesn't work, but those RFPs aren't very common. Typically, everything is written on a computer, so extracting text from the PDF isn't hard.

Next Step - Bedrock APIs

Having refined our prompts in PartyRock, we're now experimenting with integrating Amazon Bedrock into some of our tools and processes. We're a bunch of software engineers and use some open-source tools, so extending things with new functionality isn't a big step.

Bedrock is AWS's serverless way to use different foundational models, without spinning up a bunch of infrastructure. We could do this all with SageMaker and other tools, or we can build on top of Bedrock. Which is what PartyRock does.

We were a little concerned that working with documents and Bedrock was going to mean a bunch of effort by using Texttract. I was glad we were proven wrong. I was able to build a quick proof of concept using the Bedrock API in 10 - 15 minutes.

Using a very basic prompt and a small sample document I wrote myself, I was able to call Bedrock's API and get a response from Amazon's Nova Lite model.

import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import {
  BedrockRuntimeClient,
  ConverseCommand,
} from '@aws-sdk/client-bedrock-runtime';

const sampleDocument = readFileSync(join(__dirname, '..', 'sample.pdf'));

const client = new BedrockRuntimeClient({
  region: 'us-east-1',
});

client
  .send(
    new ConverseCommand({
      modelId: 'amazon.nova-lite-v1:0',
      messages: [
        {
          role: 'user',
          content: [
            {
              document: {
                name: 'rfp',
                format: 'pdf',
                source: {
                  bytes: sampleDocument,
                },
              },
            },
            {
              text: 'Review the RFP, and provide a summary of the services being requested',
            },
          ],
        },
      ],
    }),
  )
  .then((resp) => {
    console.log(JSON.stringify(resp, null, 2));
  });

Once I knew I could get the kind of output I needed, I started working on an API to allow us to easily call Bedrock on the format we needed.

API Gateway & Bedrock

I initially looked at directly integrating Amazon API Gateway with the Bedrock API through an AWS service integration. I'm a big supporter of avoiding unnecessary Lambda functions and having API Gateway directly call services whenever possible. AWS Hero Yan Cui recently posted on LinkedIn and uploaded a YouTube video discussing the same thing. Unfortunately, API Gateway doesn't currently support Bedrock. At least, not easily. Hopefully that'll be added soon.

💡 If you haven't used AWS service integrations with API Gateway, you should take a look at this tutorial from AWS.

So I created a Lambda function that handles the API call to Bedrock and had API Gateway invoke the Lambda function. Altogether, it took a couple of hours to build an API that is good enough to start adding GenAI into our processes.

The Road Ahead

We still need to use this API, but being able to use Bedrock so easily seems like a game changer. PartyRock was great when just a few of us were trying something out, but now that we have a use case worth being scaled out, it's time to use Bedrock directly and remove as much of the manual process as possible.

How to Enable Multi-Session Support in the AWS Console

Jason Butz — Thu, 16 Jan 2025 20:21:08 +0000

AWS has finally released support for users to sign in to multiple AWS accounts simultaneously, without extra browsers or using special browser features. Multi-session support has been a missing capability for a long time, and enabling it is a breeze.

Open the account menu in the console's upper-right corner and click the Turn on multi-session button. You'll get a warning telling you the console URL will be different; click on the Turn on multi-session button if that isn't a concern. The page will reload, and the console URL will have your account number in it .

Now, you click Add session to open a window to log into another AWS account. If you use the IAM Identity Center (AWS SSO ), or some other SSO mechanism, you can open another account through that as well. I tested this with IAM Identity Center, and each account I opened used the new multi-session URL for the console. Multi-session support is a per-browser setting and not a per-account setting. Clearing your cookies will turn off the feature.

Multi-session support only supports 5 simultaneous sessions; if you open a 6th, you get a screen asking which session you want to log out of.

Multi-session support has been a long-awaited feature, and it's great to see it in the console. It's a more convenient alternative to using Firefox Containers, Chrome and Edge Profiles, or private browsing windows. Everything works very well and provides a smooth experience. I hope the limit of 5 accounts is adjusted or removed, but maybe having more than five accounts open isn't common for most people. How many AWS accounts do you typically need simultaneous access to? Tell me in the comments.

Chaos Engineering with AWS FIS and Lambda

Jason Butz — Wed, 20 Nov 2024 20:30:00 +0000

Recently AWS's Fauly Injection Service (FIS) added support for AWS Lambda, maybe it's the other way around, but either way, they now work together. I'd never given FIS much focus; most organizations I work with aren't ready for or interested in chaos engineering. However, the more I looked into FIS, the more I realized I had misjudged what FIS was capable of and where I could use it.

What is Chaos Engineering?

"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production."
— Principles of Chao Engineering

I first heard about chaos engineering due to Netflix and their Chaos Monkey tool, which later evolved into the now retired Simian Army and has since been split out again. I've actually worked in an environment where Chao Monkey was running, and it does influence how you build applications, but once you lay your foundations and patterns, it's not too bad.

AWS primarily identifies FIS as a resiliency-focused tool, but it uses chaos engineering principles at its heart. You're impacting the performance of your system through an experiment to see how things behave to different failures.

Introduction to AWS FIS

AWS FIS enables you to run experiments on certain AWS resources to test how your system responds to different fault conditions. FIS has a limited selection of actions that can be performed against different targets, i.e., AWS resources. At the time of writing, FIS can target 18 different types of resources:

Aurora DB clusters
RDS DB instances
DynamoDB global tables
EBS volumes
EC2 Auto Scaling groups
EC2 instances
EC2 Spot Instances
ECS clusters
ECS tasks
EKS clusters
EKS node groups
EKS Kubernetes pods
S3 buckets
VPC subnets
Lambda functions
ElastiCache (Redis OSS) Replication Groups
IAM roles
transit gateways

In a FIS experiment, you combine actions and targets, generally running them for a specific duration. You can add additional parameters that help focus your experiment, such as stopping only 1% of EC2 instances.

During and after your experiment, you can inspect your logs and other metrics to see how the system performed. FIS offers a feature to generate a report that collects CloudWatch metrics and combines them with experiment details into a PDF report. I've looked at the example report, and I'm not sure it's worth the $5 cost to generate the report.

FIS and Lambda

With AWS's recent release, there are three actions available that can target Lambda functions:

Invocation delay (invocation-add-delay)
Invocation error (invocation-error)
Invocation HTTP integration response (invocation-http-integration-response)

Adding an invocation delay is what it sounds like. It's similar to a Lambda cold start, but the delay added by FIS is after any cold starts. In addition to simulating a cold start, you can create timeout events by configuring the added latency higher than the Lambda function's timeout.

The invocation error action allows you to mark function invocations as failed. That will be helpful for testing error handling and retry mechanisms. Interestingly, you can also decide if you want to allow the Lambda function to execute its handler. That could be useful if you want to test whether certain operations are actually idempotent, i.e., performing the same action multiple times has no effect after the first time the action was performed. It might also be a way to avoid interrupting known idempotent operations and allow them to process events despite the experiment.

The HTTP integration response action is intended to work with Application Load Balancers (ALBs), API Gateways, and VPC Lattice. You can select a content type and HTTP response code that are returned. You cannot set the response body, which I find very disappointing. You can also decide whether to allow the Lambda function to execute its handler with this action. Again, this could test idempotency or limit the disruption caused by the experiment.

How does FIS work with Lambda?

The Lambda functions targeted by your FIS experiments must have the FIS Lambda Layer added. This is central to how FIS can perform different actions. You also need an S3 bucket to store the experiment configuration.

When the FIS experiment is initializing, FIS uses the service role you define in the experiment template to write configuration files to the S3 bucket at a well-known path. The FIS Lambda Layer, using your Lambda function's execution role, checks that well-known path in the configured S3 bucket at a configured frequency. Those checks from your Lambda function to the S3 bucket happen regardless of any FIS experiments, so they are increasing the number of S3 API calls and, presumably, the billed duration for your Lambda functions.

The FIS Lambda Layer uses the AWS Lambda Runtime API proxy to intercept function invocations. This is before the invocation reaches the runtime, which makes the FIS layer runtime agnostic. As part of configuring your Lambda functions for FIS, you set the environment variable AWS_LAMBDA_EXEC_WRAPPER to /opt/aws-fis/bootstrap. This is what enables that Lambda Runtime API proxy. FIS uses some of the Lambda runtime environment modification capabilities to provide functionality. You don't need to worry too much about the details unless you are using additional extensions to the Lambda environment, in which case you'll need to set up a proxy chain.

Setting up a Lambda function for FIS

As mentioned earlier, you must have an Amazon S3 bucket to use with the FIS experiment. It must be in the same region as the experiment you plan to run. You must also update your Lambda execution role with a policy to allow access to the S3 bucket as specific prefixes. You will also need an IAM policy on the role associated with your FIS experiment that grants access to the bucket, allows FIS to inspect Lambda functions, and enables FIS to do tag-based lookups. Details on what these IAM policies should look like are in the FIS documentation.

Once you have all of that, you need to make a few minor modifications to your Lambda functions that will be involved in the experiment. First, you should add the FIS Lambda layer, details and ARNs for the layers in different regions are in the AWS documentation. Then you should add two environment variables to the functions. The first is AWS_FIS_CONFIGURATION_LOCATION with an S3 bucket ARN that points to the FisConfigs prefix in the S3 bucket you set up, for example arn:aws:s3:::my-config-distribution-bucket/FisConfigs/. This lets the FIS layer know where it should look for configuration details. The second is AWS_LAMBDA_EXEC_WRAPPER with /opt/aws-fis/bootstrap as the value. This sets up the Lambda Runtime API proxy mentioned earlier.

This configuration is only the most basic configuration; depending on your functions, additional considerations may exist. Some of these considerations are:

Short experiment action durations
Using SnapStart
Fast and infrequently invoked functions
Functions already using Lambda extensions
Functions using container runtimes

The AWS FIS documentation outlines what you need to know.

How about an example?

These details are great, but how about an example showing what you can do with FIS and Lambda?

Let's take the architecture sketched out above. Our Lambda function is invoked with messages from the SQS queue, but if the messages fail to process three times, they are sent to the DLQ and onto an EventBridge pipe, which leads to logic not relevant to this example. In this example, we follow AWS best practices and have our visibility timeout on the SQS queue set to six times the Lambda timeout. With a Lambda timeout of 10 seconds, our visibility timeout is 60 seconds.

For this example, our EventBridge pipe and the associated error-handling system are new and need testing. We've already tested our logic by sending messages directly to the DLQ, but now we need to test the entire thing. We've decided to use FIS to simulate the Lambda function taking too long to process messages. We've also decided to run this experiment in our development environment. We're lucky and can break the processing of these messages for a short period of time.

In our FIS experiment template, we'll configure an aws:lambda:invocation-add-delay action with a startup delay of 30,000 milliseconds (30 seconds). This will ensure our Lambda function invocations time out. Since we can break the processing of these messages, we can set our action to run 100% of the time. We need to do a little math to ensure we keep our experiment running long enough.

Once our experiment is running, our Lambda function will be invoked (and time out) three times for every message added to the queue. Between those invocations, the message will be delayed for at least the length of time of our visibility timeout. Because our Lambda function will be timing out, its duration doesn't matter. The SQS queue's visibility timeout is the important duration here.

60\ seconds \times 3\ delivery\ attempts = 180\ seconds

It will take at least 180 seconds before each message is added to the DLQ. Assuming we can cause messages to be added to the queue quickly, we should be able to get multiple messages to the DLQ within a 5-minute duration for this action. That gives us a little wiggle room. If getting messages into the queue takes longer, we will want a longer duration, for example, 10 minutes. We should get some information if the duration is at least 3 minutes. Less than 3 minutes or 3 minutes exactly, and we might not receive messages to our DLQ. There can be a delay of up to a minute before the Lambda function follows the actions outlined in our experiment, and the visibility timeout on the SQS is a minimum duration of time before the message is placed back in the queue. It does not mean it will be immediately reprocessed at that time.

We should be able to test this path in our application with all these configurations, setting up our targets for the experiment and making the changes needed to our Lambda function.

Final Thoughts

Digging into AWS FIS's Lambda support expanded my understanding of chaos engineering. It helped me see far more possibilities for where it can be used than I had ever thought about. Chaos engineering is much more than terminating instances; with carefully planned experiments, you can test exactly how your system responds to a variety of issues. I hope the FIS team will continue to expand support for additional functionality with Lambda and other AWS services.

Lit - Lighting Fast Web Components

Jason Butz — Fri, 11 Oct 2024 13:00:00 +0000

Ever since the first release of web components I've been interested in them. A way to build components that are isolated from each other without needing a bulky library or framework sounded amazing. Unfortunately, they tend to be hard to work with and browser support was slow to come.

Lit is a small library, built on top of web components, that makes it much easier to build interoperable web components. The team released Lit version 3 last year, and I just got around to trying it out. I'm impressed with what Lit is capable of. There are a few things missing that I'd want to see before building out a major application, like a router API, but it seems like a great fit if you need to build some web components to drop into an existing site. It might even be nice for a framework agnostic design system.

Let's take a look at a simple project. In this case I'll make a very simple to-do list application.

Start the Project

Lately, when it comes to starting a new frontend project Vite has been my go-to tool. They have starting templates that fit for most of the libraries I would use, and it "just works".

To start out we need to generate our project, I'm choosing the Lit TypeScript template:

npm create vite@latest my-lit-app -- --template lit-ts

That should give you a directory tree similar to what I have below:

my-lit-app
├── public
│   └── vite.svg
├── src
│   ├── my-element.ts
│   ├── index.css
│   ├── vite-env.d.ts
│   └── assets
│       └── lit.svg
├── .gitignore
├── index.html
├── package.json
└── tsconfig.json

From there you will need to install your NPM dependencies, then we can start the application to see what we have.

# Install NPM dependencies
npm install
# Start the application in development mode
npm run dev

If you open the URL provided, this is what your page should look like.

Building a Component

To keep things simple, let's replace the component the template provided with one of our own. I've replaced the contents of src/my-element.ts with what I have below. This is a very simple boilerplate that will still build and render successfully, serving as a foundation.

import { LitElement, css, html } from 'lit'
import { customElement, property } from 'lit/decorators.js'

@customElement('my-element')
export class MyElement extends LitElement {
  static styles = css``
  render() {
    return html`
      <div></div>
    `
  }
}

declare global {
  interface HTMLElementTagNameMap {
    'my-element': MyElement
  }
}

Expanding from here, I want to get my HTML elements onto the page. After all, that's what is going to let us see progress and start adding functionality. Inside the render method you'll notice a template literal, specifically a tagged template named html, is being used to return the HTML the component will render. I'll add a form with a text input field and a button, along with an empty unordered list element. This special tagged template is part of what Lit provides to make building web components easier. It's important to note, this isn't JSX. It's HTML. You don't have the same restrictions, like needing a single parent element.

// ...
render() {
  return html`
    <form>
      <input type="text" name="todoItem" />
      <button>Add Item</button>
    </form>
    <ul></ul>
  `;
}
// ...

Now, we need a place to put the items for our to-do list and a way to display them on the page. To do that we will use the @state directive. This directive lets us add a private or protected property that still triggers updates to the component. Our array is going to contain objects, so one thing we need to do is help Lit with determining if our value has changed. This is because a new object, even if it has the exact same properties, will not be considered equal when using the === operator. One easy way to make this comparison is to convert our values to JSON strings, and then compare them. That's what I have done below with the hasChanged function.

This code can be added at the top of our class near the style property.

// ...
@state({
  hasChanged: (value?, oldValue?) => {
    return JSON.stringify(value) !== JSON.stringify(oldValue);
  },
})
private todoItems: { id: number; complete: boolean; value: string }[] = [
  {id: 1, complete: false, value: 'Item 1'},
  {id: 2, complete: false, value: 'Item 2'},
];
// ...

Now that we have items for our list, we need to update our HTML to display them. Lit has something to help make sure we keep DOM updates to the minimum, it's called the repeat directive. We pass our array of items, a function that returns an item's id, and a function that renders an item to the directive and it returns the HTML for our list. When our array of items changes, it will help minimize the number of updates. This behavior is something you have to purposefully use, but it's there for the same reason the key attribute is in JSX/React. You may notice the odd ?checked=${...} notation below. That is how Lit handles boolean attributes.

render() {
  return html`
    <form>
      <input type="text" name="todoItem" />
      <button>Add Item</button>
    </form>
    <ul>
      ${repeat(
        this.todoItems,
        (item) => item.id,
        ({ id, complete, value }) =>
          html`<li>
            <input type="checkbox" ?checked=${complete} /> ${value}
          </li>`,
      )}
    </ul>
  `;
}

We can see our list now, but we can't add new items and we aren't saving the checked state of our items.

Making it Interactive

Let's start with adding new items to our to-do list. First, use the @query decorator to make it easy to access our text input element. We pass the decorator the CSS selector for the element we want to find. Then we are able to access the element through the variable the decorator is applied to. That will make it easy for us to get the input element's value. Next, we'll need to create a private handler method to update our to-do list with the new item. Similar to how you would do it with React, I am defining a new array and assigning it to the variable. To create a unique ID value I am getting the current unix epoch and adding a random number. The epoch number is probably good enough, but a little randomness isn't going to hurt. At the end, after adding the item to the list, I am clearing the value of the text input and setting focus to the input element.
You might notice the e.preventDefault(); line. I'm using the submit event here, and that line will prevent the default behavior when submitting a form. That is, to make an HTTP request to the page defined in the action property, or the current page if that is undefined. We don't need that behavior so do disabling it is the way to do. We could avoid it with a click event, but by using the submit event a user typing something in and pressing enter works exactly as you'd expect with no additional effort.
On the HTML form tag you can see a new attribute that attaches the submit event listener to our form. The @submit syntax is how you attach the listener to a given element. Next, we'll be attaching a click event for our checkboxes.

// ...
@query('input[name="todoItem"]')
private addItemInputEl!: HTMLInputElement;

private _handleAddItem(e: SubmitEvent) {
  e.preventDefault();
  this.todoItems = [
    ...this.todoItems,
    {
      id: Math.floor(Date.now() + Math.random() * 1000),
      complete: false,
      value: this.addItemInputEl.value,
    },
  ];
  this.addItemInputEl.value = '';
  this.addItemInputEl.focus();
}
// ...
render() {
 return html`
   <form @submit=${this._handleAddItem}>
   // ...
 `;
 }
// ...

We'll need to create a private handler method for the click event we'll be generating. We'll take advantage of event bubbling so that clicking on the checkbox and the to-do item itself both cause our checkbox to be checked. To help with identifying the correct item, we'll add a data attribute that contains our item's ID. Our handler needs to determine the item ID to search for and update. The currentTarget property of the event will contain a reference to our list item element, since that is where the event handler is attached. From there, it's a matter of creating a new array with the modified to-do item with the complete value set to the opposite of what it is currently. This ensures we can both check and uncheck an item by clicking on it.

private _handleItemClick(e: PointerEvent) {
  const el: HTMLInputElement = e.currentTarget as HTMLInputElement;
  this.todoItems = [
    ...this.todoItems.map((item) => {
      if (item.id.toString() === el.dataset.id) {
        return {
          ...item,
          complete: !item.complete,
        };
      }
      return item;
    }),
  ];
}

render() {
  return html`
    // ...
        html`<li data-id="${id}" @click=${this._handleItemClick}>
            <input type="checkbox" ?checked=${complete} /> ${value}
          </li>`,
    )}
    // ...
  `;
}

Our to-do list is working now, but it's not obvious you can click on an item to toggle the checkbox. With just a little CSS we can change that. Update the string inside the css tagged template and then your mouse cursor will change to a pointer anytime you are hovering over one of the to-do items.

static styles = css`
  ul li {
    cursor: pointer;
  }
`;

One of the nice things about web components and Lit, is this CSS is scoped to our component. So it won't affect anything else on our page. With a few more CSS changes we can have this to-do list looking even better.

Below is the full component. With some additional CSS changes to the index.css file we can have the application looking even better. Our to-do list disappears if you refresh the page, but I'm not going to go into what it would take to save and restore that from localStorage, that is an exercise I'll leave up to you.

Lit is a fast way to develop web components. This example results in a JS bundle that is about 22kB, which could be smaller but also isn't horrible. Lit has good documentation, and makes it easy to get started. There are still some features I'd like to see, but a lot of them are already in the work. What do you think? Do you like Lit? Do you want me to create the same application in vanilla web components to see what the difference in difficulty and bundle size really is? Let me know on social media.

import { LitElement, css, html } from 'lit';
import { customElement, query, state } from 'lit/decorators.js';
import { repeat } from 'lit/directives/repeat.js';

@customElement('my-element')
export class MyElement extends LitElement {
  static styles = css`
    ul {
      list-style: none;
      padding-left: 0;
    }
    ul li {
      cursor: pointer;
    }
    ul li input[type='checkbox'] {
      margin-right: 0.5em;
    }
  `;

  @state({
    hasChanged: (value?, oldValue?) => {
      return JSON.stringify(value) !== JSON.stringify(oldValue);
    },
  })
  private todoItems: { id: number; complete: boolean; value: string }[] = [
    { id: 1, complete: false, value: 'Item 1' },
    { id: 2, complete: false, value: 'Item 2' },
  ];

  @query('input[name="todoItem"]')
  private addItemInputEl!: HTMLInputElement;

  private _handleAddItem(e: SubmitEvent) {
    e.preventDefault();
    this.todoItems = [
      ...this.todoItems,
      {
        id: Math.floor(Date.now() + Math.random() * 1000),
        complete: false,
        value: this.addItemInputEl.value,
      },
    ];
    this.addItemInputEl.value = '';
    this.addItemInputEl.focus();
  }

  private _handleItemClick(e: PointerEvent) {
    const el: HTMLInputElement = e.currentTarget as HTMLInputElement;
    this.todoItems = [
      ...this.todoItems.map((item) => {
        if (item.id.toString() === el.dataset.id) {
          return {
            ...item,
            complete: !item.complete,
          };
        }
        return item;
      }),
    ];
  }

  render() {
    return html`
    <form @submit=${this._handleAddItem}>
      <input type="text" name="todoItem" />
      <button>Add Item</button>
    </form>
    <ul>
      ${repeat(
      this.todoItems,
      (item) => item.id,
      ({ id, complete, value }) =>
        html`<li data-id="${id}" @click=${this._handleItemClick}>
            <input type="checkbox" ?checked=${complete} /> ${value}
          </li>`,
    )}
    </ul>
  `;
  }
}

declare global {
  interface HTMLElementTagNameMap {
    'my-element': MyElement
  }
}

AWS Certified AI Practitioner Beta Exam Reaction

Jason Butz — Sat, 21 Sep 2024 19:26:00 +0000

The other day, I took and passed the new AWS Certified AI Practitioner beta exam. I was surprised at how difficult and technical it was for a foundational exam. To be completely clear, AI/ML is not my focus, and I have limited experience with it. I've used GenAI and other ML models but with no tuning. I know about many concepts like knowledge bases, agents, and embeddings, and I have some understanding of how they work. I know about some prompt engineering techniques but haven't studied them intensely. Machine learning wasn't a focus during my undergraduate studies, but I did learn a little about machine learning and how some of it works. Generative AI wasn't even a concept back then. I passed the exam with a score of 704; I needed a 700 to pass.

I should start by thanking AWS and the training and certification team. Through messages I received as an AWS Community Builder, I was able to apply for a voucher to take the beta exam for free. I think the goal was to get people with the correct skill set for the exam to take it and ensure the team behind the exam gets good data to move the exam from a beta version to a final version. I qualified for and received one of those vouchers, so I took the exam at no financial cost to myself.

I did not study for this exam. Part of that was because I believed I had the knowledge for it, considering it was a foundational exam, and I exceeded the target audience based on the intended candidate and the candidate role examples on the AWS website . I may have overestimated my knowledge because I barely passed, or the exam may not target the desired audience as well as it could. Maybe a little of both.

The AWS website says the intended candidates are "Individuals who are familiar with, but do not necessarily build, solutions using AI/ML technologies on AWS." They list "business analyst, IT support, marketing professional, product or project manager, line-of-business or IT manager, sales professional" as example candidate roles. I would not expect candidates in those roles to pass this exam without intensive study.

I can't tell you the questions that cause me to think that. I must comply with AWS's certification agreement , so I can't discuss the exam's questions. But I can talk about what is in the exam guide on AWS's page for the certification . I should also mention that the exam has 15 unscored questions, so it is entirely possible that the questions I thought were too difficult for the audience were those unscored questions. The beta exam also has more questions than the standard exam because AWS is evaluating whether the questions are appropriate [ Source ].

To add more credence to why what I say matters, I'm a part of the AWS Certification SME program and have supported the development of the AWS Certified Developer Associate exam. That adds some additional complexity to sharing my perspective. I know more about how the exams are developed, but I am restricted in what I can share by NDAs. It means I know a little more about what goes on behind the scenes.

Recommended AWS Knowledge

Familiarity with the core AWS services (for example, Amazon EC2, Amazon S3, AWS Lambda, and Amazon SageMaker) and AWS core services use cases

Familiarity with the AWS shared responsibility model for security and compliance in the AWS Cloud

Familiarity with AWS Identity and Access Management (IAM) for securing and controlling access to AWS resources

Familiarity with the AWS global infrastructure, including the concepts of AWS Regions, Availability Zones, and edge locations

Familiarity with AWS service pricing models

The level of AWS knowledge for this exam is similar to that of the Cloud Practitioner exam, based on the exam guide. From my experience with the exam, some questions went beyond "familiarity" with these services and concepts. But, the recommended AWS knowledge section of the exam guide isn't where the focus of questions is drawn from. That's the domains and the task statements under them. I'll go through those next.

Domain 1: Fundamentals of AI and ML

This domain seems like some of the most important information someone with this certification should have, but it only comprises 20% of the scored content. Domains 2 and 3 have a higher percentage of the exam at 24% and 28%, respectively.

This domain has three task statements:

Task Statement 1.1: Explain basic AI concepts and terminologies.

Task Statement 1.2: Identify practical use cases for AI.

Task Statement 1.3: Describe the ML development lifecycle.

I have no issues with Task Statement 1.1; it seems ideally suited to this exam. Task Statement 1.2 is also generally well-suited, though I wouldn't expect marketing professionals or line-of-business managers to know much about the capabilities of AWS-managed AI/ML services.

Task Statement 1.3 and its objectives stretch what I expect the target candidates for this exam to know. I can accept that knowing about the components of an ML pipeline (objective 1.3.1), understanding the sources of ML models (objective 1.3.2), and understanding fundamental concepts of ML operations (MLOps) (objective 1.3.5) would be a benefit for the target candidates that need to know about AI. I expect it's something many of these target roles would need to study specifically for this exam. I don't think that model performance metrics for evaluating ML models (objective 1.3.6) are appropriate. I would expect data engineers and scientists to be doing that and guide what is going to be the best option. Understanding why performance metrics are important is appropriate, but not specific ones, and which ones should be applied.

Domain 2: Fundamentals of Generative AI

Given the hype around GenAI, this domain also seems like a good fit for the exam and has three task statements:

Task Statement 2.1: Explain the basic concepts of generative AI.

Task Statement 2.2: Understand the capabilities and limitations of generative AI for solving business problems.

Task Statement 2.3: Describe AWS infrastructure and technologies for building generative AI applications.

Like in Domain 1, Task Statement 2.1 makes sense. The foundation model lifecycle, the last objective for that Task Statement, is something the target candidate roles will probably need to study for the exam. Task Statement 2.2 is perfect for the exam. Anyone with AI or ML credentials should understand the limitations of GenAI. Task Statement 2.3 is also reasonable, for the most part. Some of the target candidate roles will need to specifically study the AWS services and features for developing GenAI applications (objective 2.3.1).

Domain 3: Applications of Foundation Models

Again, because of the hype focused on GenAI, questions around foundational models (FMs) and prompt engineering seem reasonable. There are four task statements focused on the FMs domain:

Task Statement 3.1: Describe design considerations for applications that use foundation models.

Task Statement 3.2: Choose effective prompt engineering techniques.

Task Statement 3.3: Describe the training and fine-tuning process for foundation models.

Task Statement 3.4: Describe methods to evaluate foundation model performance.

The task statements seem reasonable at the top level, but once you start looking at their specific objectives, they go beyond what I expect most of the target audience to know or need to know. Why would a marketing professional need to "Identify relevant metrics to assess foundation model performance" such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), or BERTScore? Identifying technical metrics is something engineers should take the lead on. I would expect the stakeholders, such as a marketing professional, to validate that choice by working with the engineers and ensuring business objectives are met. Coincidentally "determine whether a foundation model effectively meets business objectives" is one of the objectives under task statement 3.4.

Domain 4: Guidelines for Responsible AI

Using AI responsibly is something most people should agree with, and I have been seeing more and more focus on this. It doesn't surprise me to see it on the exam. There are only two task statements for this domain, but task statement 4.1 has a long list of objectives.

Task Statement 4.1: Explain the development of AI systems that are responsible.

Task Statement 4.2: Recognize the importance of transparent and explainable models.

The objectives for this domain are, for the most part, great. They are well suited to ensuring people are thinking about how to use AI responsibly. The only areas that go outside of what I expect the target audience to know are when it comes to tools and AWS features for ensuring responsible AI use.

Domain 5: Security, Compliance, and Governance for AI Solutions

A security, compliance, and governance domain seems a good idea. Still, it's a topic that can quickly get too detailed for someone who is only supposed to be familiar with AI/ML and does not need an in-depth understanding. There are only two task statements outlined for this domain:

Task Statement 5.1: Explain methods to secure AI systems.

Task Statement 5.2: Recognize governance and compliance regulations for AI systems.

Task Statement 5.1 immediately concerns me. Security is essential with AI and ML, but should someone at a foundational level of understanding be able to explain the methods used to secure an AI system? I don't think that's reasonable. Why should a sales or marketing professional need to know how to secure an AI system? Objective 5.1.3 says, "Describe best practices for secure data engineering (for example, assessing data quality, implementing privacy-enhancing technologies, data access control, data integrity)." This is a foundational AI/ML certification, not the associate-level data engineer certification .

Task Statement 5.2, at the top level, doesn't concern me, but the objectives seem to exceed the task statement. The task statement says to " recognize governance and compliance regulations" [emphasis mine], but the objectives ask candidates to identify regulatory standards and describe processes to follow governance protocols. I could see an argument being made that identifying standards counts as recognizing regulations, but describing processes to follow a regulation does not. I can recognize and identify, at least in most cases, when PCI DSS is relevant, but I can't describe all the processes necessary to be compliant. I expect to have an expert involved to ensure we follow the correct process. The same applies to any significant regulations, including AI-related regulations.

What have others said?

I started a chat with other AWS Community Builders, and there seems to be a consensus that the Certified AI Practitioner exam is much more difficult than we expected, especially considering it's a foundational exam. One Community Builder said the questions were foundational in length but required associate levels of knowledge. Many of us wouldn't expect non-technical people to pass the exam, which is a problem considering they are part of the target candidate groups. These conversations within the AWS Community Builder community have led to discussions with AWS. I spoke with two people from the AWS Certifications group and shared direct feedback, including most of what I have written here.

What I am Saying

The AWS Certified AI Practitioner exam is still in beta and is not in its final form. The questions on the exam will change. Hopefully, the questions will be better suited to the target candidates. I hope the exam guide and its task statements and objectives will change, too. The guide I am looking at is version 1.4 AIF-C01. Creating these exams is hard, and I don't want to discount the incredible effort that went into developing this exam. It's just that there is still work to do. The AWS team is listening to feedback, and I am sure they will adjust this exam.

I would not recommend this exam to non-technical people right now. Let the final version come out first. If you are going to take the beta exam, study the topics listed in the exam guide. I expect this exam to get easier, so if you have only a foundational level of experience with ML, you should count it as an achievement to pass the beta exam.

Good luck.

IAM Policy Conditions & SQS Queue Access

Jason Butz — Thu, 22 Feb 2024 14:10:00 +0000

The other week, I was helping a client work through an interesting challenge. The entire problem resulted from decisions made when the company was designing how they would build and connect their AWS accounts. They had decided there would be two kinds of AWS accounts: one with access to their internal network and one without. The accounts with access to the internal network cannot have any ingress from the public internet; all ingress must be through the corporate network. From a security perspective, I see why they made that choice. They decided to trust AWS and allow ingress to their network from these accounts but disallow ingress from the internet to reduce the chances of an adversary gaining access to their network. For the team I am working with, this presents a problem. They need to integrate with an outside SaaS service and receive webhook requests, but the Kubernetes cluster they are using is part of a platform in one of these accounts without internet ingress. That leaves them unable to receive the webhook events. I proposed we use one of these accounts with public internet ingress to build the integration and then take advantage of the cluster using EKS and supporting IAM Roles to gain access to resources in a different account and integrate with SQS to receive the webhook events.

I created an API Gateway with a single endpoint to receive the webhook request from the SaaS application. I built a Lambda function to validate the webhook request, perform limited validation, and then publish the event to an SNS topic. I connected that Lambda to the API Gateway with a Lambda integration. I didn't use a Lambda Authorizer with the API Gateway because validating the webhook required access to the HTTP request's body, which isn't available to Lambda Authorizers. The SNS topic filters messages and ensures only the right messages reach the SQS queues. Overall, it's a simple integration, but it's very effective.

Identity & Access Management

I have multiple AWS certifications and good working knowledge of IAM policies, but it sometimes feels like magic. The Policy Evaluation Logic page in the AWS IAM documentation is a must-have reference when doing more complex work. One of my weak points with IAM policies is the conditions. I haven't used them enough to be entirely confident, so I go through trial and error. In this case, the SQS resource policy and working with cross-account access meant I spent a lot of time in the IAM documentation.

The easy part of the queues' resource policy was allowing the SNS topic to send messages. I found an example in AWS's documentation and barely had to think about the conditions. I knew it should "just work." The policy below lets the SNS service send messages to the specified SQS queue when the source is the listed SNS topic.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-2:000000000000:MyQueue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-2:000000000000:MyTopic"
        }
      }
    }
  ]
}

To allow the pods in the other account access to receive and delete messages, I figured I could do something similar, so I copied and modified the policy statement. I've got an example of what I came up with below. For those of you who know your IAM policies, you might already see where I messed up.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-2:000000000000:MyQueue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-2:000000000000:MyTopic"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage"],
      "Resource": "arn:aws:sqs:us-east-2:000000000000:MyQueue",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": [
            "arn:aws:iam::999999999999:role/MySemiPredictableRoleName*"
          ]
        }
      }
    }
  ]
}
// NOTE: This policy does not work as expected; do not use it

This policy didn't work out; the application running on the pods was logging IAM errors about not having access due to the queue's resource policy. I began working through the policy evaluation logic for IAM, focusing on resource policies. I tried using the ArnEquals condition type. I tried using the IAM role session instead of the IAM role. I tried changing where the wildcard (*) was. I couldn't get it to work. So, after about 30 minutes of frustration and redeploying the policy, I started digging into the IAM condition keys.

Looking into the AWS docs for aws:SourceArn I was about to find my answer. The aws:SourceArn condition key is for when making service-to-service requests where the principal is an AWS service. Instead, I needed to use aws:PrincipalArn. It was all spelled out for me in the documentation. I've included an excerpt from the AWS documentation below.

Use this key to compare the Amazon Resource Name (ARN) of the resource making a service-to-service request with the ARN that you specify in the policy, but only when the request is made by an AWS service principal. When the source's ARN includes the account ID, it is not necessary to use aws:SourceAccount with aws:SourceArn.

This key does not work with the ARN of the principal making the request. Instead, use aws:PrincipalArn.

As it turned out, aws:SourceArn is a condition key used to avoid the confused deputy problem during actions between services. The documentation says to only use the condition key in resource-based policies where the Principal is an AWS service principal. The confused deputy problem is a security issue where an entity without access to a resource coerces a more privileged entity to access the resource. The IAM documentation has a great page discussing the confused deputy problem.

Now that I knew the correct condition key, the application could access the SQS queues. Below is what my resource policy looked like.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-2:000000000000:MyQueue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-2:000000000000:MyTopic"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage"],
      "Resource": "arn:aws:sqs:us-east-2:000000000000:MyQueue",
      "Condition": {
        "ArnLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::999999999999:role/MySemiPredictableRoleName*"
          ]
        }
      }
    }
  ]
}

This policy did precisely what I needed, and the application could pull messages from the SQS queues as soon as I deployed my update. It's such a small change, and there isn't anything profound about it. I usually write posts to share something interesting I found, how I do things, or to show something off. This post does show something off, but it's one of my mistakes. It reminds me to double-check that I'm using IAM conditions right, and hopefully, next time, I won't make this same mistake.