Forem: Matthew Wilson

Eliminating additional bandwidth charges for multi-zone sites on Vercel

Matthew Wilson — Fri, 22 Dec 2023 16:03:10 +0000

When deploying applications to any kind of cloud platform, be that AWS, Azure, GCP or even a higher level abstraction like Vercel, monitoring cost is an important aspect of the development lifecycle. We should be continuously trying to reduce our monthly spend whenever possible. In this post, we show how we eliminated additional bandwidth charges on a recent multi-zone Vercel client project.

A Multi Zone Project

The client in question has a large Vercel-based application that currently has 10 projects deployed with a routing structure like this:

Project structure showing parent project rewriting to several child projects.

Next.js/Vercel refer to this kind of setup as a “Multi-Zone” Project.

With multi zones support, you can merge both these apps into a single one allowing your customers to browse it using a single URL, but you can develop and deploy both apps independently.

This enabled us to deploy multiple versions of the same project with different configuration but with the feel of a single site to the customers.

Multi Zones == Double Billing

There is one issue, however, that both Vercel and Next.js documentation neglects to mention and that is, you will be "double billed" for bandwidth usage in a multi zone project. This is because the rewrites work by fetching and serving the content from the child site. Therefore, both the parent site and the rewritten target would consume bandwidth by serving the same content.

This is fine if you can safely live within the 1TB limit on bandwidth but go above this limit and things start to get expensive at $40 per 100GB (at the time of writing this article).

We decided to see if we could provide the same functionality provided by the rewrites on Vercel but not pay a premium for the privilege.

Rewriting the rewrites

Our site used Cloudflare to handle all our DNS needs. Cloudfare has a number of rewrite options but as we needed to make a decision based on a) the URL path and b) rewriting to a specific domain, the only option (as a non enterprise customer) was to use Cloudflare workers (via worker routes).

The functionality is simple - we configure a worker route for the path that should rewrite to a child project, fetch the content from the child site in our Cloudflare worker and then return it.

Configuring a Cloudflare worker route, with a wildcard based path and which worker to execute

Worker routes have wildcard support as documented here, so it’s easy to add a catch-all route for a specific path.

The code in the url rewrite worker is simple too:

export default {
  async fetch(originalRequest, env, ctx) {
    const originalUrl = new URL(originalRequest.url);
    const newHost = "child-project.co.uk";
    const newUrl = originalRequest.url.replace(originalUrl.host, newHost);
    const request = new Request(newUrl, {
      body: originalRequest.body,
      headers: originalRequest.headers,
      method: originalRequest.method,
      redirect: originalRequest.redirect
    });
    return await fetch(request);
  },
};

This snippet simply fetches the content from child-project.co.uk but maintains the path, headers, body etc.

Let's talk cost

With Cloudflare workers you don’t pay any egress fees. So this simple hack will immediately save you the $40 per 100GB bandwidth cost. For our client, that represented a total saving of $400 per month based on an average monthly usage of 2TB. (Remember the first 1TB is free and Vercel will round the 1.9TB bandwidth usage up to 2TB!)

Total bandwidth usage for the project

Looking across 7 days of requests on Cloudflare for our busiest child project, you can see that our little workers are handling plenty of traffic. This particular site serves double the traffic of all our other child projects combined.

Graph showing how many times the worker has been invoked over a 7 day period.

The costs for these requests are incredibly cheap as well. When we plugged these numbers into the Unofficial Cloudflare Workers' pricing calculator, the weekly cost for this site was $5.53. The cost for all sites works out at about $50 per month.

Conclusion

In conclusion, by using Cloudflare workers and configuring worker routes, we were able to replicate the rewrite functionality provided by Vercel without incurring any egress fees. This resulted in significant cost savings for our customer, with an estimated reduction of 87.5% in monthly costs. With Cloudflare Workers free egress fees, our customer saved $40 per 100GB of bandwidth. This cost-effective approach allowed us to efficiently handle high traffic volumes while keeping cost low.

A beginner's guide to AWS Best Practices

Matthew Wilson — Wed, 13 Dec 2023 16:00:57 +0000

I was recently asked to run a session on “AWS Best Practices” with some engineers who were starting their careers at Instil.

The topic itself is huge and would be impossible to distil down to just a few points, so this post covers a few of the obvious choices and how you can find out more information.

Security

This should be the obvious first choice. When you are developing cloud applications, security should be at the very forefront of your mind. Security is a never-ending game of cat and mouse, it cannot be a checkbox exercise that is marked as “done”, as your application evolves so must your security posture. Here are a few key points to consider from the outset.

Understand the Shared Responsibility Model

Thankfully with AWS, part of the security burden has already been taken care of for you. This is explained with what AWS call the “Shared Responsibility Model”.

This is usually summarised down to:

AWS responsibility is the “Security of the Cloud”, the customer responsibility is “Security in the Cloud”

This is an important concept to understand. You can minimise your responsibility by choosing Serverless or managed services, but there will always be some level of responsibility as a customer of AWS.

Secure your Root Account

This might not be something you ever have to do, normally account creation will be taken care of by your organisation (and would hopefully be automated) but I thought it was worthwhile sharing as you might be setting up a personal AWS account for some training, experimentation or even for certifications.

When setting up a brand new AWS account you will start by creating a “root user”. It can be tempting to then start using this user for deploying and managing your application, however the root user has “the keys to the kingdom” - if it is compromised in any way there is the potential to lose everything in that account, rack up an enormous AWS bill and potentially expose further information about your organisation.

While securing your root account isn't a magic bullet, you should consider it part of the “Swiss cheese” approach to security:

The Swiss cheese model of accident causation illustrates that, although many layers of defense lie between hazards and accidents, there are flaws in each layer that, if aligned, can allow the accident to occur. In this diagram, three hazard vectors are stopped by the defences, but one passes through where the "holes" are lined up.

AWS Community Builder Andres Moreno has written a great post that covers the recommended settings for your root account, which they have summarised as:

Do not create access keys for the root user. Create an IAM user for yourself with administrative permissions.
Never share the root user credentials.
Use a strong password. (Use a password manager if possible)
Enable multi-factor authentication.

I would also add that these steps should be followed for any subsequent account creations and enforced as well.

In a multi account setup, AWS automatically creates a strong password and does not give you details of that password by default specifically to discourage use of the root user. You can then use SCPs to deny all actions for the root user in a member account.

Principle of least privilege (POLP)

While securing your account is an important first step in setting up your infrastructure, the Principle of least privilege is an important aspect to consider during the entire software development lifecycle.

The idea here is that a principal should only be able to perform the specific actions and access the specific resources that is required for it to function correctly.

More info can be found in the Well Architected framework - https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_permissions_least_privileges.html#

Practically speaking, Infrastructure as Code (IAC) frameworks like CDK (Cloud Development Kit) make this very easy to implement. In fact it’s considered a best practice of CDK itself to use their built in grant functions. For example:

This single line adds a policy to the Lambda function's role (which is also created for you). That role and it's policies are more than a dozen lines of CloudFormation that you don't have to write. The AWS CDK grants only the minimal permissions required for the function to read from the bucket.

Handling Secrets

It is highly likely that you will have to handle some kind of secret when developing your application. To quote Corey Quinn:

“Let’s further assume that you’re not a dangerous lunatic who hardcodes those secrets into your application code.”

AWS gives two sensible options - Systems Manager Parameter Store or AWS Secrets Manager. There are pros and cons to both options, but understand that Parameter Store is just as secure as Secrets Manager, provided you use the SecureString\ parameter type. More info here.

In addition to choosing your secret service (pun intended) here’s a few more pointers:

Use your IAC framework to create and maintain secrets for you.
Ensure that secrets are not shared across environments.
Never store secrets in plaintext on developer machines.
Never include secrets in log messages. (You can configure Amazon Macie to automatically detect credentials)
Rotate your secrets where possible, and again let the services handle this for you.

Encryption

When handling sensitive data it’s important to protect it. One technique you can use is encryption, both while the data is in transit and at rest. It may surprise you that for many services like S3 and DynamoDB, encryption at rest is disabled by default.

Thankfully using tools like cdc-nag you can help enforce some best practices around this and enable encryption from the beginning.

Cost

It’s becoming increasingly apparent that cloud developers need to understand the cost of the changes they are making. AWS provide a number of tools to help you analyse the cost of your application when it is up and running. However, cost should also be factored into the design stage of the development process as well.

Cost has become such an important factor in architecting cloud applications that it featured heavily in Amazon CTO Werner Vogels’ keynote at re:Invent 2023. In it he shared The Frugal Architect which contains some "Simple laws for building cost-aware, sustainable, and modern architectures."

When we talk about cost, yes we should consider the cost of the service itself, but we should also consider the Total Cost of Ownership as well.

Familiarise yourself and learn how to read the pricing tables of various services on AWS.
Create budget alerts to avoid unexpected bills.
Include cost optimisation as part of your design stage.
Get comfortable with using Cost Explorer - when costs spike it's important to understand how to drill down into costs.

Serverless First

Serverless has grown to mean many things and can even mean something different depending on whose cloud you are paying for. But for our team and for the purpose of this article, serverless is a way of building and running applications without having to manage the underlying infrastructure on AWS.

Of course there are still servers running in a data centre somewhere but those servers are not our responsibility. With Serverless we as developers can focus on our core product instead of worrying about managing and operating servers or runtimes.

By choosing “Serverless” or “Managed Services” as much as possible we allow AWS to do the “undifferentiated heavy lifting” for us. These are things that are important for the product to function but do not differentiate it from your competitors.

The benefits of Serverless are clear:

No infrastructure provisioning or maintenance
Automatic scaling
Pay for what you use
Highly available and secure

What we don't want to be is dogmatic, it’s very tempting to be “Serverless Only”, however in certain circumstances this can lead to increased complexity (for no reason) and increased cost (especially at a certain scale).

Serverless first means to treat it as the first step, use it to make a start and iterate quickly but don't be sad if it’s not a good fit for certain customers or workloads.

Furthermore it’s important to treat any decision you make as a “two way door”, architecture decisions are never final, if something doesn't work or starts to cost too much, change it!

Master the basics

It’s entirely possible that you land on a Serverless project and avoid having to do some of the undifferentiated heavy lifting we have already eluded to. But this doesn't mean you should be ignorant to it. Networking concepts and core AWS services should be understood, even if they are not part of the “Serverless” offering.

AWS Certifications are a great way to be exposed to these kinds of topics. Check out my colleague Tom’s post for more info) but as a start I cannot recommend the free “Tech Fundamentals” course by Adrian Cantrill enough. It is not AWS specific but covers fundamental knowledge that will help you design, develop and debug applications in the cloud.

Automate everything

We’ve mentioned Infrastructure as code a number of times already. Using IaC frameworks ensures that we have consistency and repeatability in resource provisioning and management. While you might not be dealing with physical servers, I find the Snowflake Server concept a really nice way of reminding myself why we spend time automating deployments.

Snowflakes are unique:

good for a ski resort, bad for a data center.

We’ve got quite a bit of experience using CDK across a number of projects, so I would recommend it as a great place to start. There are a number of posts on our blog which help cover some fundamental aspects:

CDK Lessons learned - https://instil.co/blog/cdk-lessons-learned/
Improving the CDK development cycle - https://instil.co/blog/improving-cdk-development-cycle/

An added benefit of CDK is that there are plenty of ready made constructs and patterns for you to re-use on your projects.

Well Architected Framework

The Well Architected Framework could be considered “the mother of all AWS best practice guides”. It breaks down what a well architected application looks like using 6 pillars:

Operational Excellence
Security
Reliability
Performance Efficiency
Cost Optimisation
Sustainability

While I hope you have found this post useful, I would recommend you use it as a start, and then branch into the Well Architected Pillars for further reading and then apply what you have learnt to your projects.

There is also a Well Architected Tool that can help you assess your application against these pillars. It’s important to note that this is not a “checklist” and you should not aim for a “perfect score”. Instead it should be used as a guide to help you identify areas of improvement.

Testing Serverless Applications on AWS

Matthew Wilson — Thu, 02 Nov 2023 19:28:17 +0000

Introduction

The topic of Serverless testing is a hot one at the moment - there are many different approaches and opinions on how best to do it. In this post, I'm going to share some advice on how we tackled this problem, what the benefits are of our approach and how things could be improved.

The project in question is Stroll Insurance, a fully Serverless application running on AWS. In previous posts, we have covered some of the general lessons learnt from this project but in this post we are going to focus on testing.

For context; the web application is built with React and TypeScript which makes calls to an AppSync API that makes use of the Lambda and DynamoDB datasources. We use Step Functions to orchestrate the flow of events for complex processing like purchasing and renewing policies, and we use S3 and SQS to process document workloads.

The Testing 'Triangle'

When the project started, it relied heavily on unit testing. This isn't necessarily a bad thing but we needed a better balance between getting features delivered and maintaining quality. Our testing ~pyramid~ triangle looked like this:

Essentially, we had an abundance of unit tests and very few E2E tests. This worked really well for the initial stages of the project but as the product and AWS footprint grew in complexity, we could see that a number of critical parts of the application had no test coverage. Specifically, we had no tests for:

Direct service integrations used by AppSync resolvers + StepFunctions
Event driven flows like document processing

These underpin critical parts of the application. If they stop working, people will unable to purchase insurance!

One problem that we kept experiencing was unit tests would continue to pass after a change to a lambda function but subsequent deployments would fail. Typically, this was because the developer had forgotten to update the permissions in CDK. As a result, we created a rule that everyone had to deploy and test their changes locally first, in their own AWS sandbox, before merging. This worked but it was an additional step that could easily be forgotten, especially when under pressure or time constraints.

Balancing the Triangle

So, we agreed that it was time to address the elephant in the room. Where are the integration tests?

Our motivation was simple:

There is a lack of confidence when we deploy to production, meaning we perform a lot of manual checks before we deploy and sometimes these don't catch everything.

This increases our lead time and reduces our deployment frequency. We would like to invert this.

The benefits for our client were clear:

Features go live quicker, reducing time to market but while still maintaing quality
- Gives them a competitive edge
- Reduces feedback loop, enabling iteration on ideas over a shorter period of time
Critical aspects of the application are tested
- Issues can be diagnosed quicker
- Complex bugs can be reproduced
- Ensures / Reduces no loss of business

Integration testing can mean a lot of different things to different teams, so our definition was this:

An integration test in this project is defined as one that validates integrations with AWS services (e.g. DynamoDB, S3, SQS, etc) but not third parties. These should be mocked out instead.

Breaking down the problem

We decided to start small by first figuring out how to test a few critical paths that we had caused us issues in the past. We made a list of how we “trigger” a workload:

S3 → SQS → Lambda
DynamoDB Stream → SNS → Lambda
SQS → Lambda
StepFunction → Lambda

The pattern that emerged was that we have an event that flows through a messaging queue, primarily SQS and SNS. There are a number of comments we can make about this:

There's no real business logic to test until a Lambda function or a State Machine is executed but we still want to test that everything is hooked up correctly.
We have most control over the Lambda functions, it will be easier to control the test setup in there.
We want to be able to put a function or a State Machine into “test mode” so that it will know when to make mocked calls to third parties.
We want to keep track of test data that is created so we can clean it up afterwards.

Setting the Test Context

One of the most critical parts of the application is how we process insurance policy documents. This has enough complexity to be able to develop a good pattern for writing our tests so that other engineers could build upon it in the future. This was the first integration test we were going to write.

The flow is like this:

File is uploaded to S3 bucket
This event is placed onto an SQS queue with a Lambda trigger
The Lambda function reads the PDF metadata and determines who the document belongs to.
It fetches some data from a third party API relating to the policy and updates a DynamoDB table.
File is moved to another bucket for further processing.

We wanted to assert that:

The file no longer exists in the source bucket
The DynamoDB table was updated with the correct data
The file exists in the destination bucket

This would be an incredibly valuable test. Not only does it verify that the workload is behaving correctly, it also verifies that the deployed infrastructure is working properly and that it has the correct permissions.

For this to work, we needed to make the Lambda Function aware that it was running as part of a test so that it would use a mocked response instead. The solution that we came up with was to attach some additional metadata to the object when it was uploaded at the start of test case - an is-test flag:

If the S3 object is moved to another bucket as part of its processing then we also copy its metadata. The metadata is never lost even in more complex or much larger end-to-end workflows.

The Middy Touch

Adding the is-test flag to our object metadata gave us our way of passing some kind of test context into our workload. The next step was to make the Lambda Function capable of discovering the context and then using that to control how it behaves under test. For this we used Middy.

If you're not familiar, Middy is a middleware framework specifically designed for Lambda functions. Essentially it allows you to wrap your handler code up so that you can do some before and after processing. I'm not going to do a Middy deep dive here but the documentation is great if you haven't used it before.

We were already using Middy for various different things so it was a great place to do some checks before we execute our handler. The logic is simple:

In the before phase of the middleware, check for the is-test flag in the object's metadata and if true, set a global test context so that the handler is aware it's running as part of a test.
In the after phase (which is triggered after the handler is finished), clear the context to avoid any issues for subsequent invocations of the warmed up function:

export const S3SqsEventIntegrationTestHandler = (logger: Logger): middy.MiddlewareObj => {

  // this happens before our handler is invoked.
  const before: middy.MiddlewareFn = async (request: middy.Request<SQSEvent>): Promise<void> => {
    const objectMetadata = await getObjectMetadata(request.event);
    const isIntegrationTest = objectMetadata.some(metadata => metadata["is-test"] === "true");

    setTestContext({isIntegrationTest});
  };

  // this happens after the handler is invoked.
  const after: middy.MiddlewareFn = (): void => {
    setTestContext({isIntegrationTest: false});
  };

  return {
    before,
    after,
    onError
  };
};

Here's the test context code. It follows a simple TypeScript pattern to make the context read-only:

export interface TestContext {
  isIntegrationTest: boolean
}

const _testContext: TestContext = {
  isIntegrationTest: false
};

export const testContext: Readonly<TestContext> = _testContext;

export const setTestContext = (updatedContext: TestContext): void => {
  _testContext.isIntegrationTest = updatedContext.isIntegrationTest;
};

I think this is the hardest part about solving the Serverless testing “problem”. I believe the correct way to do this is in a real AWS environment, not a local simulator and just making that deployed code aware that it is running as part of a test is the trickiest part. Once you have some kind of pattern for that, the rest is straightforward enough.

We then built upon this pattern for each of our various triggers, building up a set of middleware handlers for each trigger type.

For our S3 middleware we pass the is-test flag in an object's metadata, but for SQS and SNS we pass the flag using message attributes.

A note on Step Functions

By far the most annoying trigger to deal with was Lambda Functions invoked by a State Machine task.

There is no easy way of passing metadata around each of the states in a State Machine - a global state would be really helpful (but would probably be overused and abused by people). The only thing that is globally accessible by each state is the Context Object.

Our workaround was to use a specific naming convention when the State Machine is executed, with the execution name included in the Context Object and therefore available to every state in the State Machine.

For State Machines that are executed by a Lambda Function, we can use our testContext to prefix all State Machine executions with "IntegrationTest-". This is obviously a bit of a hack, but it does make it easy to spot integration test runs from the execution history of the State Machine.

We then make sure that the execution name is passed into each Lambda Task and that our middleware is able to read the execution name from the event. (Note that $$ provides access the Context Object).

Another difficult thing to test with Step Functions is error scenarios. These will often be configured with retry and backoff functionality which can make tests too slow to execute. Thankfully, there is a way around this which my colleague, Tom Bailey, has covered in a great post. I would recommend giving that a read.

Mocking third party APIs

We're now at a point where a Lambda Function is being invoked as part of our workload under test. That function is also aware that its running as part of a test. The next thing we want to do is determine how we can mock the calls to our third party APIs.

There are a few options here:

Wiremock: You could host something like wiremock in the AWS account and call the mocked API rather than the real one. I've used wiremock quite a bit and it works really well, but can be difficult to maintain as your application grows. Plus it's another thing that you have to deploy and maintain.
API Gateway: Either spin up your own custom API for this or use the built in mock integrations.
DynamoDB: This is our current choice. We have a mocked HTTP client that instead of making an HTTP call, queries a DynamoDB table for a mocked response, which has been seeded before the test has run.

Using DynamoDB gave us the flexibility we needed to control what happens for a given API call without having to deploy a bunch of additional infrastructure.

Asserting that something has happened

Now it's time to determine if our test has actually passed or failed. A typical test would be structured like this:

it("should successfully move documents to the correct place", async () => {
    const seededPolicyData = await seedPolicyData();

    await whenDocumentIsUploadedToBucket();

    await thenDocumentWasDeletedFromBucket();
    await thenDocumentWasMovedToTheCorrectLocation();
  });

With our assertions making use of the aws-testing-library:

async function thenDocumentWasMovedToTheCorrectLocation(): Promise<void> {
  await expect({
    region,
    bucket: bucketName,
  }).toHaveObject(expectedKey);
}

The aws-testing-library gives you a set of really useful assertions with built in delays and retries. For example:

Checking an item exists in DynamoDB:

await expect({
  region: 'us-east-1',
  table: 'dynamo-db-table',
}).toHaveItem({
    partitionKey: 'itemId',
});

Checking an object exists in an S3 bucket:

await expect({
  region: 'us-east-1',
  bucket: 's3-bucket',
}).toHaveObject(
  'object-key'
);

Checking if a State Machine is in a given state:

await expect({
  region: 'us-east-1',
  stateMachineArn: 'stateMachineArn',
}).toBeAtState('ExpectedState');

It's important to note that because you're testing in a live, distributed system, you will have to allow for cold-starts and other non-deterministic delays when running your tests. It certainly took us a while to get the right balance between retries and timeouts. While at times it has been flakey, the benefits of having these tests far outweigh the occasional test failure.

Running the tests

There are two places where these tests get executed, developer machines and on CI.

Each developer on our team has their own AWS account, they are regularly deploying a full version of the application and running these integration tests against it.

What I really like to do is get into a test driven development flow where I will write the integration test first and make my code changes, which will be hot swapped using CDK and then run my integration test until it turns green. This would be pretty painful if I was waiting on a full stack to deploy each time, but hot swap works well at reducing the deployment time.

On CI we run these tests against a development environment after a deployment has finished.

It could be better

There are a number of things that we would like to improve upon in this approach

Temporary Environments - We would love to run these tests against temporary environments when a Pull Request is opened.
Test data cleanup - Sometimes tests are flaky and don't clean up after themselves properly. We have toyed with the idea of setting a TTL on DynamoDB records when data is created as part of a test.
Run against production - We don't run these against production yet, but that is the goal.
Open source the middleware - I think more people could make use of the middleware than just us, but we haven't got round to open sourcing it yet.
AWS is trying to make it better - Serverless testing is a hot topic at the moment, AWS have responded with some great resources you can find here - https://github.com/aws-samples/serverless-test-samples

Summary

While there are still some rough edges to our approach, the integration tests really helped with the issues we have already outlined and can be nicely summarised with three of the four key DORA metrics:

Deployment Frequency - The team's confidence increased when performing deployments, which increased their frequency.
Lead Time for Changes - Less need for manual testing reduced the time it takes for a commit to make it to production.
Change Failure Rate - Permissions errors no longer happen in production and bugs are caught sooner in the process. The percentage of deployments causing a failure in production reduced.

Zero to Serverless Car Insurance - Part 3

Matthew Wilson — Tue, 18 Apr 2023 16:06:54 +0000

In Part 1 and 2 of this series we've shared some of the technical challenges we faced when building a Serverless car insurance platform.

In this post we are going to take a look at some ways in which you can help guide your team on their transition to Serverless.

In the 1970s & 80s, the expression “nobody ever got fired for buying IBM” was coined to illustrate IBMs utter dominance of the IT industry and how executives, who were playing it safe, kept buying IBM.
Rather than look at the competition, which could potentially save them time and money in the long run, they opted for the perceived safe choice.

I have seen a similar trend in regards to Serverless, many teams have reluctance when it comes to adopting a Serverless-First approach, despite the benefits being clear and proven:

No infrastructure provisioning or maintenance
Automatic scaling
Pay for what you use
Highly available and secure

There are potentially many reasons for this lack of adoption, but what I want to focus on is fear.

Fear when it comes to the developer experience, fear when it comes to modelling databases, fear when it comes to cost and scaling.

When I started on the Stroll project I was comfortable with Lambda functions, messaging queues but honestly I was afraid of DynamoDB. When you work with DynamoDB you are told that “you have to know all your access patterns up front?”. As an engineer you know that it's so easy for requirements to change and needing to know your access patterns seemed like a huge risk.

I wanted to see if this fear was just something I experienced, or had it impacted other teams as well. So I asked a few of my fellow Serverless AWS Community Builders a question:

When you first started your Serverless journey what were you, your team or your company afraid of, and how did you overcome it?

And I got some great answers:

We were afraid that getting up to speed with the Serverless paradigm would take ages and that we'd end up investing a lot over a long time before we could reap any benefits.
We overcame this by breaking things down and achieving small wins.
Getting the first Lambda in production only took a couple of days and felt great!
From there, we kept on adding the next small building block, learning what we needed to know as we went.

On the company that I was working at that time the main fears/blockers were:
Questioning if Serverless would provide value to our systems and customers.
The misconceptions: Serverless is more expensive than X, Serverless is harder to test than X or Serverless is more complex than X.
Education, not only in Serverless but in distributed systems in general.

This was late 2016/early 2017:
Vendor lock-in - got over it quickly though
That Serverless wouldn't gain market share - that we'd end up with an architecture with no community backing which we'd struggle hiring for

If I circle back to my own fears around DynamoDB, the solution to that problem was education. I educated myself on DynamoDB (Alex DeBrie's book is a great starting point), eliminated the fears that I had and have come to the realisation that planning, discovring and knowing your access patterns upfront is actually a good thing that leads to a better architecture overall.

One of the things that is key to our success is that we had a confident Serverless team. This confidence wasn't something we started out with, we had to eliminate the fears and concerns that the team had along the way.

One of the best things about our jobs is that we start with an empty git repository and turn it into a business for someone, that experience never gets old. What we have learnt is that you can do this better and faster with Serverless.

Encouraging the Serverless mindset

One of the first things we had to learn as a team was this notion of a Serverless mindset.

And if you’re not really in the Serverless world, drinking the Serverless Kool-aid as one of my colleagues would say, I'm sure that “Serverless mindset” isn't even a term you would be familiar with.
I think the origin of this comes from a blog article written by Ben Kehoe entitled “Serverless is a State of Mind”
And really what it boils down to is that if you want to succeed with Serverless technology you need to think differently about how and why you build things, focus on delivering business value more than the underlying technology you are using.

Encouraging this Serverless mindset is key to building confidence.

If you really want to encourage this mindset, it can't start at an individual level, it has to come from an organisational level.

At Instil, this idea of adopting a Serverless mindset, or going Serverless-first as some may say started from the top. We have an engineering strategy which lays out a number of things that we want to focus on from an engineering perspective and in the “cloud” section this was the first point:

In the last few years, Serverless has exploded and dramatically changed how applications are built - no longer do we need to worry about capacity planning, patching servers or wasting money on under utilised resources, we now have (virtually) infinite computing power at our disposal ready to scale up massively in a fraction of a second. We want to embrace Serverless but this requires a cultural change in how we approach building software. We all understand that code is a liability and in the Serverless world, our focus shifts to connecting events and services through configuration while reducing the amount of code we write. Right now, the Functionless (aka Serverfull) movement is also gaining popularity driven in part by AWS enabling application developers to ditch Lambda functions in favour of direct integrations between managed services.

That’s a confidence booster right there, if leadership are encouraging a Serverless first approach, that should help teams take that first step.
But its one thing saying that you want to take this approach, but a very different thing putting it into action.

The Serverless mindset was definitely something that our team struggled with, at the very beginning we were treating Lambda Functions as just another way to get some compute in the cloud.

As developers we focused too much on what Serverless meant for us, and I think when you take that narrow minded approach you can come up short on some good reasons for choosing Serverless:

Is it easier to deploy? Perhaps…
Is it easier to test? Not if you try to do it locally.
Is it easier to code? Only if you educate yourself first.

Nothing really stands out from that list.

In fact I know that early on on the project there were people that had their doubts if Serverless was a good choice.
The team needed to change their mindset and instead of solely focusing on the technology and how that made their lives as developers easier, instead look at the bigger picture, what does this mean for Instil and what does it mean for our customer.

One of the first steps on this journey is enlightening developers that writing less code is better.

Enlightening developers that writing less code is better

One of the key steps to instilling confidence in our Serverless team was to learn to write the code that really matters. In Part 2 we shared how our team came to this realisation.

And when we think about how we encourage the Serverless mindset in our teams, we need to have some key people to drive that forward. The lead engineers and architects of the team need to have the confidence that Serverless is the right choice. You can't expect the rest of the team to succeed if the people making the decisions have their doubts.
For me I certainly had my doubts, but that confidence came from educating myself. When I first joined instil Garth, our Director of Training shared a quote:

Being a software engineer is just agreeing to do homework for the rest of your life.

So if you are a lead engineer and you want to encourage the Serverless mindset in your team, you first have to educate yourself and build up your own confidence.

At Instil we strive for engineering excellence and that kind of culture attracts people that love to write code, but how do you convince a team of die hard programmers to write less code?

As developers were told that “code is a liability”. But let's be honest with ourselves, do we really believe that?, or do we say to ourselves, their code is a liability, but mine? mine is perfect!

We wanted to encourage the engineers to really own the problems they were trying to solve and to do that we needed to create a safe space for them to experiment, propose new ideas, get feedback from their peers and then be given the opportunity to actually implement the changes.

Now I need to make a disclaimer, I’m a huge Apple fanboy, and secretly my dream job is to be an independent iOS developer living off my App Store profits. If you’re familiar with iOS development you'll know that the main languages you can use is Swift.

For new Swift language features there is a process called Swift Evolution, it’s quite a big template with a few headers that enables people to propose new features. Now I’m not saying that Apple invented this, I know that Kotlin has a similar process and I’m sure other languages do too, but remember Apple Fanboy, so history and facts are not important. But this is where I drew inspiration from.

So we created a process called Stroll Evolution, a colleague summarised it like this:

Stroll Evolution, in this sense, includes a proposed solution to achieve some goal (motivation), with information about the proposed design, what effects there would be, as well as alternatives considered. It’s always good when you can answer “but why didn’t you do it some other way?” At least even for yourself in the future.

And with a simple template with the following headers:

Introduction
Motivation
Proposed solution
Detailed design
Effect on Web App
Effect on Mobile App
Alternatives Considered
Decision

We created a way for engineers to own a problem and help find a solution. This process helped us to:

Do our first DynamoDB data model, where we followed the steps laid out in Alex DeBrie’s DynamoDB book, came up with our access patterns and primary key design to help enable them.
Plan out what our first step function would look like and why we wanted to use step functions in the first place.
Decide how we process insurance policy documents and what the best AWS messaging service was for solving that problem.

So not only did it help us solve project specific problems, it gave team members a way to educate themselves on the Serverless offerings of AWS. Let them see for themselves, that they can either spend weeks building a thing, or just connect services together and focus on actually delivering business value instead.

I’m not saying that this exact process is going to work for your team, I’m just saying that there needs to be a way for engineers to own the problems they are trying to solve.

On our team, anyone can come up with a Stroll Evolution, from Senior engineers to Apprentice software engineers.
And with Serverless the engineers need to own more than they might do in other architectures. A Serverless engineer can't just write some code and chuck it over the wall for someone else to deploy, they need to understand what that looks like and own the end to end solution. And in my opinion that's a good thing.

But if we want engineers to own the end to end solution, they also need to take ownership of deploying it to production.

Enabling production deployments with confidence

There’s one final point I want to make around building confidence in your Serverless teams and that’s around production deployments, specifically around verifying that things are working as expected.

We wanted everyone on the team to have the confidence to kick off a production deployment. But for a period of time this wasn't the case, deployments were left for senior engineers to kick off, this impacted our lead time when delivering new features and reduced our deployment frequency.

No one actually came out and said “only this group of people can manage deployments”, it was just something that happened naturally because people were worried about breaking something. The reason behind this was there was a gap in our testing approach.

Some of you may be familiar with the testing triangle. Where you might have E2E at the top, integration in the middle and unit at the bottom.

On Stroll however we were missing the bit in the middle, we had plenty of unit tests. In fact I would say we had too many unit tests. Also, because we encouraged developers to use things like direct service integrations and step functions, unit testing became less useful.

We had some e2e tests that used the UI to drive the test but there was no integration tests.

The reason being that with Serverless, integration testing is hard.

You have two choices, try and run everything locally by simulating AWS on your developer machine. Or deploy to AWS and test in the cloud.

But this was a gap in our verification process that we needed to plug. So (using a Stroll Evolution) we put together a plan to write some integration tests and from that plan we came up with a definition of what integration testing meant for the project:

An integration test (on this project) is defined as one that tests integrations with AWS services, e.g. DynamoDB, S3, SQS. But does not test integrations with third parties, these should be mocked out instead.

The aim of these tests were to firstly verify that the workload is functioning correctly but also that you have the correct IAM permissions and that the necessary resources have deployed correctly.

If you’re deploying a workload to AWS it is really important for everyone involved in the project to be confident that it is running correctly. We have found that writing tests for each step of the process has been immensely helpful in giving us this confidence and as a result production deployments are thankfully not something that are left to an individual but are instead the responsibility of the entire team.

In a our next post a colleague will be sharing how we ensured our Step Function State Machines were executing correctly.

Zero to Serverless Car Insurance - Part 2

Matthew Wilson — Tue, 11 Apr 2023 10:27:23 +0000

Welcome to Part 2 of our series on building from Zero to Serverless Car Insurance. In Part 1, we introduced the platform and discussed how we built an end-to-end solution using Serverless technology on AWS.

In this post, we'll be focusing on some key improvements we've made to the platform, particularly focusing on how writing less code is a good thing!

It was time to break down the Lambdalith! This had a number of key benefits for the project:

We no longer had to worry about maintaining our own Apollo server and could instead hand that responsibility over to AWS. Security patches and updating for new features are no longer a concern for us.
We could create smaller domain based services instead enabling developers to make changes with less fear of breaking other parts of the application.
It becomes easier to test.
It reduces the overall size of the Lambda function package, which reduces the duration of cold starts.

Before we dive deeper into this change we need to get one thing straight:

Serverless is not just about Lambda functions!

You could build an entire serverless application on AWS with no containers or functions at all. Our decision to adopt AppSync has enabled us to make use of more of the Serverless offerings of AWS, which leads to the next exciting phase of our project.

Going Lambda-less!

There I was, working from home during a global pandemic. We were building the customer portal for Stroll, enabling customers to login and see their policy information, download documents and submit claims.

The task was simple, execute a query to get policy data from DynamoDB. I am sure most of us have built some CRUD functionality similar to this.

Not to sound too over dramatic, but little did I know that my life was about to change forever.

To understand this life changing moment we first need to understand how AppSync works.
I am not going to take too much time discussing GraphQL concepts in this post, that information is freely available online. But for context:

GraphQL is just a schema, there are many different implementations of a GraphQL server, AppSync being one of them. I mentioned Apollo server in this series as well.
The benefit of using GraphQL it is that you have a well defined representation of your data model that clients can query. And they only need to query the information they need.

AppSync provides the GraphQL server, authentication and data source integrations. You provide your schema and resolvers for the fields within it.

There are 2 core concepts with AppSync:

Data Source - A persistent storage system, e.g. a DynamoDB Table or a trigger e.g. a Lambda function
Resolver - Resolvers are comprised of request and response mapping templates. These templates map your GraphQL query to the appropriate request for your data source. For example if you wanted to query a DynamoDB table, your request mapping template would transform the GraphQL query into a DynamoDB query.

So if we jump back to our updated diagram, each of these Lambda functions (the orange icons) are configured as an individual data source in AppSync.

Up until this point we had been using Lambda Data Sources exclusively for our AppSync API. Every time a GraphQL query was executed, AppSync was invoking Lambda functions to get the data it needed.

But AppSync is so much more powerful than this! You can resolve data directly from various sources, including DynamoDB.

I was able to build the various queries needed for the customer portal without having to invoke a single Lambda function or write any “code” to make it happen.

Because we were now using two managed services (AppSync + DynamoDB) we were able to offload the gluing of these two services together into configuration rather than code. This is a good thing, instead of spending time writing “glue” code, we can instead write the code that matters to our customers, the code that is going to help them have some unique selling points in their marketplace.

Not only did this approach get rid of some code for us (Remember code is a liability). It also had a nice added benefit of being incredibly fast!

No Lambda function, no cold start!

This was the life changing moment for me; Combining the two services together directly resulted in performance that I was unable to achieve through writing my own Lambda function. It was from this moment on that I started to “trust” the managed services more and really dig deep into the Serverless offerings of AWS.

Like all technology decisions, we need to look at the downsides to this decision.

The big one is you have to use VTL templates. These are used to map the GraphQL queries to, in this example, a DynamoDB query:

It’s a brilliant idea and because most of the managed services are using HTTP APIs, you can effectively integrate with any of them.

For whatever reason someone at AWS decided that velocity templates were the way to go when we build up the requests and responses from these direct integrations.

These are hard to unit test and have limited utility methods vs something like a TypeScript Lambda function.

Thankfully AWS have recently released JavaScript resolvers, although still quite limited, they enable developers to instead write their resolver templates using JavaScript rather than VTL. A welcome improvement and we hope to adopt these for future use cases.

In our next post we will look at how we helped the team grow in confidence when working with the Serverless architecture of Stroll.

Zero to Serverless Car Insurance - Part 1

Matthew Wilson — Mon, 03 Apr 2023 14:26:54 +0000

In early 2021, we embarked on an ambitious project to build a car insurance platform from scratch using Serverless technology on AWS. Over the past two years, our team has learned a great deal about what it takes to build a Serverless platform - what it means to be Serverless-first, what works well, what doesn't, and most importantly, how to fully embrace the Serverless offering on AWS.

The official vision for Stroll is:

A digitally led broking business, combining innovative technology with industry-leading expertise to deliver exceptional customer experiences. Along the way, Stroll is transforming how people buy insurance and manage their policy, starting with the car insurance segment with a clear vision of building into other areas and across new territories.

The innovative technology mentioned in this vision is a shiny new car insurance platform built with managed services on AWS.
From the beginning we wanted to build it using Serverless technology.

Serverless has grown to mean many things and can even mean something different depending on which cloud provider you are paying for. For our team and for the purpose of this article, Serverless is a way of building and running applications without having to manage the underlying infrastructure.
Of course there are still servers running in a data centre somewhere but those servers are not our responsibility. With Serverless we as developers can focus on our core product instead of worrying about managing and operating servers or runtimes.

AWS Shared Responsibility Model

Most cloud providers have this concept of a shared responsibility model which breaks down what your responsibility is as a customer of AWS but also what AWS will take responsibility for.
What we want to do is move that dotted line up as much as possible and allow AWS to do the “undifferentiated heavy lifting” (the stuff that adds no value to the product) for us.

The benefits of Serverless are clear:

No infrastructure provisioning or maintenance
Automatic scaling
Pay for what you use
Highly available and secure

Through this series of blog posts, we're excited to share our team's Serverless journey with you and hope you learn something useful regardless of your favourite cloud provider.

Phase 1 - The Lambdalith

When I joined Instil 4 years ago one of the things I noticed right away was their focus on quality. Testing was at the core of every project you quickly buy in and appreciate the benefits of test driven development and automated testing. This project was no different. I can say with the utmost confidence that my colleagues had good unit testing in from the very start.
But there was one mindset that we started with that needed to change and that was the ability to build, run and test the backend locally.

It would be madness to not be able to run the entire stack on your machine right?!

However when you are building Serverless applications you really need to switch your mindset from running locally to running in the cloud. This will have a profound impact on not only how you verify that your application is working correctly but also how you architect the solution.
This project started off treating AWS Lambda as just another way to get some compute in the cloud. Yes we were using a Lambda function, but really it could have easily been a docker container or an EC2 instance instead.

The team had built what you could call a Lambdalith. A single Lambda running Apollo server and resolving all of the GraphQL mutations and queries within that single Lambda. Now why do such a thing?

The entire backend can be ran locally

Click the green play button in your IDE, or run a terminal command and off you go. You can have your client talking to the backend in no time.

The solution is more portable.

There is less chance of “vendor lock-in”, meaning if AWS suddenly gets too expensive you can move to another provider with less effort.

And yes those are valuable things, but what are the downsides?

Cold Starts

Spend anytime investigating AWS Lambda and you will most likely encounter the term “cold start”. Your functions will shutdown if they haven’t been invoked in a while so there is a period of time when they need to initialise the underlying runtime, load your function code and then execute it.

Source: https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/

In the early days of Lambda this cold start introduced so much latency that it ruled the service out for many teams. However because it’s a managed service AWS have been able to optimise this process and you essentially get those optimisations for “free” without needing to make any code changes.
Because of our decision to create a Lambdalith, we were doing some heavy lifting of our own after already incurring a cold start penalty. Namely the initialisation of Apollo server. The desire to test and run locally caused us to incur a performance hit in production!

The importance of testing locally forced the team to make an important architectural decision of a Lambdalith.

This not only meant that we were taking a performance and cost hit but we also were not making full use of other AWS managed services like AppSync (AWS' managed GraphQL Service).

So how do we make testing in the cloud easier for developers?

Ensure that every developer has their own AWS account. This is crucial, having a sandboxed environment makes it easy for developers to not only test their code but also test their deployments as well. You could have one shared developer account and prefix resources with developer names or something like that but this is just asking for some kind of conflict or accidental deletion.
Make the deployment process fast. We make heavy use of the hotswap and watch functionality in CDK, enabling developers to hot-reload their changes automatically.
If you need an even shorter feedback loop, consider invoking your Lambda code locally. It’s just a function after all and AWS services are accessed using HTTP APIs. You just need to make sure you have the correct permissions and resources deployed.

I like to think of this as phase one of the project. It worked, it was built using Serverless technologies but it had much room for improvement. The team realised that this approach was not going to work going forward and decided to replace Apollo server with AppSync, a good decision as we no longer had to worry about maintaining our own Apollo server and could instead hand that responsibility over to AWS. This change also enabled us to break down the Lambdalith into smaller domain based services instead, which we will cover in our next post.

Lambda Container Images

Matthew Wilson — Wed, 06 Jul 2022 20:37:00 +0000

Recently AWS released a new way for developers to package and deploy their Lambda functions as "Container Images". This enables us to build a Lambda with a docker image of your own creation. The benefit of this is we can now easily include dependencies along with our code in a way that is more familiar to developers. If you have used docker containers before, then this is much simpler to get started with than the other option - Lambda layers.

AWS have provided developers with a number of base images for each of the current Lambda runtimes (Python, Node.js, Java, .NET, Go, Ruby). It is easy for a developer to then use one of these images as a base, and build their own image on top.

Of course there are many sensible use cases for container images. Perhaps you want to include some machine learning dependencies? Maybe you would love to have FFMPEG in your lambda for your video processing needs? Or you want to nuke your entire AWS account to avoid a hefty bill?

You heard me, in this blog article, we are going to build a container image with aws-nuke installed! This will delete everything in an AWS account (excluding our fancy new container image lambda). Nuke is built using Go but we are going to get started with the node.js base image and build our own Lambda using JavaScript. This library isn't available on NPM, so there is no easy way to pull it into our Lambda function, but with container images we see that it provides a way for developers to mix and match different tools to build a scalable solution to the problem they are trying to solve.

To get started with our new container image, we can create a Dockerfile like so

FROM public.ecr.aws/lambda/nodejs:12
LABEL maintainer="Instil <team@instil.co>" 

COPY ./lambda/nuke.js ./lambda/package*.json ./
RUN npm install
CMD [ "nuke.lambdaHandler" ]

As you can see we are building from the lambda/nodejs:12 base image, and copying over our Lambda function code.
Notice the last line of our Dockerfile, CMD [ "nuke.lambdaHandler" ]. Because we are using one of the base images, it comes pre-installed with the Lambda Runtime Interface Client

The runtime interface client in your container image manages the interaction between Lambda and your function code. The Runtime API, along with the Extensions API, defines a simple HTTP interface for runtimes to receive invocation events from Lambda and respond with success or failure indications.

Therefore CMD [ "nuke.lambdaHandler" ] lets the interface client know what handler function to call when it receives an invocation event.

Before we add the nuclear option, lets create the skeleton for our handler function:

exports.lambdaHandler = async (event) => {
    const response = { statusCode: 200 };
    return response;
};

For now it simply returns a 200 response.

Not only does our container image include the Lambda Runtime Interface Client, it also includes the Runtime Interface Emulator. This allows you to test your function locally, which in my opinion, is one of the killer reasons to adopt container images for your project.

Given we have a project structure like this:

.
├── Dockerfile
├── docker-compose.yml
└── lambda
    ├── nuke.js
    └── package.json

Then to build our container image, we simply use the docker cli:

docker build -f ./Dockerfile -t instil-nuke .

And to run it locally:

docker run -p 9000:8080 instil-nuke

Then to test our function locally, we just need to hit our lambda with an http request. In this example we are posting an empty JSON body:

➜  curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
{"statusCode":200}%

The url seems strange, but the Runtime Interface Emulator is simply providing an endpoint that matches the Invoke endpoint of the Lambda API. The only difference between this local URL and the real API URL is that our function name is hardcoded as function.

Being able to run our function locally like this greatly reduces the feedback loop when developing your Lambda. There are other options out there for running Lambdas locally; for example sam local, but the container image approach gives you a local test environment that is much closer to how it will be ran on AWS.

Now that we have our project structure in place, lets take a look at adding aws-nuke to our container image.

FROM public.ecr.aws/lambda/nodejs:12
LABEL maintainer="Instil <team@instil.co>" 

RUN yum -y update
RUN yum -y install tar gzip

COPY ./resources/aws-nuke-v2.15.0.rc.3-linux-amd64.tar.gz ./resources/nuke-config.yml ./
RUN tar -xzf ./aws-nuke-v2.15.0.rc.3-linux-amd64.tar.gz && mv aws-nuke-v2.15.0.rc.3-linux-amd64 aws-nuke

COPY ./lambda/nuke.js ./lambda/package*.json ./
RUN npm install
CMD [ "nuke.lambdaHandler" ]

Adding dependencies is just how you would expect if you have used docker before. In the above example we are adding dependencies using yum and copying nuke onto our image.

Then all we need to do it update our function to execute aws-nuke.

const { execSync } = require('child_process');

function run(command) {
    console.log(command);
    const result = execSync(command, {stdio: 'inherit'});
    if (result) {
        console.log(result.toString());
    }
}

function nuke() {
    console.log("Nuking this AWS account...");
    const accessKey = process.env.AWS_ACCESS_KEY_ID;
    const secretAccessKey = process.env.AWS_SECRET_ACCESS_KEY;
    const sessionToken = process.env.AWS_SESSION_TOKEN;
    run(`./aws-nuke -c nuke-config.yml --access-key-id ${accessKey} --secret-access-key ${secretAccessKey} --session-token ${sessionToken} --force --force-sleep 3`);
    console.log("Your AWS account has been nuked, you can sleep peacefully knowing that you will no longer get an unexpected bill.");
}

exports.lambdaHandler = async (event) => {
    nuke();
    const response = { statusCode: 200 };
    return response;
};

We can use execSync to execute a command in our running lambda, it's easy to see how simple it is to utilise external dependencies in our Lambda environment with this new container image option. Notice that we are pulling AWS access tokens from environment variables so that nuke can use them, this is default behaviour for Lambda functions and they are the access keys obtained from the function's execution role.

With our updated container image ready to nuke our account, all we need to do is deploy it. For this we need to create an ECR repository and push our image to it:

# Replace [AWS_ACCOUNT_NUMBER] with your own AWS account number
aws ecr create-repository --repository-name instil-nuke --image-scanning-configuration scanOnPush=true
docker tag instil-nuke:latest [AWS_ACCOUNT_NUMBER].dkr.ecr.eu-west-1.amazonaws.com/instil-nuke:latest
aws ecr get-login-password | docker login --username AWS --password-stdin [AWS_ACCOUNT_NUMBER].dkr.ecr.eu-west-1.amazonaws.com
docker push [AWS_ACCOUNT_NUMBER].dkr.ecr.eu-west-1.amazonaws.com/instil-nuke:latest

Now that our container image lives in AWS, we just need to create our Lambda function. In the Create function page of the AWS management console, you will notice there is a new option to use a Container image as your starting point:

Choosing this option then enables you to pick your container image; click the Browse images button to select your freshly uploaded image. You should be left with something like this:

And that's it! All that's left to do is trigger our Lambda function. For our example we could detonate the nuke once we get a billing alarm over a certain threshold but for the sake of keeping this article focused on container images, lets just trigger it with a test event for now and inspect the output. We will publish another article in the future explaining how to hook this up to a billing alarm.

Notice the very disappointing output of our detonation:

The above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources.

You didn't think I was actually going to nuke my AWS account did you? 😊

AppSync and DynamoDB Lessons Learned

Matthew Wilson — Wed, 06 Jul 2022 20:22:00 +0000

When building a serverless first platform it’s very hard to ignore the compelling feature set offered by AWS AppSync and DynamoDB. Each service has their own list of great features, but the magic lies when you combine these two services together. I have often said that the sweet spot when working with AWS is when you start to play AWS lego; use the building blocks provided by AWS to “click” the services together using configuration rather than code.

We are using these two services to help build a fully serverless insurance platform and in this article I’m going to share some lessons we have learnt along the way.

1. Determine if you want to use single or multi table design with DynamoDB

One thing the serverless community can agree on is that Alex DeBrie’s DynamoDB book is a must read if you are starting your journey with DynamoDB. DeBrie lays out the best practices for working with DynamoDB, single table design and modelling your tables. He also gives advice on when it may not be appropriate to use single table design, including when you are working with GraphQL.

This is a decision that you will want to make early on in your project. At this stage it's almost universally accepted that single table design is the right approach for most use cases.

For our project we decided on a mixed approach. Where appropriate, we will use single table design to group closely related items together. Closely related items could be defined as items that are likely to be queried together. Determining your access patterns before working on a feature will help you to better understand the relationship between your entities and the types of queries you are going to use.

For loosely related items that sometimes are queried together but also separately, we typically store those entities in separate tables. We rely on AppSync to resolve our entities and aggregate the data into a single response for the client.

We would also use separate tables if the entity stored in a table is versioned. Rather than storing multiple different entities in a table, we would instead store multiple versions of an entity. Keeping the versioned entity in its own table helps to reduce complexity of that table and makes it easier to work with when resolving data from the table.

2. Model your tables following single table design best practices

This point applies regardless if you are using a single or multiple tables!

Part of single table design is decoupling your partition key and sort key from the actual record you are trying to store.

Let's take a record that has an ID, it would be tempting to make this field your partition key and perhaps Date Created as your sort key.

This approach makes your table a lot less flexible for the future. Perhaps you will realise that storing multiple entities in this table is a good idea, by having a partition key named ID it makes it harder to put something else in that field, for example a username.

Instead store the entity type and identifier in a field called partitionKey and sortKey. This is commonplace in single table design, but is still a good practice to follow when using multiple tables.

Some developers prefer to use PK and SK as the names for these fields. At Instil, we tend to avoid acronyms where possible and prefer to use clear and unambiguous naming conventions. Therefore we name these fields “partitionKey” and “sortKey”. At scale this might be something to consider when you want to shorten your attribute names to save on storage but that isn’t an issue for our particular project.

One side point is to always include a sort key field even if you don’t need it right away. You can't add a sort key after a table has been created!

3. Use AppSync DynamoDB resolvers

A game changing moment for our platform was when we switched from using Lambda resolvers to DynamoDB resolvers. The ability to connect your API directly to your database delivers performance that a Lambda cannot match. It also enables developers to quickly add new features to your API whilst adding very little "code".

Multi-table design makes this approach much simpler, it removes the need for complex VTL resolver templates that transform data returned from a table following single table design. AppSync will effortlessly query your tables in parallel and aggregate the responses into the format that matches your GraphQL schema.

4. Be wary of the N+1 Problem

The N+1 problem isn’t specific to AppSync, it can occur in any GraphQL system when one top level query produces N items whose type contains a field that must also be resolved.

For example, in the query below, listOrders returns N items, and each item has a reviews field that must be resolved in turn.

query getOrders {
  listOrders { // 1 top level query
    items { // N items
      id
      reviews { // N additional queries
        author
      }
    }
  }
}

AWS has some advice on how to work around this - https://aws.amazon.com/blogs/mobile/introducing-configurable-batching-size-for-aws-appsync-lambda-resolvers/

But I think this should instead be factored into your GraphQL schema and DynamoDB table design to eliminate the N+1 problem where possible. Perhaps there is an argument here to store the items and reviews in a single table to aid with querying that data, but other entities can still live in different tables.

Hopefully you have found these points useful as you continue your development journey with AppSync and DynamoDB. We have found that the combination of these two services has provided a flexible architecture that we have been able to extend and refactor over time as we deliver new features.

We develop custom cloud and mobile software products for some of the world's leading brands. If you would like to learn more get in touch.

Improving CDK development cycle

Matthew Wilson — Wed, 16 Mar 2022 22:50:00 +0000

Here at Instil we have been working on a fully serverless car insurance platform that utilises the AWS CDK for deployments. One of the lessons we learnt early on in this project is that there is no substitution for deploying and testing our code in AWS; each of our developers have their own AWS accounts that they regularly deploy code to. However with CDK sometimes these deployments can take a while...

As an example, we have some versioned Lambda functions that run pre-traffic test functions with CodeDeploy to verify that our deployed code is configured and behaving correctly. Typically these can take 3-4 mins to deploy, run tests, and switch traffic when using the cdk deploy [stackname] command.

While this functionality is extremely useful, those 3-4 mins spent waiting on deployments add up, thankfully CDK has a few ways of reducing this feedback loop that we should all be aware of.

The AWS CDK Toolkit (aka the CDK CLI) page on GitHub covers all of the CLI commands in detail and is worth keeping up-to date with for any future improvements!

What is hotswapping?

You can pass the --hotswap flag to the deploy command:

$ cdk deploy --hotswap [StackNames]

If you have only changed the code of a Lambda function, and nothing else in CDK this will attempt a faster deployment by skipping CloudFormation, and updating the affected resources directly. If something cannot be hot swapped then CDK will fall back and perform a normal deployment.

Currently hotswapping is supported in the following scenarios:

Code asset (including Docker image and inline code) and tag changes of AWS Lambda functions.
AWS Lambda Versions and Aliases changes.
Definition changes of AWS Step Functions State Machines.
Container asset changes of AWS ECS Services.
Website asset changes of AWS S3 Bucket Deployments.
Source and Environment changes of AWS CodeBuild Projects.
VTL mapping template changes for AppSync Resolvers and Functions.

This list could update so be sure to check out this page for the most up-to date list.

The hotswap command deliberately introduces drift in CloudFormation stacks in order to speed up deployments. For this reason, only use it for development purposes. Never use this flag for your production deployments!

How does CDK know when to hotswap?

Behind the scenes CDK uses another useful CLI command: cdk diff.

The diff command computes differences between the infrastructure specified in the current state of the CDK app and the currently deployed application.

If you run cdk diff after making only code changes you will see output that looks like this:

Stack MyStack
Resources
[~] AWS::Lambda::Function
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac.zip
 │       └─ [+] 21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac
         └─ [+] asset.21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0

Notice the changes only refer to an Asset and nothing else in the stack. This is considered “hot-swappable” and will be hot swapped successfully by using the --hotswap flag.

Running cdk diff when there are non-code changes would output something like this:

Resources
[~] AWS::Lambda::Function
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac.zip
 │       └─ [+] 21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0.zip
 ├─ [~] Environment
 │   └─ [~] .Variables:
 │       └─ [+] Added: .MATTHEW_WAS_HERE
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac
         └─ [+] asset.21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0

Notice the changes includes an environment variable addition as well as code changes. This change could not be hot swapped and would fall back to a normal deployment.

Watching for changes

CDK also provides a nice command called cdk watch, this uses the hotswap functionality and continuously monitors the files of the project, and triggers a deployment whenever it detects any changes.

https://github.com/aws/aws-cdk/tree/master/packages/aws-cdk#cdk-watch

A nice feature of this command is that it will also tail the CloudWatch logs after deploying your code! Hopefully reducing your feedback loop even more.

Summary

In our testing using --hotswap for our versioned lambdas shaved ~4 mins off our deployment times!

To ensure that hotswaps function correctly you want to adhere to the CDK principle of keeping your stacks deterministic, for example avoid dynamically generating properties of your constructs (e.g. using a timestamp, or network lookups).

If you or your team want to learn more about CDK, be sure to check out our TypeScript for AWS Serverless Course.

CDK Lessons Learned

Matthew Wilson — Wed, 16 Mar 2022 22:48:26 +0000

The AWS Cloud Development Kit (CDK) allows you to define your AWS resources using the programming languages you know and love. This concept piqued the interest of many of us here at Instil; when someone offers us the ability to use Typescript instead of YAML we’re sold!

I have been using CDK for the past 3 years for container based and serverless projects, and what I think is CDK’s greatest strength are the guard rails it provides to the developer:

Like any good API it is self-documenting, the declarative style helps to guide you on what resources are compatible with each other without the need to know about every feature in every AWS service.
The built-in helpers for generating IAM roles give you safe defaults to ensure you are following security best practices.
The high level (level 3) constructs as well as open source versions (like CDK patterns and Construct Hub) accelerate your application development by providing patterns that ensure your architecture is scalable, cost effective and secure.

Despite all of CDK’s strengths it’s important to know the weaknesses of the framework and protect your team from any pitfalls they could encounter. Here are our top 5 lessons we have learnt on our CDK journey.

1. Separate your stateless and stateful resources into different stacks

Keeping your stateful resources, (e.g. DynamoDB tables, RDS instances or Cognito User Pools) in separate stacks to your stateless resources, (e.g. Lambda functions or ECS Services) means that you will be able to delete and re-create your stateless stacks without losing any data during the process. This is a lesson we learnt early on our CDK journey and has proven a useful pattern across multiple projects.

2. Turn on termination protection for your production environments

It’s very easy for a developer to log in to the wrong AWS account and accidentally delete a CloudFormation stack (don’t ask me how I know this!). Turning on termination protection is an extra layer of protection that helps avoid those pesky developers taking down an environment by mistake 🙈. Using CDK makes it easy to include conditional logic to only enable termination protection in production accounts, reducing overhead on developers day to day.

https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_core.Stack.html#terminationprotection

Note: I used the term "extra layer" of protection. Termination protection shouldn't be used as an excuse for poorly managed IAM roles with more privileges than required!

3. Refactoring isn't as easy as you think

This is a lesson that has taken us longer to learn. We love clean code at Instil, refactoring is everything to us! But because CDK is “code” you can be lulled into a false sense of security when it comes to refactoring. Simply moving a resource from one stack to another could cause it to be deleted and recreated. Sometimes you can have a beautifully refactored stack, only to learn at deployment time that you have broken something as a resource is referenced in other stacks.

A rule of thumb we like to follow is this: when a stack has been deployed to production, take extra care when refactoring. Deploy the original version to your own developer account first, and then try to deploy your refactored version.

Be careful not to change the logical ID of your CDK resources, this will result in a deletion and re-creation of the resource.

4. Use the IAM convenience methods

One of the best guard rails provided by CDK are the convenience methods for automatically creating IAM roles. For example if you had a lambda function that you wanted to have read-only access to a single dynamodb table:

myDynamoTable.grantReadData(myLambdaFunction);

Most of these convenience methods start with .grant making discoverability easy in your IDE.
Using these convenience methods makes it easier to follow the principle of least privilege without having to write your own IAM roles. If you find yourself writing your own roles, think twice and investigate if a convenience method is there instead.

5. Use abstractions carefully

There are multiple levels of constructs that CDK provides:

L1 - “L1 constructs are exactly the resources defined by AWS CloudFormation”
L2 - “L2 constructs also represent AWS resources, but with a higher-level, intent-based API. They provide similar functionality, but provide the defaults, boilerplate, and glue logic you'd be writing yourself with a CFN Resource construct”
L3 - “These constructs are designed to help you complete common tasks in AWS, often involving multiple kinds of resources.” These are more like an opinionated pattern.

It is tempting to build a bunch of “L2.5” constructs that are specific to your project. For example ProjectNameS3Bucket. There is a time and a place where you might want to use these in a project, especially when you want to avoid duplicating code. However we recommend that you don't try to do this too early in the project. Try to keep these pointers in mind:

Are you reducing flexibility of the underlying L2 construct? - It’s hard to make one configuration of a bucket useful across multiple stacks in a project.
Do your default values work for all use cases of your construct? - Could someone change a default property of the shared construct that unknowingly breaks another part of the application?

I hope that you found this helpful. As I said at the beginning there are many strengths with CDK but it is important to be aware of the pitfalls. If you or your team want to learn more about CDK, be sure to check out our TypeScript for AWS Serverless Course.