Forem: Wojciech Matuszewski

Automatic AWS CloudFormation rollbacks upon a test failure in your CI pipelines

Wojciech Matuszewski — Thu, 10 Nov 2022 04:38:00 +0000

As serverless usage grows within an organization, developers deploy an ever-increasing number of microservices. In a healthy company culture, these services require tests and validations to indicate that the service is ready for deployment. Since we are only humans, mistakes happen, and sometimes, a given deployment requires a rollback due to failure in said checks.

Luckily for us, the advent of the infrastructure as code tools and the cloud made it much easier to incorporate testing, validation checks, and rollback mechanisms in your deployment pipeline.

This article describes four methods one could use to perform validations on their deployed infrastructure and roll the changes back if necessary in the context of AWS and serverless services.

Many ways to test the AWS serverless stack

In my professional career, I've used many techniques to confirm the code I deploy meets the business and customer demands.

Having a vast suite of unit tests for business logic.
Writing end-to-end and integration tests to ensure the different services work with each other as expected.
Maintaining a set of canary tests that constantly exercise the API to ensure the service is available to the users.

While having different return on investment characteristics (for example, the end-to-end and integration tests usually have higher ROI than the unit tests), these methods have been proven reliable through many applications I had the pleasure to work on.

There is a small catch to consider – if the tests (be it end-to-end or any other relevant to your situation) do not integrate with AWS CloudFormation deployment lifecycle, rolling back the deployment is very hard when they fail. The trick is to ensure that we can safely rollback to the previous application deployment when a test fails. I will show you how to achieve that in the following sections.

In the above example, the tests run after the deployment happens. If the tests detect a regression, the developers must manually revert the change to bring the system back to the previous, presumably correct, working state. Such a setup is suboptimal from the operational perspective – it brings toil and frustration, especially in high-stress situations.

Using CodeDeploy deployment group

AWS's developer suite of products includes the AWS CodeDeploy offering, which can help developers deploy AWS Lambda functions and other compute-related services.

Since this blog focuses on AWS serverless services, we will look at AWS Lambda integration with AWS CodeDeploy and omit other services it integrates with.

The AWS CodeDeploy has the ability to shift traffic to our AWS Lambda functions gradually, in a specific time window (so-called rolling window). The deployment group functionality integrates well with AWS CloudFormation – the AWS CloudFormation deployment will not finish until we either shift all traffic to our updated AWS Lambda function. If the AWS CodeBuild detects a regression (usually an alarm alarming), it will instruct the AWS CloudFormation to rollback the current deployment.

Keep in mind that we are not constrained to a single alarm. If you wish to monitor multiple alarms, you can create a composite alarm for the AWS CodeBuild to keep an eye for.

The main benefit of using the AWS CodeDeploy service is that it is a native AWS primitive. It works well for most common use cases and is relatively easy to set up.

The main drawback for me personally is the inflexibility of the service. To the best of my knowledge, creating a custom rolling window configuration is impossible. If you wish to do so, you must develop a bit of custom infrastructure (see the "Using the IntrinsicValidator package section).

Using the IntrinsicValidator package

Specific to AWS CDK, the IntrinsicValidator is a third-party package that works similarly to the AWS CodeBuild offering. But it is, in my humble opinion, more extendable and flexible than the service mentioned before. The underlying implementation relies on AWS StepFunctions to poll for various checks and alarm statuses.

The main benefit of using the IntrinsicValidator over a service like AWS CodeDeploy is flexibility. Complete control over how long we probe for an alarm or failure during the deployment is excellent – it allows us to fine-tune the speed of the CI pipelines. If your team is unhappy with monitoring options, you can always fork the construct and amend it to fit your needs.

The main drawback I see here is that the AWS Lambda traffic shifting, which is native to AWS CodeBuild, is not that easy to implement with the IntrinsicValidator. I'm unsure how to replicate such functionality using this construct, as the AWS Lambda deployments are "all or nothing" by default (I'm happy to be corrected here).

Using a AWS CloudFormation Custom resource

Another way to run code during the AWS CloudFormation deployment is to use the AWS CloudFormation Custom resource. Here, similar to how the IntrinsicValidator package operates, we instruct the AWS CloudFormation to run a bit of code (in our case an AWS Lambda Function) during the deployment. But, instead of relying on a collection of resources from a third party, we stick to native AWS primitives.

The main difference between the IntrinsicValidator and the AWS CloudFormation Custom resource is developer experience and ergonomics. As mentioned before, the IntrinsicValidator is, at its core, an AWS CloudFormation Custom resource with several layers of abstractions baked in. It handles returning the correct response to the AWS CloudFormation service and other details required to make custom resources work.

I would recommend using the custom resource for relatively simple one-off checks or if you do not wish to bring a dependency related to deployments to your project (which is understandable and very reasonable). If you go the "raw" custom resource route, make sure to use any deployment frameworks or Lambda Function-related libraries. They usually make it much easier to integrate with the AWS CloudFormation lifecycle correctly (for example, the custom_resources AWS CDK package).

Using the cdk-triggers package

Since we are discussing the AWS CloudFormation Custom resources, the cdk-triggers deserves an honorable mention. The package provides an abstraction layer over the custom_resources module with a couple of niceties on top.

I have not seen much development work going into this package, so I would be wary of using it in production. Still, nevertheless, I think it is a vital alternative to any other way of couping tests with the AWS CloudFormation lifecycle we have seen so far.

The bottom line

In the end, what matters is that you test your application. Running tests after the deployment, albeit not very optimal due to difficulties with rolling back, is still much better than not doing so. Consider coupling some of your tests with the AWS CloudFormation deployment lifecycle as your product matures. Automatic rollback upon a test failure is a great way to prevent pushing broken builds to production environments.

Closing words

In this article, we have discussed three and a half ways to couple your tests with the AWS CloudFormation lifecycle. The list is far from exhaustive. I'm confident that there are other ways to do so. If you have an experience in this space and want to share some details, please feel free to reach out to me.

For more AWS / serverless content, consider following me on Twitter - @wm_matuszewski

Thank you for your precious time.

AWS CDK and Amazon DynamoDB global tables

Wojciech Matuszewski — Fri, 02 Sep 2022 03:55:20 +0000

Amazon DynamoDB is THE database for serverless AWS applications. Its HTTP-based connection model makes integrating with serverless computing services like AWS Lambda easy. One of the additional capabilities of Amazon DynamoDB is the global tables, which allows you to run replica instances of the database in other regions.

In this blog post, I will touch on Amazon DynamoDB global tables in the context of the AWS CDK framework – there are a couple of nuances with huge implications you should be aware of.

Let us dive in.

Traditional way of creating global tables

There are two ways I'm aware of to create global tables using AWS CDK, with one of them being, in my humble opinion, much better for the maintainability of your application.

Let us start with the "traditional" way of creating the Amazon DynamoDB global tables via the AWS CDK – by using the aws_dynamodb.Table construct and specifying the replicationRegions property.



const globalTable = new dynamodb.Table(this, "Table", {
  partitionKey: { name: "id", type: dynamodb.AttributeType.STRING },
  replicationRegions: ["us-east-1", "us-east-2", "us-west-2"],
  billingMode: dynamodb.BillingMode.PROVISIONED
});

Here I've defined a table with three replication regions. If you deploy this code snippet in the AWS DynamoDB web console, you will see that the table has three replicas – one for each region.

On the surface, all looks good and works as expected. But as soon as you peek "under the curtains" of the construct, you will see something unexpected – that deploying the "Table" created an AWS CloudFormation custom resource alongside the DynamoDB resource.

The problem with the custom resource

You can find the code the custom resource in question uses here.

The custom resource is responsible for creating the replica tables. It uses the AWS SDK updateTable API and provides the appropriate value for the ReplicaUpdates property. On the surface, it does not sound bad, but by creating the replicas via the AWS SDK call, we effectively detach those resources from your AWS CloudFormation template and AWS CDK code.

The implications of this architecture are enormous from the maintainability perspective. Here are the two cases of confusion and toil I had to deal with multiple times when operating global tables deployed via the replicationRegions property.

The `DeletionPolicy` attribute

If you are running a production workload, it is considered good practice (I would argue a must-have) to specify the DeletionPolicy (in AWS CDK, the name is removalPolicy) of RETAIN for resources that are stateful and hold production data.

If you delete the CloudFormation stack, the CloudFormation will not delete the resources when we set the DeletionPolicy to RETAIN. As you can imagine, this can save you from a disaster.

Usually, in AWS CDK, this is done by using the applyRemovalPolicy method on a given construct.



const globalTable = new cdk.aws_dynamodb.Table(this, "Table", {
  // ...
});
globalTable.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);

The problem is that replicas created via the custom resource will not inherit that removalPolicy. This setting will only apply to the "root" table.

So, If I were to delete a CloudFormation stack, the CloudFormation will preserve the "root" table but delete the custom resource. Since the custom resource triggered with a Delete event, it would issue the Delete action for all the replica tables. And with that, all of your replica tables are gone.

To make sure the Delete event never gets to the custom resource the AWS CDK uses, you have to set the DeletionPolicy on the custom resource itself. That is not so easy as you have to "find" the custom resource node in the AWS CDK tree and override its DeletionPolicy attribute. Here is how I would do it.



const globalTable = new cdk.aws_dynamodb.Table(this, id, {
  // ...
});

/**
 * This only applies to the "root table" and NOT THE REPLICAS!
 */
globalTable.applyRemovalPolicy(removalPolicy);

/**
 * Make sure we do not remove the custom resource
 */
const customReplicaResource = globalTable.node.children.find(child => {
  return (child as any).resource?.cfnResourceType === "Custom::DynamoDBReplica";
}) as cdk.CfnResource;

customReplicaResource.applyRemovalPolicy(removalPolicy);

Enabling point-in-time recovery on the replica tables

For production usage, enabling point-in-time recovery (PITR) is arguably a must. It is not only helpful in data-corruption scenarios but also for disaster recovery. In CDK, applying the PITR on a global table (created via the replicationRegions property) might initially seem easy.



const globalTable = new cdk.aws_dynamodb.Table(this, id, {
  pointInTimeRecovery: true
  // ...
});

The problem is, similarly to the DeletionPolicy we have looked at earlier, that the pointInTimeRecovery property only applies to the "root" table. It will NOT be "forwarded" to the replica tables. An auditing tool will catch that for you in the best-case scenario. In the worst, you will not be able to recover your replica tables if something goes wrong.

All hope is not lost, though. You could also use a workaround to ensure that the replica tables have the same PITR setting as your "root" table. This involves calling updateContinuousBackups yourself via the AwsCustomResource construct for each region where the replica resides.



const replicationRegions = ["eu-center-1", "eu-west-1"];

const globalTable = new cdk.aws_dynamodb.Table(this, id, {
  pointInTimeRecovery: true,
  replicationRegions
  // ...
});

for (const replicationRegion of replicationRegions) {
  new cdk.custom_resources.AwsCustomResource(this, id, {
    onUpdate: {
      service: "DynamoDB",
      action: "updateContinuousBackups",
      parameters: {
        TableName: globalTable.tableName,
        PointInTimeRecoverySpecification: {
          PointInTimeRecoveryEnabled: true
        }
      },
      region: replicationRegion,
      physicalResourceId: cdk.custom_resources.PhysicalResourceId.of(id)
    },
    policy: cdk.custom_resources.AwsCustomResourcePolicy.fromSdkCalls({
      resources: [globalTable.tableArn]
    })
  });
}

A different way of creating global tables

Instead of using the aws_dynamodb.Table construct, consider using the aws_dynamodb.CfnGlobalTable. To the best of my knowledge, this resource was made available to us in May 2021, and I would highly encourage you for it to be your default, even if you do not anticipate using replica tables in the immediate future in your architecture.

If you want to dive deeper into technical details about the resource to which CfnGlobalTable corresponds, look no further than this excellent blog post.

I advocate for the CfnGlobalTable usage because the gotchas mentioned in the previous section do not apply to this construct. For example, to specify the DeletionPolicy, which would apply to all tables, even the replicas, all you need to do is to use the applyRemovalPolicy method available on the construct.



const globalTable = new cdk.aws_dynamodb.CfnGlobalTable(this, id, {
  // ...
});
globalTable.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);

What about the PITR? To enable it on all the tables (replicas and the "root" table) do not need to fiddle with AWS SDK calls through a custom resource (or some other way). You can set the PITR for the "root" table and the replica tables.



const globalTable = new cdk.aws_dynamodb.CfnGlobalTable(this, id, {
  // ...
  replicas: [
    {
      region: "eu-center-1",
      pointInTimeRecoverySpecification: {
        pointInTimeRecoveryEnabled: true
      }
    },
    // Let us say that, the root table lives in "eu-west-1"
    {
      region: "eu-west-1",
      pointInTimeRecoverySpecification: {
        pointInTimeRecoveryEnabled: true
      }
    }
  ]
});

Another significant reason for using the CfnGlobalTable is that this resource uses a newer version of Amazon DynamoDB global tables. You can read about different versions of global tables here.

Migration

I'm not yet ready to publish a migration story here (I'm still evaluating different avenues to do so), so please keep an eye on my next article. If you want to perform migration now, I would base the steps necessary on this and this article.

Closing words

I hope that the information in this article will save you some toil and frustrations I had to go through while working with global tables created via the replicationRegions property in CDK.

As I mentioned, the CfnGlobalTable is objectively better suited for the job. Yes, you might lose some of the niceties of the L2 (L3?) CDK construct, but using the newer version of the Amazon DynamoDB global tables will pay dividends in the long run.

For more similar content, please consider following me on Twitter – @wm_matuszewski.

Thank you for your time.

Strategies to test AWS AppSync VTL templates

Wojciech Matuszewski — Sat, 06 Aug 2022 07:40:55 +0000

AWS AppSync is a serverless GraphQL and Pub/Sub API service that many developers love. Turing the complex job of creating and managing a GraphQL API with real-time capabilities into a few lines of IaC (or clicks in the console) is a great win for AWS serverless developers.

The AWS AppSync enables you to integrate with other AWS services via AWS Lambda functions or VTL mapping templates. Historically it was pretty hard to test the logic written in the VTL templates. As the service matured, AWS added more and more ways to do so, enabling us, developers, to ensure the service we are developing is working correctly.

In this blog post, I will showcase all the strategies for testing AWS AppSync VTL templates I've seen in the wild. This, by no means, is a comprehensive list – these are the ways I'm familiar with. If you know a different way, please feel free to reach out!

You can find the code for all the strategies listed in the article in this GitHub repository.

Testing VTL templates using the AWS SDK

This is a technique I've discovered recently, and it quickly became my favorite. The AppSync SDK exposes the evaluate-mapping-template CLI call for parsing AppSync VTL templates using the AppSync SDK.

When I read about this in this AWS blog post I was thrilled as I was not very pleased with how I've been testing the templates so far (another strategy that I will touch on in this article).

The test boils down to evaluating the VTL template via the AWS SDK, then asserting on the result. You have only one dependency to maintain – the AWS SDK for AppSync, which is a huge win. The following is a snippet that evaluates a VTL template via the AppSync SDK responsible for fetching a user from AWS DynamoDB.

You can find the full source code for this testing strategy here.

import {
  AppSyncClient,
  EvaluateMappingTemplateCommand
} from "@aws-sdk/client-appsync";

const client = new AppSyncClient({});

test("template test", async () => {
  const template = `{
    "version": "2018-05-29",
    "operation": "GetItem",
    "key": {
        "id": $util.dynamodb.toDynamoDBJson($context.arguments.id),
    }
}`;
  const context = JSON.stringify({ arguments: { id: "USER_ID" } });

  const templateEvaluationResult = await client.send(
    new EvaluateMappingTemplateCommand({
      template,
      context
    })
  );

  expect(templateEvaluationResult.evaluationResult).toMatchInlineSnapshot(`
    "{
        \\"version\\": \\"2018-05-29\\",
        \\"operation\\": \\"GetItem\\",
        \\"key\\": {
            \\"id\\": {\\"S\\":\\"USER_ID\\"},
        }
    }"
  `);
});

Since we are using the official AWS SDK, we can be pretty confident that if we were to deploy an API and use this template, this is how the AWS AppSync service would render it internally.

One drawback of this way of testing VTL templates is that the environment in which the test is running has to have access to AWS credentials. That never was a problem for me, but it all depends on your situation, so be mindful of that.

Testing VTL templates using VTL parsers

This is the technique I've used before I learned about the AppSync SDK's ability to render templates. Instead of relying on the SDK, we use libraries the AWS Amplify uses internally to render the template.

The test setup is a bit more involved, but the result is pretty much the same as in the previous example. The following is the snippet rendering the VTL template using Amplify VTL parsers.

You can find the full source code for this testing strategy here.

import velocityUtil from "amplify-appsync-simulator/lib/velocity/util";
import velocityTemplate from "amplify-velocity-template";
import velocityMapper from "amplify-appsync-simulator/lib/velocity/value-mapper/mapper";

test("template test", async () => {
  const template = `{
    "version": "2018-05-29",
    "operation": "GetItem",
    "key": {
        "id": $util.dynamodb.toDynamoDBJson($context.arguments.id),
    }
}`;

  const errors: any[] = [];
  const now = new Date();
  const graphQLResolveInfo = {} as any;
  const appSyncGraphQLExecutionContext = {} as any;

  const vtlUtil = velocityUtil.create(
    errors,
    now,
    graphQLResolveInfo,
    appSyncGraphQLExecutionContext
  );

  const vtlAST = velocityTemplate.parse(template);
  const vtlCompiler = new velocityTemplate.Compile(vtlAST, {
    valueMapper: velocityMapper.map,
    escape: false
  });

  const context = {
    arguments: { id: "USER_ID" },
    args: { id: "USER_ID" }
  };

  const renderedTemplate = vtlCompiler.render({
    util: vtlUtil,
    utils: vtlUtil,
    ctx: context,
    context
  });

  expect(renderedTemplate).toMatchInlineSnapshot(`
    "{
        \\"version\\": \\"2018-05-29\\",
        \\"operation\\": \\"GetItem\\",
        \\"key\\": {
            \\"id\\": {\\"S\\":\\"USER_ID\\"},
        }
    }"
  `);
});

As I mentioned earlier, I'm not a massive fan of this technique. My main gripe is that I have to pull additional dependencies, which I have no confidence are well maintained, to my project.

Another issue I have is that the dependencies might be out of date and yield a template that might be different than the one produced internally inside the AWS AppSync service.

As for positives – since we are not making any calls to AWS, this technique does not require you to have AWS credentials in place. It might be a huge plus for some, but not so much for others.

Testing VTL templates using AWS AppSync simulator

Next up, going up in the complexity spectrum, this technique sits in a weird spot between local emulation and exercising deployed AWS infrastructure.

Instead of rendering the template and asserting its shape, we employ the amplify-appsync-simulator package to process the template and make the request to an actual (or local instance of) AWS service. This strategy stands on the shoulders of the AWS Amplify mocking and testing capabilities.

You can find the full source code for this testing strategy here.

import { AppSyncUnitResolver } from "amplify-appsync-simulator/lib/resolvers";
import {
  AmplifyAppSyncSimulator,
  AmplifyAppSyncSimulatorAuthenticationType,
  RESOLVER_KIND
} from "amplify-appsync-simulator";

test("template test", async () => {
  const simulator = new AmplifyAppSyncSimulator();
  simulator.init({
    dataSources: [
      {
        type: "AMAZON_DYNAMODB",
        name: "dynamodb",
        config: { tableName, endpoint: tableEndpoint }
      }
    ],
    appSync: {
      name: "testAppSyncAPI",
      additionalAuthenticationProviders: [],
      defaultAuthenticationType: {
        authenticationType: AmplifyAppSyncSimulatorAuthenticationType.API_KEY
      }
    },
    schema: {
      content: `
        schema {
          query: Query
        }

        type Query {
          getUser(id: ID!): User!
        }

        type User {
          id: ID!
          name: String!
        }
      `
    }
  });

  const requestTemplate = `{
    "version": "2018-05-29",
    "operation": "GetItem",
    "key": {
        "id": $util.dynamodb.toDynamoDBJson($context.arguments.id)
    }
}`;

  const responseTemplate = `$util.toJson($context.result)`;

  const resolver = new AppSyncUnitResolver(
    {
      requestMappingTemplate: requestTemplate,
      responseMappingTemplate: responseTemplate,
      kind: RESOLVER_KIND.UNIT,
      fieldName: "mockFieldName",
      typeName: "mockTypeName",
      dataSourceName: "dynamodb"
    },
    simulator
  );

  const result = await resolver.resolve(
    "SOURCE",
    { id: "USER_ID" },
    {
      appsyncErrors: []
    },
    {
      fieldNodes: []
    }
  );

  expect(result).toMatchInlineSnapshot(`
    {
      "id": "USER_ID",
      "name": "CREATED_AT_INFRASTRUCTURE_DEPLOY_TIME",
    }
  `);
});

To be perfectly honest, I'm not a massive fan of this testing technique. My biggest problem is the amount of effort required to make the test work. In addition to pulling additional dependencies into my project, I have to either set up a local mock of a given AWS service or deploy a given service before the test run.

Testing VTL templates by making GraphQL requests

And the last testing strategy I've used are the end-to-end tests. Instead of rendering the VTL template or using simulation, one could fire a GraphQL request to the API endpoint and assert the result. These tests usually run for much longer than the ones I've previously talked about but the confidence you gain that your system is working by utilizing end-to-end tests is massive.

The test body is tiny. We shift the complexity away from the test setup to the infrastructure configuration. To use this testing technique effectively, you must already have all the cloud resources related to the AWS AppSync deployed.

You can find the full source code for this testing strategy here.

import { gql, request } from "graphql-request";

test("template test", async () => {
  const query = gql`
    query {
      getUser(id: "USER_ID") {
        id
        name
      }
    }
  `;
  const response = await request(
    endpointUrl,
    query,
    {},
    {
      "x-api-key": endpointAPIkey
    }
  );
  expect(response).toMatchInlineSnapshot(`
    {
      "getUser": {
        "id": "USER_ID",
        "name": "CREATED_AT_INFRASTRUCTURE_DEPLOY_TIME",
      },
    }
  `);
});

This testing strategy is my go-to when I want to exercise the whole system I'm working on.

Closing words

I've used these four AWS AppSync VTL testing strategies with varying degrees of success. Hopefully, after reading this article, you found a technique that fits your needs.

I'm always open to feedback and learning new things. If you are aware of a different way of, directly or indirectly, testing VTL templates – please let me know!

Consider following me on Twitter – @wm.matuszewski if you wish to see more serverless content on your timeline.

Thank you for your precious time.

Two ways to directly integrate AWS Lambda function with Amazon API Gateway

Wojciech Matuszewski — Sun, 17 Jul 2022 04:04:46 +0000

Since the inception of the concept of "serverless", the combination of Amazon API Gateway and AWS Lambda function has become one of the most popular serverless patterns on AWS.

With the advent of deployment frameworks like AWS CDK or AWS SAM, it has never been easier, using IaC, to deploy this pattern for the world to see.

As with many things in programming and computer science, there are often multiple ways to do something. The integration of API Gateway with the Amazon Lambda function is no different – one could use the proxy and non-proxy integration type.

This blog post will discuss the difference between the proxy and non-proxy integration and in which situations you might want to use a given integration method.

The proxy and non-proxy are used here only in the context of the integration type and have nothing to do with service proxies or proxy resources.

The proxy integration

I would argue that the proxy integration type is the most commonly deployed one. Partly because, to my knowledge, it is the default one that most of the deployment frameworks utilize, and partly because it is relatively light on the configuration side of things.

The following is a simplified diagram of the API Gateway integration with the AWS Lambda function via the proxy type.

Note that all the diagrams in this article omit concepts related to authorization and authentication as they are not relevant to the ideas presented in the text.

With the proxy integration, you lose the ability to perform data manipulation directly via the API Gateway mapping templates.
- Since you cannot use the mapping templates and the VTL, your function code might be a bit more complex than the non-proxy counterpart.
Very light on the configuration side of things. The IaC declaration is relatively easy to maintain.
You still able to take advantage of the API Gateway validation models. A feature that, in my humble opinion, is underutilized.

Before using the API Gateway validation models, consider reading this article as it surfaces some of the issues you might encounter.

For completeness' sake, here is the minimal configuration required to integrate the API Gateway with an AWS Lambda function using AWS CDK and TypeScript.

const api = new aws_apigateway.RestApi(this, "API", {});

const handler = new aws_lambda_nodejs.NodejsFunction(this, "Handler", {
  entry: join(__dirname, "./handler.ts")
});

api.root.addMethod("GET", new aws_apigateway.LambdaIntegration(handler));

Pretty minimal, right? Now, onto the non-proxy integration type.

The non-proxy integration type

It took me a while to familiarize myself with some of the options the API Gateway REST API exposes. Looking back, I've allocated a sizeable portion of that time to understand how one could utilize the mapping templates and the non-proxy integration in the real world. There are a LOT of knobs one could tweak in this type of integration.

When using the non-proxy integration type, you have to specify the response mapping template. If you do not, the API Gateway will fail with "Internal Server Error".
When writing API Gateway mapping templates you do NOT have the ability to use features that you might have used in AWS AppSync resolver templates. You should consider the VTL in API Gateway and VTL in AppSync resolvers as two separate entities.

Here is the most minimal AWS CDK code snippet to declare an API Gateway route backed by non-proxy integration I could come up with.

const api = new aws_apigateway.RestApi(this, "API2", {});

const handler = new aws_lambda_nodejs.NodejsFunction(this, "Handler", {
  entry: join(__dirname, "./handler.ts")
});

const nonProxyIntegrationResource = api.root.addResource("non-proxy");

nonProxyIntegrationResource.addMethod(
  "GET",
  new aws_apigateway.LambdaIntegration(handler, {
    proxy: false,
    /** Optional */
    requestTemplates: {
      /**
       * This is what the Lambda function is invoked with.
       */
      "application/json": '{"foo": "bar"}'
    },
    /**
     * Required
     */
    integrationResponses: [
      {
        statusCode: "200",
        responseTemplates: {
          "application/json": "$input.json('$.body')"
        }
      }
    ]
  }),
  {
    /**
     * Required
     */
    methodResponses: [
      {
        statusCode: "200"
      }
    ]
  }
);

My recommendation

When integrating API Gateway with AWS Lambda function, consider using the proxy integration as the default integration type, especially if you are new to the service or are working with a team yet to familiarize themselves with API Gateway and AWS Lambda.
Consider using the non-proxy integration for massaging the input data and precomputing given properties. Consult with your team on this one. It all depends on the use case.
Take some time to learn the non-proxy integration as the notion of mapping templates and other closely related settings are what drives the API Gateway direct service integrations.

Closing worlds

I find it very valuable to dive deep into a given AWS service, especially when I'm unfamiliar with a particular way of configuring it – I've mainly used the proxy integration throughout, albeit not that long, my career. I encourage you to do a similar and write about your findings. This way, the whole community can benefit and learn together!

For more serverless content, consider following me on Twitter - @wm_matuszewski

And as always, thank you for your precious time.

Better error messages for non-existing API resources with Amazon API Gateway

Wojciech Matuszewski — Sun, 26 Jun 2022 13:19:53 +0000

One of the most popular serverless services in the AWS offering is the Amazon API Gateway. With three flavors of APIs to choose from – REST, HTTP, and WebSocket, the service is an essential piece of the AWS serverless ecosystem.

This blog post will look at the REST and HTTP versions of the Amazon API Gateway APIs and how to ensure the path or resource not found responses contain messages which are helpful to the end-user.

You can find the code from this article in this GitHub repository.

The problem

We all make mistakes – after all, we are only human. When interacting with APIs, one of the mistakes one can make is to try to request a resource that does not exist (either the path or the method is incorrect).

In an ideal world, the response generated from the API would help the user quickly resolve their issue – in our case, amend the request to use the correct resource/method combination.

In the case of the Amazon API Gateway, the default behavior for handling such situations is not ideal, especially when it comes to the REST APIs. Let us look at that flavor of the Amazon API Gateway next.

Unknown resource response in the context of Amazon API Gateway REST API

By default, if the request resource/method combination is incorrect, the API Gateway REST API will respond with the following payload with a statusCode of 403.

{ "message": "Missing Authentication Token" }

Ouch, there is room for improvement here. In fact, even the official documentation states that this behavior is confusing.

If the official documentation states that the default response is confusing, that begs the question, why have such a response be the default in the first place?

To my best knowledge, we can significantly improve the API ergonomics in this area by either modifying Amazon API Gateway gateway responses or by having a "greedy" route with ANY method that acts as a fallback for such requests. Since I'm biased toward solutions that involve configuration, let us first start with gateway responses.

Modifying gateway responses

The gateway responses define what happens if the API Gateway fails to process an incoming request.

In our particular scenario, we are interested in amending the MISSING_AUTHENTICATION_TOKEN response type – that is the response type returned to the end-user when the method/resource combination of the request to the REST API is incorrect.

The following is an AWS CDK TypeScript snippet that re-defines the gateway response in question.

const restAPI = new aws_apigateway.RestApi(this, "RestAPI", {
  /**
   * Configuration...
   */
});

restAPI.addGatewayResponse("PathOrMethodMaybeFound", {
  type: aws_apigateway.ResponseType.MISSING_AUTHENTICATION_TOKEN,
  templates: {
    "application/json": `{"message": "$context.error.message", "hint": "Is the method/path combination correct?"}`
  }
});

Since the MISSING_AUTHENTICATION_TOKEN response type could also be triggered when the authentication token is missing (the method/resource combination is correct), I've opted to leave the original error message intact.
I did not employ any "smartness" to the template since one cannot use any VTL programming constructs in the gateway responses templates – API Gateway will ignore them. I could have tailored the message even more, using conditional statements, but in the gateway responses templates, I'm constrained to only using variables.

Now, If I were to make the request to the API and mistype the path or use a wrong method (for example, POST instead of GET), I would get the following response.

{
  "message": "Missing Authentication Token",
  "hint": "Is the method/path combination correct?"
}

An improvement, but we can do better. Let us look at the second way I mentioned earlier – using a "greedy" path with the ANY method.

Deploying a "greedy" route with `ANY` method

As an alternative to modifying the gateway responses, one can deploy a "greedy" route with the ANY method. Suppose the API Gateway invokes the integration backing that route. In that case, we can be confident that none of our other routes match the request instead of lacking the authentication token (if our API requires authorization).

Note
In some cases, developers might use the ANY method with a "greedy" route for monolithic AWS Lambda deployments (think running express on AWS Lambda. While I'm not a fan of such practice, I also understand that it might work well for some. If that is the case for you, you should consider tackling the issue I'm describing using your framework of choice.

Greedy route backed by AWS Lambda function

This is, arguably, a more accessible, from the IaC configuration standpoint, variant to pull off. The "greedy" ANY route is backed by an AWS Lambda function. Since we have the power of a programing language at our disposal, one has the opportunity to tailor the "method/path does not exist on this API" message to a given API user.

Note
AWS Lambda function is not the only service the API Gateway can integrate. In this scenario, AWS Step Function Express workflows would also do the trick.

The following is an AWS CDK definition of the "greedy" ANY route backed by the AWS Lambda function.

const restAPI = new aws_apigateway.RestApi(this, "RestAPI", {
  /**
   * Configuration...
   */
});

const anyAPIHandler = new aws_lambda_nodejs.NodejsFunction(
  this,
  "AnyAPIHandler",
  {
    entry: join(__dirname, "./handler.ts")
  }
);

restAPI.root
  .addResource("{proxy+}")
  .addMethod("ANY", new aws_apigateway.LambdaIntegration(anyAPIHandler));

And here is the handler.

import { APIGatewayProxyHandler } from "aws-lambda";

export const handler: APIGatewayProxyHandler = async () => {
  return {
    statusCode: 404,
    body: "Method/Path combination not found. Please check the documentation."
  };
};

Now, If I were to call the API and use either an incorrect method or path (or both) instead of the generic Missing Authentication Token message, the response would be the one I've encoded in the AWS Lambda.

Method/Path combination not found. Please check the documentation.

One important thing to remember when going this route is to ensure that we treat the "greedy" route AWS Lambda function just like any other AWS Lambda functions in our infrastructure. This means adding throttling limits and so on (if applicable). The last thing you want is a denial of wallet attack just because you forgot to add authorization to the "greedy" endpoint.

Greedy route backend by Mock Integration

The second, and the option I'm more in favor of, is to ditch the AWS Lambda function and use the Amazon API Gateway Mock Integration as the "brains" that creates the response for the "greedy" ANY route.

The following is an AWS CDK definition of the "greedy" ANY route backed by Amazon API Gateway Mock Integration.

const restAPI = new aws_apigateway.RestApi(this, "RestAPI", {
  /**
   * Configuration...
   */
});

restAPI.root.addResource("{proxy+}").addMethod(
  "ANY",
  new aws_apigateway.MockIntegration({
    passthroughBehavior: aws_apigateway.PassthroughBehavior.NEVER,
    requestTemplates: {
      "application/json": `{"statusCode": 404}`
    },
    integrationResponses: [
      {
        statusCode: "404",
        responseTemplates: {
          "application/json": `"Method/Path combination not found. Please check the documentation."`
        }
      }
    ]
  }),
  {
    methodResponses: [{ statusCode: "404" }]
  }
);

The main benefit of using the mock integration as opposed to the AWS Lambda function is that you do not have to pay for AWS Lambda function invocations. The main drawback is the fact that now you have to deal with mapping templates, VTL, and relatively complex IaC configuration.

Remember that mock integration templates allow you to perform conditional logic – something the gateway responses templates lacked (worth noting that you are still unable to harness the full power of the VTL, like in the case of AWS AppSync resolver templates).

Unknown resource response in the context of Amazon API Gateway HTTP API

By default, if the request resource/method combination is incorrect, the API Gateway HTTP API will respond with the following payload with a statusCode of 404.

{
  "message": "Not Found"
}

Not bad! An improvement over the REST API counterpart. You might wish to improve the message even further – in that case, let us look at how you could do that.

Greedy route backed by AWS Lambda function

Since the HTTP flavor of the API Gateway API does not expose any way to manipulate request/response templates, to my best knowledge, the only way to change to configure this behavior is to have some compute service behind a "greedy" ANY route. Like Amazon API Gateway REST API, one might use the AWS Step Function Express Workflows or the AWS Lambda function to achieve the desired result.

For completeness, here is the AWS CDK definition of a "greedy" ANY route for API Gateway HTTP API backed by AWS Lambda function.

const httpAPI = new aws_apigatewayv2_alpha.HttpApi(this, "HttpAPI", {
  /**
   * Configuration...
   */
});

const greedyPathHandler = new aws_lambda_nodejs.NodejsFunction(
  this,
  "GreedyPathHandler",
  {
    entry: join(__dirname, "./greedy/handler.ts")
  }
);

httpAPI.addRoutes({
  integration: new aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration(
    "GreedyPathIntegration",
    greedyPathHandler
  ),
  path: "/{proxy+}",
  methods: [aws_apigatewayv2_alpha.HttpMethod.ANY]
});

And here is the handler code.

import { APIGatewayProxyHandlerV2 } from "aws-lambda";

export const handler: APIGatewayProxyHandlerV2 = async () => {
  return {
    statusCode: 404,
    body: "Method/Path combination not found. Please check the documentation."
  };
};

Like in the case of the REST counterpart, there is not that much to it. I have intentionally did not add any logic to my handlers as the logic might be highly dependent on your specific case.

Closing words

We can, and I would argue we should change the default response the API Gateway returns when the user makes a request to our API with an incorrect method/path combination, especially for the REST flavor of the API Gateway APIs. The confusion one single error message could cause cannot be underestimated.

Consider following me on Twitter – @wm_matuszewski if you are interested in similar content like this blog post.

Thank you for your precious time.

Amazon EventBridge archive and ordered replay of events

Wojciech Matuszewski — Sat, 11 Jun 2022 11:14:49 +0000

Edited on 17.06.2022

With many serverless event-first services available, utilizing events in application backends has become very popular. Those asynchronous workflows are often more cost-efficient and resilient than their synchronous counterparts.

In AWS, one service that usually works great for event-driven workflows is the Amazon EventBridge. With a rich integration with other AWS services, extensive event filtering capabilities, an option to archive events, and the ability to send HTTP requests via API destinations, it seemingly has everything you would need to place it in the core of your application.

In this blog post, I would like to zoom in on the event archival capabilities of the Amazon EventBridge, mainly how to ensure that the replayed events flow through your application in order they have been sent to the event bus.

About the event archive

If enabled, Amazon EventBridge will save all the events that flow through the event bus into the so-called event archive. You can configure the archive to have a given retention period or only to save events that match a specific shape. You can define multiple event archives per event bus.

The following is a high-level diagram of an EventBridge event bus that utilizes the event archive for event archival.

One of the more exciting capabilities of the event archive is the ability to replay events from the archive. As you can imagine, if something goes wrong in your system or you have to backfill historical data, this capability comes in handy.

The archive event replay

In the context of event replay, there are a few essential things to consider – the period from which the events should be replayed and the fact that EventBridge does not guarantee the order of events when replaying them to a given target.

In a perfect world, the systems we create should withstand the out-of-order delivery of events and proceed with the workflows as if the order of events is a non-factor, but sometimes it is impossible to do so. Replaying events in the order, they arrived on the event bus could be helpful while developing new features or trying to debug a particular issue.

With the help of AWS Step Functions, it is possible to force the "correct" order of the events back to our system when the event replay is active. Let us take a look at how next.

Ordered replay with the help of AWS Step Functions

The following is an example structure of an EventBridge event replayed by the archive.

{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail-type": "EC2 Instance State-change Notification",
  "source": "aws.ec2",
  "account": "111122223333",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0"
  ],
  "replay-name": "myEventReplay",
  "detail": {
    "instance-id": "i-1234567890abcdef0",
    "state": "terminated"
  }
}

To distinguish between a replayed event and the regular event, the EventBridge archive adds the replay-name property to the event. This property is handy for writing filtering rules that only allow the replayed event to pass onto the underlying integration. In our particular situation, the time property is the one that will allow us artificially enforce order on the replayed events.

The EventBridge service automatically adds the time property to the event. You do not have (but you definitely can) to supply the timestamp in the event body.

By computing the difference between the current timestamp and the timestamp of the event, we can deduce the amount of time the event has to wait before being forwarded to the target. If we apply this heuristic to all replayed events, we effectively enforce order based on the event time property. The work of computing the time difference and the waiting will be done by a middleman service – in our case, an AWS Step Function.

Instead of forwarding the events to the original destination, we deploy a middle-man, the AWS Step Function, to enforce the order and deliver the events to the original destination. Inside the Step Function workflow, we first calculate the time difference between the replay start timestamp and the event time property, then, using the Wait state, we idle for that period and finally forward the event to its original destination.

The following is a simplified design of the AWS Step Function step machine that takes care of ordering logic.

You can find an example GitHub repository where I've implemented the ordered replay here.

Note that we cannot publish the event back to the event bus. If we were to do that , we would lose the ordering guarantee enforced by our Step Function – EventBridge does not guarantee strict ordering of events when invoking targets.

You could also forward the event to Amazon DynamoDB (and then use DynamoDB Streams) or Amazon Kinesis Data Streams and read the events from those services. Both of these services have ordering guarantees.

Cost considerations

Enforcing ordering for the replayed events is not free. In fact, it can get quite expensive for large amounts of events. Since we have to invoke an AWS Step Functions state machine for each replayed event, we incur additional charges subject to the AWS Step Functions pricing dimensions. Personally, I would only use the feature of ordered replay in development, where I control the volume of the incoming events.

Closing words

I hope you found this little nugget of AWS knowledge as helpful as I did. I encourage you to explore ways to enforce event order in other parts of your system, if only for training purposes – I guarantee you will learn something new and exciting.

For more AWS serverless content, consider following me on Twitter – @wm_matuszewski.

And as always, thank you for your precious time!

Processing large payloads with Amazon API Gateway asynchronously

Wojciech Matuszewski — Wed, 27 Apr 2022 06:09:09 +0000

In my previous article, I've talked about Synchronous AWS Lambda & Amazon API Gateway limits and what to do about them.

As stated in the blog post, the ultimate solution for a "big payload" problem is making the architecture asynchronous. Let us then zoom in on the aspect of asynchronous communication in the Amazon API Gateway service context and build a serverless architecture based on the notion of "pending work".

Problem refresher

The following illustrates the Amazon API Gateway & AWS Lambda payload size problem I've touched about in the previous article.

No matter how hard you try, you unable to synchronously send more than 6 MB of data to your AWS Lambda function. The limit can seriously mess with your significant data-processing needs.

Luckily there are ways to process much more data via AWS Lambda using asynchronous workflows.

The starting line - pushing the data to the storage layer

Suppose you were to ask the community about the potential solution to this problem. In that case, I wager that the most common answer would be to use Amazon S3 as the data storage layer and utilize presigned URLs to push the data to Amazon S3 storage.

While very universal, the flow of presigned URL to Amazon S3 can be nuanced, especially since one can create the presigned URL in two ways.

Describing these would make this article a bit too long for my liking, so I'm going to defer you to this great article by my colleague Zac Charles which did the topic much more justice than I could ever do.

I have the data in Amazon S3. Now what?

Before processing the data, our system must know whether the client used the presigned URL to push the data to the storage layer. To my best knowledge, there are two options we can pursue here (please let me know if there are other ways to go about it, I'm very keen to learn!)

I omitted the AWS CloudTrail to Amazon EventBridge flow on purpose as I think engineers should favor the direct integration with AWS EventBridge instead.

S3 Event Notifications

The Amazon S3 event notifications, till recently, was a de facto way of knowing whether data landed into Amazon S3.

While sufficient for most use-cases, the feature is not without its problems, the biggest of which, I would argue, are the misunderstandings around event filtering and IaC implementation.

The most essential thing to have in mind when it comes to event filtering is that you cannot use wildcards for prefix or suffix matching in your filtering rules. There are more things to consider, though. If you plan to use the filtering feature of S3 event notifications, I strongly encourage you to read this documentation page thoroughly.

On the IaC side of things, know that, in creating the AWS CloudFormation template, you might end up with circular dependency problems. Deployment frameworks like AWS CDK will make sure that should never happen. However, I still think you should be aware of this potential problem, even if you use deployment frameworks.

EventBridge events

In late 2021, AWS announced Amazon EventBridge support for S3 Event Notifications. The announcement had a warm welcome in the serverless community as EventBridge integration solves most of the "native" S3 event notifications problems.

Utilizing Amazon EventBridge for S3 events gives you more extensive integration surface area (you can forward the event to more targets) and better filtering capabilities (one of the strong points of Amazon EventBridge). The integration is not without its problems, though.

For starters, the events are sent to the "default" bus, and using EventBridge might be more costly for high event volumes. To learn more about different caveats, consider giving this great article a read.

Munching on the data

You have the data in the Amazon S3 storage, and you have a way to notify your system about that fact. Now what? How could you process the data and yield the result back to the user?

Given the nature of AWS, for better or worse, there are multiple scenarios one can move forward. We will start from the "simplest" architecture and move our way up to deploying an orchestrator with shared, high-speed data access.

Processing with a dedicated AWS Lambda

The AWS Lambda service is often called a "swiss knife" of serverless. In most cases, processing the data in-memory within the AWS Lambda is sufficient.

The only limitations are your imagination and the AWS Lambda service timeout – the 15-minute maximum function runtime. Please note that in this setup, your AWS Lambda function must spend some of that time downloading the object.

I'm vague about saving the results of the performed work on purpose as it is very use-case dependant. You might want to keep the outcome of your work back on Amazon S3 or add an entry to a database. Up to you.

But what if that 15-minute timeout limitation is a thorn at your side? What if the process within the compute layer of the solution is complex and would benefit from splitting it into multiple chunks? If that is the case, keep on reading, we are going to be talking about AWS Step Functions next.

Processing with Amazon StepFunctions

If the compute process within the AWS Lambda we looked at previously is complex or takes more time than the hard timeout limit, the AWS Step Functions might be just the service you need.

Take a note of the number of times the code within your Step Function definition needs to download the object from Amazon S3 storage (the illustration being an example, of course). Depending on the workload, that number may vary, but no matter the workflow, after a certain number threshold, it does feel "wasteful" for me to have to download the object again and again.

Keep in mind that AWS Step Functions maximum payload size is 256KB. That is why you have to download the object repeatedly whenever you need access to the object.

You could implement in-memory caching in your AWS Lambda function, but that technique only applies to a single AWS Lambda and is tied to its container lifecycle.

Depending on the requirements and constraints, I like to use Amazon EFS integration with AWS Lambda in such situations. Amazon EFS allows me to have a storage layer shared by all AWS Lambda functions that partake in the workflow. Let us look into that next.

Shared AWS Lambda storage

Before starting, understand that using Amazon EFS with AWS Lambda functions requires Amazon VPC. For some workflows, this fact does not change anything. For others, it does. I strongly advocate keeping an open mind (I know some people from serverless community despise VPCs) and evaluating the architecture according to your business needs.

This architecture is, arguably, quite complex. You have to consider some networking concerns. If your object is quite large and downloading it takes a lot of time, I would advise you to look into this architecture – VPCs do not bite!

Suppose you yearn for a practical example, here is a sample architecture I've built for parsing media files. It utilizes Amazon EFS for fast access to that video file across all AWS Lambda functions involved in the AWS Step Function workflow.

Yielding back to the user

With the compute part behind us, it is time to see how we might notify the user that our system processed the object and that the results are available.

No matter what kind of solution you choose to notify the user about the result, the state of the work has to be saved somewhere. Is the work pending? Is it finished? Maybe an error occurred?

Keeping track of the work status

The following is a diagram of an example architecture that keeps track of the status of the performed work.

The database sends CDC events to the system. We could use these CDC events to notify the user about the work progress in real-time! (although that might not be necessary, in most scenarios, polling for the results by the client is sufficient).

In my humble opinion, the key:value nature of the "work progress" data makes the Amazon DynamoDB a perfect choice for the database component (nothing is stopping you from using RDS, which also supports CDC events, which themselves are optional here).

Notifying the user about the results

The last piece of the puzzle is making sure the user has a way to be notified (or retrieve) the status of the request.

Usually, in such situations, we have two options to consider – either we implement an API in which the user is going to pool for the updates, or we push the status directly to the user.

The first option, pooling, would look similar to the following diagram in terms of architecture.

This architecture is sufficient up to a particular scale. The architecture must scale according to the pollers hitting the API. The more pollers actively engage with the API, the higher the chance of throttling or other issues (refer to the Thundering herd problem).

In higher throughput scenarios, one might want to replace the polling behavior with a push model based on WebSockets. For AWS-specific implementations, I would recommend looking at Amazon API Gateway WebSocket support or AWS IoT Core WebSockets for broadcast-like use-cases.

I've written an article solely focused on serverless WebSockets on AWS. You can find the blog post here.

Closing words

And that is the end of our journey. We have looked at how to get the large payload into our system, process it, and respond to the user. Implementing this architecture is not a small feat, and I hope you found the walkthrough helpful.

For more AWS serverless content, consider following me on Twitter - @wm_matuszewski

Thank you for your precious time.

Synchronous AWS Lambda & Amazon API Gateway limits and what to do about them

Wojciech Matuszewski — Sun, 10 Apr 2022 15:27:20 +0000

In the era of AWS serverless computing, there are two services that, one would argue, are used together very frequently – the AWS Lambda and Amazon API Gateway.

Combining these two services allows developers to create APIs that serve millions of customers daily, scaling elastically depending on the workload. It is not all sunshine and rainbows, though.

In your AWS serverless journey, you will, at some point, end up faced with inbound payload size limits, most likely when interacting with AWS Lambda and Amazon API Gateway.

In this article, I will lay out different alternatives you might want to consider when you end up in a situation where Amazon API Gateway and AWS Lambda inbound payload size limits constrain your synchronous workflow.

The typical AWS Lambda Amazon API Gateway synchronous architecture

Let us examine what, I would argue, is the most typical synchronous architecture that combines the AWS Lambda and Amazon API Gateway services.

There is nothing wrong with it. I would argue that the architecture is just fine for the vast majority of use-cases, especially for creating API routes in a serverless way.

The payload size difference problem

It is crucial to notice one critical detail – the difference in maximum payload size of Amazon API Gateway and AWS Lambda. This delta might be frustrating and confusing. If you send a payload bigger than 6 MB, Amazon API Gateway will happily accept it and fail when invoking the AWS Lambda.

I'm not aware of any other solution to this problem than to restrict the payload size in the user-land. Amazon API Gateway can validate the request via JSONSchema, but, to my best knowledge, you cannot validate the size of the payload.

One might also look into AWS WAF, where request body validation is possible, but only for payloads of size up to 8192 bytes.

I need to process bigger payloads synchronously

If you cannot afford or do not want to re-architect your solution to be asynchronous (which would, in theory, enable you to process potentially unbound payload sizes), there is one quick-win type of change you can make that could fit your use case. I'm referring to compressing the payload before sending it to Amazon API Gateway.

Notice that the Amazon API Gateway is NOT decompressing the payload. While the service in question has this capability, it is the opposite of what we want to do. If we were to decompress at the Amazon API Gateway level, the service would try to send a 15 MB payload to AWS Lambda, resulting in an error.

I need more time to process my synchronous request

By compressing the payload in the user-land, you can, to some extent, get around the 6 MB payload size limit. What about the 30-second response timeout?

If your payload is large, you might need more time to process the request synchronously. Luckily, with some changes to our infrastructure, it is possible to bump the response timeout to 15 minutes (the AWS Lambda timeout).

AWS Lambda function URL

A very recent addition to the AWS Lambda-related family of features, the AWS Lambda URL feature might be just what you need. We ditch Amazon API Gateway in this architecture since it was the primary response timeout bottleneck.

Keep in mind that ditching Amazon API Gateway could have enormous consequences depending on the use case. Amazon API Gateway is rich in features like validation, caching, usage keys/plans, etc. By utilizing the AWS Lambda URL feature, we gain the longer synchronous request timeout but lose all the niceties of Amazon API Gateway. Keep this in mind while reaching out for this feature.

AWS Lambda fronted with Application Load Balancer

Instead of using Amazon API Gateway for fronting the AWS Lambda function, we can use the Application Load Balancer. Like in the case of the AWS Lambda function URL feature, you will lose many of the Amazon API Gateway features but gain a much longer response timeout.

In some architectures, with high-enough traffic and small sizes of the requests, fronting AWS Lambda with ALB is much more cost-effective than using Amazon API Gateway. You can read more about it in this article.

Note about "a lot" you see for Maximum payload size and Response timeout for ALB. I could not find quotas referring to these variables in the AWS documentation. If you know the limits for those, please let me know!

The ultimate solution

If the 6 MB payload size is not enough, even with compressed payloads, and the 15 minutes of response timeout is too short, consider re-architecting for asynchronous communication. In the end waiting for responses is expensive and reaching out for tools like AWS Lambda function URL or fronting AWS Lambda with ALB to go around the limits might be a sign that asynchronous processing would be the right fit for your use case.

AWS exposes various services to facilitate such workloads, ranging from AWS Step Functions to AWS Batch or AWS Fargate and storing data on Amazon S3. This article would be too long if I were to describe all the possible asynchronous ways to get around the 6 MB AWS Lambda payload size limit – do not worry, that article is next on my list!

Until that, I will leave you with this great article written by Yan Cui as a sneak-peek of what I will be covering.

Closing words

I hope you find the approaches described in this blog post helpful. Remember that all limits have a reason, be it related to technical or hardware limitations. Some might seem off-putting, like the 6 MB AWS Lambda payload size limit. Still, I do wholeheartedly believe that some of them, most likely by accident, push your architecture more towards asynchronous communication, which in most cases is a good thing!

Consider following me on Twitter – @wm_matuszewski so that you do not miss the follow-up article (and more AWS serverless-related content).

Thank you for your valuable time.

Minimal AWS SSO setup for personal AWS development

Wojciech Matuszewski — Thu, 17 Mar 2022 13:48:29 +0000

I would argue that the AWS SSO is a "hidden gem" of AWS service. With AWS SSO, you do not have to deal with AWS IAM Users and long-lived credentials.

I have a suspicion that there is a lot of misinformation about AWS SSO out there. Some common misconceptions I've heard are:

"You need OKTA to use AWS SSO".
"You have to use AWS Control Tower to use AWS SSO".
"The AWS SSO setup is complex, and only experienced AWS developers can utilize it".

None of the above are correct. I'm using AWS SSO for my personal AWS stuff, and I've never had to interact with AWS Control Tower or OKTA in any shape or form. It is entirely possible to have AWS SSO working with a bare minimum AWS Organizations and AWS IAM configuration, and this blog post will show you how.

The why of AWS SSO for AWS development

"Why should I bother using/taking my time to learn AWS SSO?" is a valid question, especially when time is the scarcest resource we have. My arguments in favor of learning/using AWS SSO follow.

The first point is that you do not have to worry about long-lived credentials for your AWS accounts anymore (except the root one), which drastically improves the security posture of your environment.

The second point is that if set up correctly, AWS SSO is extremely convenient to use. In my current personal dev workflow, all I do to obtain a time-bound AWS session is type one short command and pick the right account/IAM role combo.

The third point is that knowing just a little about AWS SSO could help you in your day job where you most likely use OKTA (or any other identity provider) with AWS SSO (If you are not, and the configuration you have works for you, more power to you). Someone had to know how to connect the services in the first place. That someone might be you :).

Setting the stage

Before we dive into the configuration itself, I think it is vital to set expectations regarding what I'm about to show you.

We will only use the bare minimum of AWS services/features for simplicity.
I would not recommend using this setup for your company's AWS SSO story. Look into combining OrgFormation / AWS Control Tower with AWS SSO for that.
I've tried to keep the complexity to a minimum so that people who are not familiar with AWS (for example, me from 3 years ago) can go through the listed steps without significant issues.
You will be using the AWS console through configuration steps. Creating an IaC template to automate the setup is beyond this blog post's scope.

The end goal

After you have completed the configuration steps, you will be able to do the following:

Login into a given AWS account console with a single click via the AWS SSO user portal. Here is a screenshot of my AWS SSO portal dashboard.

Assume credentials for a given AWS IAM role in a given AWS account for CLI usage with a single command.

Interested? Let us get started with the configuration.

The configuration

The prerequisites

AWS CLI installed.

The steps

1 - If you do not have an AWS account already, create one. There are many great resources on how to do just that, for example, this guide by A Cloud Guru.

I would strongly recommend this AWS account be used only for AWS SSO / AWS Organizations set up. Consider not running any deployments on this account.

2‏‏‎‏‏‎ - Pick the region you wish your AWS SSO configuration to be deployed into. To my best knowledge, the AWS SSO is a region-bound service. For demonstration purposes I will be using the eu-west-1 region throughout this blog post.

3 - In the search bar, look for AWS SSO service. The sso search phrase should do. Click on the service tile to go to its dashboard.

4 - After landing on the AWS SSO dashboard, assuming you do not have it enabled in any other AWS regions, you should be presented with a landing page that contains a big "Enable AWS SSO" button – click on it.

5 - Assuming you do not have any AWS Organizations created on this account, the AWS console will greet you with the popup to create one. Click on the "Create AWS Organization" button.

If you are new to AWS and have no idea what AWS Organizations are, consider them a way of creating multiple AWS accounts with ease. You do NOT have to know a lot about them to configure AWS SSO. Consult the AWS Organizations documentation for more information.

After waiting a bit for the AWS console to create the AWS Organization, you will be redirected to the AWS SSO service dashboard.

6 - Next up, we need to create an AWS account we will be SSOing into (and assuming given policy in that account, more on that later). As I eluded earlier, we will be using AWS Organizations to manage AWS accounts. AWS SSO service created an AWS Organization for us in the previous step. Go to the created AWS Organization by searching for the service name.

7 - Next up, click on the "Add an AWS account" button.

We want to create a new AWS account.
Pick whatever name you want. My recommendation would be to add the "Development" suffix to the name to signal that this is the account I'm using for playing around – i.e., it contains deployed AWS resources.
I would leave the "IAM role name" default value. The permission model of AWS Organizations goes beyond the scope of this article.
I would not bother with any Tags for personal-only use.

8 - After you have successfully created the AWS account and set the password (the email you have received from AWS), it is time to go back to the AWS SSO dashboard to finish the configuration.

9 - Next, we have to create a permission set (a scary name for a collection of IAM Policies) you will be acquiring when SSOing into a given AWS account. This part is what allows us to ditch the long-lived credentials. Switch to the "Permission sets" tab and click on "Create permission set".

10 - Now comes the permission set creation wizard. The following are my recommendations for each step.

For the "Type" step, I recommend sticking with "Use an existing job function policy" unless you are adept at writing AWS IAM Policies.
For the "Detail" step, I recommend picking the "PowerUserAccess" policy. Make sure you understand what a given policy grants.
When it comes to the "Tags" step, I would not bother setting them for personal use, but I leave that decision up to you.

Tags are essential in a work environment where multiple teams operate on different resources.

11 - Next, we have to create a user that can sign in to an AWS SSO user portal. We will be associating the user with a permission set and AWS account later. To create an AWS SSO user, navigate to the "Users" tab and click the "Add user" button.

Make sure to save the username you specified in the "Specify user details" step – you will need it later on.
I would suggest omitting creating groups for simplicity's sake, but that is up to you.
After you have created the user, I would strongly consider adding an MFA for that user.

12 - With the AWS SSO user created, it is time to tie everything we have been configuring together. Navigate to the "AWS accounts" tab. We will be associating the AWS SSO user with a given permission set and an AWS account.

As with AWS SSO users, the AWS console uses a wizard-style form for this configuration. The following are screenshots that should help you navigate the steps.

Testing

Okay, we have done everything we need to make the AWS SSO work in its simplest, from – having access to an AWS account without using long-lived credentials. Now it is time to test our setup to ensure everything is working correctly. Let us start with AWS console access using AWS SSO.

AWS console access through AWS SSO

After finishing step 12 from the The steps section of this article, navigate to the AWS SSO "Dashboard" tab and copy the "User portal URL"

After signing in successfully (use credentials for AWS SSO user and NOT the AWS account), you should be able to pick the AWS account and the permission set combo, we have configured in the previous section.

The "Management console" link should redirect you to a given AWS account as a federated user scoped to a given permission set.

Consider bookmarking the "User portal URL" for easy access.

AWS CLI access

With the AWS console access out of the way, let us focus on AWS CLI access and getting the AWS credentials from the AWS SSO session.

Luckily for us, the AWS CLI already has excellent support for AWS SSO.

I find the official AWS CLI SSO configuration guide to be a very well-put resource on how to configure the AWS CLI with AWS SSO correctly.

After configuring the CLI, I recommend you look into aws-export-credentials for exporting the credentials as environment variables across different shells you use.

Possible improvements

One improvement that one can make to the configuration I presented is to encapsulate it within reproducible IaC deployment. This goes beyond my area of expertise as I'm more of an application developer that never had to deal with creating IaC code for AWS Organizations and AWS SSO.

If you feel like this is what you want to do next – great! I would start by reading on either AWS ControlTower or OrgFormation.

Closing words

I hope you find this blog post helpful as I would a couple of months prior when I had no idea what I was doing going through the process I described in the "The steps" section.

We configured AWS SSO for our development account with twelve steps and ditched the long-lived credentials associated with IAM Users in the process.

For more AWS / serverless content, consider following me on Twitter - @wm_matuszewski

And, as always, thank you for your valuable time.

Curious case of AWS mapping template built-in variables

Wojciech Matuszewski — Mon, 07 Mar 2022 03:26:30 +0000

Amazon API Gateway is, along with AWS Lambda, arguably, the most frequently used service in AWS serverless architectures. The service's feature spectrum, starting from rate-limiting and ending on request/response validation, is vast, making it an ideal choice for many API-driven applications.

Amongst its features, as a "Lambda-less" enthusiast, one is particularly interesting to me – the ability to create mapping templates that transform the payload or the response without the need for AWS Lambda.

This article will touch on something I've spent many hours debugging – the behavior of the built-in mapping template variables concerning Amazon REST API Gateway VTL and talk about my assumptions that turned out to be wrong.

The code in this article is sourced from this GitHub repository.

The architecture

I often find that often the understanding lies in the concrete. This blog post is no exception. The following architecture will serve as a reference in our discussions.

The user makes a request to an API frontend with Amazon API Gateway backed by a mock integration. The AWS Lambda authorizer handles authorization. The role of the mapping template is to respond with some of the Amazon API Gateway mapping template variables, mainly the ones related to authorization.

The mapping template is crucial since its structure dictates the response of the API. Let us look at it next.

The code

Luckily, to showcase the behavior I'm teasing you about, we do not need to write a lot of code. A two-liner of a mapping template and a very bare-bones AWS Lambda authorizer will do.

The following is the mapping template code.

{
    "authorizerContextUsernameProperty": "$context.authorizer.username",
    "wholeAuthorizerVariable": "$util.escapeJavaScript($context.authorizer)",
}

The mapping template declares two keys, the authorizerContextUsernameProperty which is read directly from the $context.authorizer.username and the wholeAuthorizerVariable. I'm escaping the value of $context.authorizer to avoid parsing issues.

And here is the AWS Lambda authorizer code.

export const handler: APIGatewayTokenAuthorizerHandler = async event => {
  // Assuming the user is authenticated and the token is valid

  return {
    principalId: "1",
    context: {
      username: "test-username"
    },
    policyDocument: {
      Statement: [
        {
          Action: "execute-api:Invoke",
          Effect: "Allow",
          Resource: event.methodArn
        }
      ],
      Version: "2012-10-17"
    }
  };
};

The authorizer is not doing much. It is only there to populate the context property so that we have something to work with within the mapping template.

The surprising behavior

If I were to invoke the API, what would be the output? As a reminder, the following is our response mapping template.

{
    "authorizerContextUsernameProperty": "$context.authorizer.username",
    "wholeAuthorizerVariable": "$util.escapeJavaScript($context.authorizer)",
}

When I first started looking into API Gateway mapping templates I thought that the result would look similar to the following object.

{
  "authorizerContextUsernameProperty": "test-username",
  "wholeAuthorizerVariable": "{\"username\": \"test-username\", ... OTHER_PROPERTIES }"
}

To my surprise, that was not the case. Here is the response I have received.

{
  "authorizerContextUsernameProperty": "test-username",
  "wholeAuthorizerVariable": ""
}

Even though we clearly see that $context.authorizer.username is defined, the $context.authorizer somehow is "empty". After staring at the screen trying to come up with an answer, I've decided to check the whole $context object.

{
    "authorizerContextUsernameProperty": "$context.authorizer.username",
-   "wholeAuthorizerVariable": "$util.escapeJavaScript($context.authorizer)",
+   "wholeAuthorizerVariable": "$util.escapeJavaScript($context)",
}

And the result was surprising to me as well! (I redacted some of the values).

{
  "authorizerContextUsernameProperty": "test-username",
  "wholeAuthorizerVariable": "{resourceId=w7badv0rv4, authorizer=, resourcePath=/, httpMethod=GET, extendedRequestId=OY6_EHr1DoEFSzg=, requestTime=03/Mar/2022:04:06:30 +0000, path=/prod/, accountId=XXX, protocol=HTTP/1.1, requestOverride=, stage=prod, domainPrefix=9sd680n5z5, requestTimeEpoch=1646280390676, requestId=f9393904-3014-43f6-8e24-f594db2d7604, identity=, domainName=9sd680n5z5.execute-api.eu-west-1.amazonaws.com, responseOverride=, apiId=9sd680n5z5}"
}

As you can see, the authorizer, identity, responseOverride properties are empty. Since I did not override any request and response parameters, the emptiness of responseOverride property is understandable. But what about the authorizer and identity? I can see that $context.authorizer.username contains a value, otherwise the authorizerContextUsernameProperty property would be empty.

What about AWS AppSync?

Like Amazon API Gateway, the AWS AppSync service enables developers to write VTL mapping templates to transform data. Will AWS AppSync VTL parsing logic behave differently from the one we observed in Amazon API Gateway? Let us find out.

The architecture is very similar to the one we have looked at previously. The main difference is that instead of making a GET request, we will be making a POST request since AWS AppSync exposes a GraphQL API.

The mapping template and the response

The following is the AppSync resolver request mapping template we will be working with. Check out resolver mapping template reference for Node data source documentation for more information.

This article is not about AWS AppSync itself. If you are not familiar with the service, this GitHub repository contains a lot of AWS AppSync specific resources that you can consume.

#set($payload = {
  "authorizerContextUsernameProperty": $context.identity.resolverContext.username,
  "wholeAuthorizerVariable": $util.escapeJavaScript($context.identity.resolverContext)
})

{
  "version": "2018-05-29",
  "payload": $util.toJson($payload),
}

If the mapping templates would work the same way as those declared in the context of Amazon API Gateway, the wholeAuthorizerVariable should return an empty string whenever I perform the GraphQL Query that uses this mapping template.

That is not the case. The following is the response I've got.

{
  "data": {
    "getData": {
      "authorizerContextUsernameProperty": "test-username",
      "wholeAuthorizerVariable": "{username=test-username}"
    }
  }
}

The wholeAuthorizerVariable contains the key with a variable set by AWS Lambda authorizer – much different from how our sample Amazon API Gateway mapping template behaved.

If I were to amend the mapping template the following way:

#set($payload = {
  "authorizerContextUsernameProperty": $context.identity.resolverContext.username,
- "wholeAuthorizerVariable": $util.escapeJavaScript($context.identity.resolverContext)
+ "wholeAuthorizerVariable": $util.escapeJavaScript($context.identity)
})

{
  "version": "2018-05-29",
  "payload": $util.toJson($payload),
}

The GraphQL Query result is even more interesting as it leaks some service internals.

{
  "data": {
    "getData": {
      "authorizerContextUsernameProperty": "test-username",
      "wholeAuthorizerVariable": "com.amazonaws.deepdish.common.identity.LambdaAuthIdentity@610a9a15"
    }
  }
}

I have no idea what com.amazonaws.deepdish.common.identity.LambdaAuthIdentity@610a9a15 is.

Bottom line

It appears that Amazon API Gateway evaluates the mapping template variables in a lazy fashion. Only the "deepest" properties of a given built-in variable are evaluated at all.

The AWS AppSync acts a bit differently. As we saw earlier with the $util.escapeJavaScript($context.identity.resolverContext) returning the stringified object.

After playing a bit more with the code, I have created the following rule of thumb I will be following from now on: Whenever working with built-in mapping template variables, always work with the "deepest" properties of a given variable.

Closing words

I wrote this article because I've spent a lot of time debugging a "weird" behavior of Amazon API Gateway mapping templates. While the knowledge of what I wrote about is not necessary for you to develop incredible applications, I hope it could save you the time I've wasted.

For more AWS serverless content, consider following me on Twitter – @wm_matuszewski.

Thank you for your precious time.

Saving on AWS Lambda Amazon CloudWatch Logs costs

Wojciech Matuszewski — Mon, 21 Feb 2022 16:04:15 +0000

AWS Lambda is a go-to compute service for many serverless developers. Powerful and flexible, it allows for arbitrary code execution in various programming languages. The service is not all sunshine and rainbows – things tend to fail. And when they do, it is crucial to have visibility into what happened.

A popular choice of gaining visibility into the AWS Lambda execution cycle is logging. When implemented with care, it can save you hours of debugging time, but it is a double-edged sword. The more you log, the more you will pay for Amazon CloudWatch Logs.

So how to eat the cake and have it too? How can we ensure that our logs are sufficient to help us debug potential AWS Lambda issues while still keeping our costs down?

This blog post will showcase one technique I'm aware of that, in my personal opinion, works excellent for this particular use case.

Let us begin.

All the code in this blog post is written in TypeScript. Treat the code snippets as pseudo-code rather than actual implementation.

The problem with the extremes

One mistake that, in my experience, developers tend to make is to reach for extremes – either logging everything or not logging at all. Both scenarios are problematic from an operational perspective.

Logging every action undoubtedly gives you this warm fuzzy feeling of comfort that if something were to happen, you have every piece of context right in front of you. Hopefully, this gives you the ability to pinpoint any potential issues fast and resolve them in no time.

The problem with this approach is that your Amazon CloudWatch bill will most likely be huge (relative to other services you use). I've seen serverless applications where Amazon CloudWatch contributed to 80% of the AWS bill. Not ideal.

Not having any logs in place or logging very little can also have devastating effects. This time, it is not about the AWS CloudWatch costs but the potential time it takes to debug an issue within the AWS Lambda function. Every error turns into a murder mystery with hours of resolution time. Not ideal.

In the ideal world, we could log the bare minimum when everything is fine but have very verbose logs when an error occurred. It turns out the "ideal world" is within reach and, in my personal experience, works great for most AWS Lambda functions, no matter the programming language.

The technique

The way we could achieve the logging nirvana I eluded to earlier is surprisingly not that complex (maybe I overlooked something? If so, please let me know!).

The basic idea is as follows:

Use your logger of choice as you were before.
The log.debug calls, by default, will not be pushed to STDOUT. Instead, we will keep them in memory.
The log.info calls will be pushed to STDOUT.
Whenever an error happens, in our case followed by a log.error call, the logger releases all log.debug messages, along with the log.error message.

With this setup in place, If the execution of an AWS Lambda function was successful, we only pay for the log.info (and other, all depends on your setup) data. Conversely, if something blows up, we get all the context we need to debug the execution.

Now, onto the implementation.

The implementation

Here is a very contrived example of an AWS Lambda function.

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda";

export const handler: APIGatewayProxyHandler = async event => {
  try {
    const data = await performWork(event);
    const result = await performOtherWork(data);

    return {
      statusCode: 200,
      body: JSON.stringify(result)
    };
  } catch (e) {
    return {
      statusCode: 500,
      message: "Something blew up!"
    };
  }
};

async function performWork(event: APIGatewayProxyEvent) {
  // implementation...
}

async function performOtherWork(data: unknown) {
  // implementation...
}

The code is not instrumented in any shape or form. Let us change that by adding logging.

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda";
+ import { log } from './logger'

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda";

export const handler: APIGatewayProxyHandler = async event => {
+ log.info("event", event);
  try {
    const data = await performWork(event);
    const result = await performOtherWork(data);

+ log.info("response", {...})
    return {
      statusCode: 200,
      body: JSON.stringify(result)
    };
  } catch (e) {
+ log.error("error!", e)

    return {
      statusCode: 500,
      message: "Something blew up!"
    };
  }
};

async function performWork(event: APIGatewayProxyEvent) {
+ log.debug("entering performWork", event)
  // implementation...
+ log.debug("exiting performWork", {...})
}

async function performOtherWork(data: unknown) {
+ log.debug("entering performOtherWork", data)
  // implementation...
+ log.debug("exiting performOtherWork", {...})
}

I've added quite a generous amount of logging to our code.

Keep in mind that this example is only for demonstrational purposes. You would most likely have a logging middleware that logs parts of the request and response in the real world.

If we were to allow every log.X call to push to STDOUT, given high-enough traffic, our Amazon CloudWatch bill might be pretty high.

There are many dimensions when it comes to Amazon CloudWatch Logs pricing. One of them is the size of the log messages, which I'm not concerned about in the example. Check out the Amazon CloudWatch pricing page for more details.

But our logger has a twist I've talked about earlier. The log.debug calls will NOT be pushed to STDOUT by default. Here is a contrived implementation of the logger.

// logger.ts
export const log = {
  debugCallsBuffer: [] as (() => void)[],
  info(message: string, ...args: any[]) {
    console.log(message, ...args);
  },
  error(message: string, ...args: any[]) {
    console.error(message, ...args);

    console.log("Debug buffer");

    this.debugCallsBuffer.forEach(debugLogCall => debugLogCall());
  },
  debug(message: string, ...args: any[]) {
    this.debugCallsBuffer.push(() => console.debug(message, ...args));
  }
};

The log object methods are facades over various console methods. Instead of calling console.debug eagerly, the debug function pushes a function to the debugCallsBuffer array. When we call the error function, the debugCallsBuffer is "flushed".

You most likely use a third-party package for logging in your day-to-day work to ensure that the log messages you produce are structured. Nevertheless, the idea still stands.

Neat! Now you can write log.debug statements to your heart's content without the fear of logging a massive blob of data for every AWS Lambda function execution.

We need to think about two more things before we start using the log object for our logging.

How do we "flush" the debugCallsBuffer when AWS Lambda executes successfully – otherwise, we run a risk of logging debug messages from previous AWS Lambda executions!
How do we "flush" the debugCallsBuffer whenever your AWS Lambda is about to timeout.

Flushing after successful AWS Lambda execution

If you read the documentation for AWS Lambda execution environment, you will learn about the concepts of execution environment re-use, freezing and unfreezing the execution environment. These concepts are necessary to understand the WHY behind this section, so consider giving it a read if you have not already.

If AWS Lambda service did not re-use the execution environments, we would not have been to cache at the AWS Lambda runtime level. For more information regarding caching at the AWS Lambda runtime level, consider reading this article or this article.

The following is a visual representation of the problem we are trying to solve in this section.

If we do not clear the debugCallsBuffer when AWS Lambda succeeds, we might log debug messages from the previous execution if it errors!. Not ideal.

The solution to the problem is very implementation-specific. Adding a clear function for our contrived logger example will do.

// logger.ts

export const log = {
  debugCallsBuffer: [] as (() => void)[],
  info(message: string, ...args: any[]) {
    console.log(message, ...args);
  },
  error(message: string, ...args: any[]) {
    console.error(message, ...args);

    console.log("Debug buffer");

    this.debugCallsBuffer.forEach(debugLogCall => debugLogCall());
  },
  debug(message: string, ...args: any[]) {
    this.debugCallsBuffer.push(() => console.debug(message, ...args));
  },
+ clear() {
+   this.debugCallsBuffer = []
+ }
};

And within our AWS Lambda handler code...

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda";
import { log } from './logger'

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda";

export const handler: APIGatewayProxyHandler = async event => {
 log.info("event", event);
  try {
    const data = await performWork(event);
    const result = await performOtherWork(data);

 log.info("response", {...})
    return {
      statusCode: 200,
      body: JSON.stringify(result)
    };
  } catch (e) {
 log.error("error!", e)

    return {
      statusCode: 500,
      message: "Something blew up!"
    };
+ } finally {
+   log.clear();
+ }
};

Again, the code is very contrived, but the point still stands. If you do not "clear" the buffer after every execution, you might see weird logs in the AWS CloudWatch logs console.

Flushing before AWS Lambda timeout

AWS Lambda is a time-bounded compute service. The maximum duration the AWS Lambda function can run for, at the time of writing this article, is 15 minutes. This setting is, of course, configurable.

If you are using a framework to deploy and develop your serverless applications (which, in my humble opinion, you should), the framework you use might set a default function timeout for you. Be aware of that.

I would argue that if my AWS Lambda function is about to timeout, it would be wise to release all the logs from the debugCallsBuffer. The function timeout will result in an error. If the function is frontend with Amazon API Gateway, the Amazon API Gateway will return a 500 status code to the caller (unless you have configured it to do otherwise). With all the logs available to me, I would likely have a higher chance of fixing the timeout issues.

To know whether our AWS Lambda function is about to timeout, we could use the getRemainingTimeInMillis function retrieved from the context that the service passes to our AWS Lambda function. Here is a sample implementation of logging an error (in turn releasing the debug logs) just before AWS Lambda is about to timeout.

import {
  APIGatewayProxyEvent,
  APIGatewayProxyHandler,
  Context
} from "aws-lambda";
import { log } from "./logger";

const logBeforeTimeout = (context: Context) => {
  const deadline = context.getRemainingTimeInMillis() - 100;
  const timeoutId = setTimeout(() => {
    log.error("About to timeout");
  }, deadline);

  return () => clearTimeout(timeoutId);
};

export const handler: APIGatewayProxyHandler = async (event, context) => {
  const cleanup = logBeforeTimeout(context);

  try {
    const result = await performWork(event);

    return {
      statusCode: 200,
      body: JSON.stringify(result)
    };
  } catch (e) {
    return {
      statusCode: 500,
      message: "Something blew up!"
    };
  } finally {
    cleanup();
  }
};

async function performWork(event: APIGatewayProxyEvent) {
  // implementation...
}

The logBeforeTimeout function starts a timer and logs an error before the deadline. It is essential to clear the timer no matter if the execution was successful or not. If you do not, you are asking for a memory leak to happen!

Small note for the Node.js users out there. It is entirely possible for the setTimeout never to run at all. If the Node.js event loop is blocked (think an infinite loop somewhere in your code), your timer will never run. One of the solutions I'm aware of is to run the timer in a separate thread.

Closing words

There you have it. This is how to have low AWS CloudWatch Logs costs and have every log line available to you when an issue occurs. This blog was heavy on the theory side since I wanted to relay an idea (which I believe is language-agnostic) rather than a specific implementation.

What do you think of this approach? Is it a bad idea? Maybe you have other solutions in mind? I'm always happy to hear your thoughts and opinions.

For more serverless content, consider following me on Twitter - @wm_matuszewski.

Thank you for your valuable time. Have a great day 👋.

Testing AWS Step Functions flows

Wojciech Matuszewski — Mon, 14 Feb 2022 17:00:47 +0000

AWS Step Functions is a serverless orchestrator service. Since its inception, it has become a go-to for all my low-code and orchestration needs on AWS.

Recently, AWS announced that the Step Functions Local can now mock service integrations. I found the announcement a great opportunity and excuse to revisit the topic of testing in the context of AWS Step Functions, which historically was a bit hard.

This article will cover the techniques I use for testing flows that utilize the AWS Step Functions service as the logic orchestrator. Let us begin.

The code in this blog post is written in TypeScript and uses AWS CDK for infrastructure piece. This GitHub repository contains all the code used in this article.

WojciechMatuszewski / testing-step-functions

Testing AWS Step Functions flows

This repository contains examples of how one might test various AWS Step Functions flows.

The test files are located in the lib/__tests__ directory.

Deployment

npm run boostrap
npm run deploy

Running the tests

Ensure that Docker is running.
Pull the aws-stepfunctions-local image.
npm run test

View on GitHub

Layers of testing and confidence

Before we dive into specific techniques, we shall take a swift detour and talk about testing in general.

There are many heuristics when it comes to testing. There is the classical testing pyramid or the testing honeycomb to name a few. In my personal opinion, in the context of serverless applications, tests written according to the testing honeycomb will give you much more confidence and ROI per test written than the "traditional" testing pyramid approach.

Before you base most of your tests on the local implementation of an AWS service, I urge you to think about the confidence the test gives you and less about how easy it is to write. Give the testing honeycomb a try. I'm positive you will not regret it!

By writing integration / end-to-end tests

By utilizing e2e / integration tests, we gain the most confidence from our tests. That said, the tests are usually slow and can be hard to maintain to some degree.

Here, you will be interacting with AWS services directly, executing the AWS Step Function and asserting the side-effects of the flow. The side-effect can be, for example, a new item in Amazon DynamoDB or a message pushed to Amazon SQS.

This style of testing is not free of challenging problems to tackle. Testing AWS Step Functions flow end-to-end becomes tricky when the Step Function definition is complex and contains multiple branches and error fallbacks.

Let us start with a basic example of saving a user into the Amazon DynamoDB table, then work our way up with Step Function definition complexity.

Simplistic Step Functions workflows

The following image represents the Step Function definition we would like to test.

Lacking branching logic and error handling, all we have to do is test a single execution path.

You should most likely always have error handling in place in your Step Function definition. This particular Step Function is only here for demonstration purposes. We will tackle testing error states later in the article.

Here is how I would write the test I'm referring to.



// simplistic-e2e.test.ts

import { SFNClient, StartExecutionCommand } from "@aws-sdk/client-sfn";

const sfnClient = new SFNClient({});

test("Saves the user in the DynamoDB table", async () => {
  const startExecutionResult = await sfnClient.send(
    new StartExecutionCommand({
      stateMachineArn: process.env.SIMPLISTIC_E2E_STEP_FUNCTION_ARN,
      input: JSON.stringify({
        firstName: "John",
        lastName: "Doe"
      })
    })
  );

  await expect({
    region: process.env.AWS_REGION,
    table: process.env.SIMPLISTIC_E2E_DATA_TABLE_NAME
  }).toHaveItem(
    {
      PK: `USER#${startExecutionResult.executionArn}`
    },
    {
      PK: `USER#${startExecutionResult.executionArn}`,
      firstName: "John",
      lastName: "Doe"
    }
  );
}, 15_000);

First, I start the Step Function, and then I assert, using the aws-testing-library, whether the item was correctly saved into the Amazon DynamoDB.

Check out the sls-test-tools as well. It is a great library. I'm using aws-testing-library because I'm used to it.

And that is it! Since this Step Function did not contain multiple branches and error handling, writing an end-to-end test takes little to no effort. With the most basic example behind us, let us tackle error states and multiple branches next.

Step Functions with branching logic

Step Functions can be complex, especially when taking error handling and Choice steps into account. With the increased complexity comes increased difficulty in writing end-to-end tests.

Let us evolve our Step Function to include "background-check" AWS Lambda function. Depending on the result, the user in the DynamoDB table will have its backgroundCheck attribute populated (either "PASS", "FAIL" or "ERROR").

You can find the AWS Step Function AWS CDK definition here.

To test all possible execution paths of the Step Function, we have to have a way to force an error or particular response for the BackgroundCheckStep AWS Lambda function – not an easy task!

So what can we do in this situation? Enter aws-stepfunctions-local.

The aws-stepfunctions-local is a local implementation of the AWS Step Functions service provided to us by great folks at AWS. With this tool, we will force the lambda to error without mocking other steps.

You could write a test that executes asserts on the status returned by the "background-check" AWS Lambda function. For the interest of time, I've chosen only to write a test for the case where the "background-check" fails.

Since I will be using the Docker version of the aws-stepfunctions-local, the first step is to integrate the act of spinning up and spinning down the container into the testing flow. My personal go-to in such situations is the testcontainers package.



// branching-logic-e2e.test.ts

let container: StartedTestContainer | undefined;

beforeAll(async () => {
  const mockConfigPath = join(__dirname, "./branching-logic-e2e.mocks.json");
  container = await new GenericContainer("amazon/aws-stepfunctions-local")
    .withExposedPorts(8083)
    .withBindMount(mockConfigPath, "/home/branching-logic-e2e.mocks.json", "ro")
    .withEnv("SFN_MOCK_CONFIG", "/home/branching-logic-e2e.mocks.json")
    .withEnv("AWS_ACCESS_KEY_ID", process.env.AWS_ACCESS_KEY_ID as string)
    .withEnv(
      "AWS_SECRET_ACCESS_KEY",
      process.env.AWS_SECRET_ACCESS_KEY as string
    )
    // For federated credentials (for example, SSO), this environment variable is required.
    .withEnv("AWS_SESSION_TOKEN", process.env.AWS_SESSION_TOKEN as string)
    .withEnv("AWS_DEFAULT_REGION", process.env.AWS_REGION)
    .start();
}, 15_000);

afterAll(async () => {
  await container?.stop();
}, 15_000);

Let us unpack what is going on here.

First, the mysterious mockConfigPath and subsequent usages of this variable. The path points to a mock configuration file described on aws-stepfunctions-local documentation page and contains a mock definition for the BackgroundCheckStep.



{
  "StateMachines": {
    "BranchingLogic": {
      "TestCases": {
        "ErrorPath": {
          "BackgroundCheckStep": "BackgroundCheckError"
        }
      }
    }
  },
  "MockedResponses": {
    "BackgroundCheckError": {
      "0": {
        "Throw": {
          "Error": "Lambda.TimeoutException",
          "Cause": "Lambda timed out."
        }
      }
    }
  }
}

Notice that I'm only concerned with the BackgroundCheckStep here. By only mocking BackgroundCheckStep, I can force this particular step to fail. Other steps are not mocked. Thus the aws-stepfunctions-local will reach out to native AWS services – in our case Amazon DynamoDB.

The Docker image needs to have AWS-related environment variables populated to allow aws-stepfunctions-local to talk to other AWS services. Refer to this documentation page for more information.



.withEnv("AWS_ACCESS_KEY_ID", process.env.AWS_ACCESS_KEY_ID as string)
.withEnv("AWS_SECRET_ACCESS_KEY",process.env.AWS_SECRET_ACCESS_KEY as string)
.withEnv("AWS_SESSION_TOKEN", process.env.AWS_SESSION_TOKEN as string)
.withEnv("AWS_DEFAULT_REGION", process.env.AWS_REGION)

As for the test itself, the first thing to do is gather necessary information about the Step Function under test.



// branching-logic-e2e.test.ts

test("Handles the failure of the BackgroundCheck step", async () => {
  const sfnClient = new SFNClient({});

  const describeStepFunctionResult = await sfnClient.send(
    new DescribeStateMachineCommand({
      stateMachineArn: process.env.BRANCHING_LOGIC_E2E_STEP_FUNCTION_ARN
    })
  );
  const stepFunctionDefinition =
    describeStepFunctionResult.definition as string;
  const stepFunctionRoleARN = describeStepFunctionResult.roleArn as string;

  // Rest of the test...
}, 50_000);

We will need all of this information since we will be re-creating this Step Function locally. Remember, the aws-stepfunctions-local is the engine that runs our Step Function.

Next, let us re-create and run the Step Function that lives in the cloud using the aws-stepfunctions-local.



// branching-logic-e2e.test.ts

// Test setup ...

test("Handles the failure of the BackgroundCheck step", async () => {
  // Previous code snippet ...

  const sfnLocalClient = new SFNClient({
    endpoint: `http://localhost:${container?.getMappedPort(8083)}`
  });

  const createLocalSFNResult = await sfnLocalClient.send(
    new CreateStateMachineCommand({
      definition: stepFunctionDefinition,
      name: "BranchingLogic",
      roleArn: stepFunctionRoleARN
    })
  );

  const startLocalSFNExecutionResult = await sfnLocalClient.send(
    new StartExecutionCommand({
      stateMachineArn: `${
        createLocalSFNResult.stateMachineArn as string
      }#ErrorPath`,
      input: JSON.stringify({
        firstName: "John",
        lastName: "Doe"
      })
    })
  );

  // Rest of the test...
}, 50_000);

There are two essential pieces of detail I would like to bring your attention to.

The name of the Step Function I'm creating. The name parameter must be the same as declared in the mock configuration file.
The stateMachineArn format in the StartExecutionCommand call. The convention of ARN#TEST_CASE is required by aws-stepfunctions-local. Refer to this documentation page to learn more.

All that is left is to assert on the data of a given Amazon DynamoDB item. The assertion is almost identical as in the Simplistic Step Functions section.



// branching-logic-e2e.test.ts

// Test setup ...

test("Handles the failure of the BackgroundCheck step", async () => {
  // Previous code snippet ...

  await expect({
    region: process.env.AWS_REGION,
    table: process.env.BRANCHING_LOGIC_E2E_DATA_TABLE_NAME
  }).toHaveItem(
    {
      PK: `USER#${startLocalSFNExecutionResult.executionArn}`
    },
    {
      PK: `USER#${startLocalSFNExecutionResult.executionArn}`,
      firstName: "John",
      lastName: "Doe",
      // The `BackgroundCheckStep` must have failed for this attribute to have the `ERROR` value.
      backgroundCheck: "ERROR"
    }
  );
}, 50_000);

We assert on an Amazon DynamoDB table that lives in the AWS. Since we did not provide any mocks for the step that saves the user data to Amazon DynamoDB, the aws-stepfunctions-local made the request to the actual service, utilizing the credentials from Docker environment variables.

Click to expand (the whole test definition)



import {
  CreateStateMachineCommand,
  DescribeStateMachineCommand,
  SFNClient,
  StartExecutionCommand
} from "@aws-sdk/client-sfn";
import { join } from "path";
import { GenericContainer, StartedTestContainer } from "testcontainers";

let container: StartedTestContainer | undefined;

beforeAll(async () => {
  const mockConfigPath = join(__dirname, "./branching-logic-e2e.mocks.json");
  container = await new GenericContainer("amazon/aws-stepfunctions-local")
    .withExposedPorts(8083)
    .withBindMount(mockConfigPath, "/home/branching-logic-e2e.mocks.json", "ro")
    .withEnv("SFN_MOCK_CONFIG", "/home/branching-logic-e2e.mocks.json")
    .withEnv("AWS_ACCESS_KEY_ID", process.env.AWS_ACCESS_KEY_ID as string)
    .withEnv(
      "AWS_SECRET_ACCESS_KEY",
      process.env.AWS_SECRET_ACCESS_KEY as string
    )
    /**
     * For federated credentials (for example, SSO), this environment variable is required.
     */
    .withEnv("AWS_SESSION_TOKEN", process.env.AWS_SESSION_TOKEN as string)
    .withEnv("AWS_DEFAULT_REGION", process.env.AWS_REGION)
    .start();
}, 15_000);

afterAll(async () => {
  await container?.stop();
}, 15_000);

test("Handles the failure of the BackgroundCheck step", async () => {
  const sfnClient = new SFNClient({});

  const describeStepFunctionResult = await sfnClient.send(
    new DescribeStateMachineCommand({
      stateMachineArn: process.env.BRANCHING_LOGIC_E2E_STEP_FUNCTION_ARN
    })
  );
  const stepFunctionDefinition =
    describeStepFunctionResult.definition as string;
  const stepFunctionRoleARN = describeStepFunctionResult.roleArn as string;

  const sfnLocalClient = new SFNClient({
    endpoint: `http://localhost:${container?.getMappedPort(8083)}`
  });

  const createLocalSFNResult = await sfnLocalClient.send(
    new CreateStateMachineCommand({
      definition: stepFunctionDefinition,
      name: "BranchingLogic",
      roleArn: stepFunctionRoleARN
    })
  );

  const startLocalSFNExecutionResult = await sfnLocalClient.send(
    new StartExecutionCommand({
      stateMachineArn: `${
        createLocalSFNResult.stateMachineArn as string
      }#ErrorPath`,
      input: JSON.stringify({
        firstName: "John",
        lastName: "Doe"
      })
    })
  );

  await expect({
    region: process.env.AWS_REGION,
    table: process.env.BRANCHING_LOGIC_E2E_DATA_TABLE_NAME
  }).toHaveItem(
    {
      PK: `USER#${startLocalSFNExecutionResult.executionArn}`
    },
    {
      PK: `USER#${startLocalSFNExecutionResult.executionArn}`,
      firstName: "John",
      lastName: "Doe",
      backgroundCheck: "ERROR"
    }
  );
}, 50_000);

By decomposing AWS Step Function Tasks

Switching gears from a somewhat complex end-to-end tests world, there exists another technique I would like to highlight.

I first saw this method of testing various AWS Step Function tasks while browsing code written by my colleagues at Stedi.

My friend Graham Allan wrote about it on his blog here. Check it out!

Testing Pass state transformations

Have you ever spent way too much time trying to transform data via the Pass state to the shape you need it to be? I know I have.

As great as the AWS console and the AWS Step Functions data flow simulator is, I find the feedback loop of a test case unbeatable.

So, how can we reliably extract those Pass states from our AWS CDK code and test them? Here is my solution.

Here is our sample Construct definition. It contains the transformDataStep that we would like to test.



// pass-states-integration.ts

export class PassStatesIntegration extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const transformDataStep = new aws_stepfunctions.Pass(
      this,
      "TransformDataStep",
      {
        parameters: {
          payload: aws_stepfunctions.JsonPath.stringAt(
            "States.Format('{} {}', $.firstName, $.lastName)"
          )
        }
      }
    );

    // You can imagine the definition being a bit more complex.
    const stepFunctionDefinition = transformDataStep;

    const stepFunction = new aws_stepfunctions.StateMachine(
      this,
      "StepFunction",
      {
        definition: stepFunctionDefinition
      }
    );
  }
}

We will be using the aws-stepfunctions-local, so the test setup looks very similar to the previous code snippets in this article.



// pass-states-integration.test.ts

let container: StartedTestContainer | undefined;

beforeAll(async () => {
  container = await new GenericContainer("amazon/aws-stepfunctions-local")
    .withExposedPorts(8083)
    .withEnv("AWS_DEFAULT_REGION", process.env.AWS_REGION)
    .start();
}, 15_000);

afterAll(async () => {
  await container?.stop();
}, 15_000);

To test the transformDataStep, we must retrieve the transformDataStep ASL definition. We can do this in two ways.

Use the DescribeStateMachine API call, like we did in the Step Functions with branching logic section.
Make the transformDataStep a publicly accessible property on the PassStatesIntegration construct.

I'm going to go with option number two since we have already seen option number one in action.



// pass-states-integration.ts

export class PassStatesIntegration extends Construct {
+ public transformDataStep: aws_stepfunctions.Pass;

  constructor(scope: Construct, id: string) {
    super(scope, id);

-   const transformDataStep = new aws_stepfunctions.Pass(
+   this.transformDataStep = new aws_stepfunctions.Pass(
      this,
      "TransformIncomingDataStep",
      {
        parameters: {
          payload: aws_stepfunctions.JsonPath.stringAt(
            "States.Format('{} {}', $.firstName, $.lastName)"
          )
        }
      }
    );

    // You can imagine the definition being a bit more complex.
-   const stepFunctionDefinition = transformDataStep;
+   const stepFunctionDefinition = this.transformDataStep;

    const stepFunction = new aws_stepfunctions.StateMachine(
      this,
      "StepFunction",
      {
        definition: stepFunctionDefinition
      }
    );
  }
}

With the transformDataStep made public, the test body would look as follows.



// pass-states-integration.test.ts

// Test setup ...

test("Handles the failure of the BackgroundCheck step", async () => {
  const stack = new cdk.Stack();
  const construct = new PassStatesIntegration(stack, "PassStatesIntegration");

  const transformDataStepDefinition = construct.transformDataStep.toStateJson();
  const stepFunctionDefinition = JSON.stringify({
    StartAt: "TransformDataStep",
    States: {
      TransformDataStep: {
        ...transformDataStepDefinition,
        End: true
      }
    }
  });

  const sfnLocalClient = new SFNClient({
    endpoint: `http://localhost:${container?.getMappedPort(8083)}`
  });

  const createLocalSFNResult = await sfnLocalClient.send(
    new CreateStateMachineCommand({
      definition: stepFunctionDefinition,
      name: "PassStates",
      roleArn: "arn:aws:iam::012345678901:role/DummyRole"
    })
  );

  const startLocalSFNExecutionResult = await sfnLocalClient.send(
    new StartExecutionCommand({
      stateMachineArn: createLocalSFNResult.stateMachineArn,
      input: JSON.stringify({
        firstName: "John",
        lastName: "Doe"
      })
    })
  );

  await waitFor(async () => {
    const getExecutionHistoryResult = await sfnLocalClient.send(
      new GetExecutionHistoryCommand({
        executionArn: startLocalSFNExecutionResult.executionArn
      })
    );

    const successState = getExecutionHistoryResult.events?.find(
      event => event.type == "ExecutionSucceeded"
    );

    expect(successState?.executionSucceededEventDetails?.output).toEqual(
      JSON.stringify({ payload: "John Doe" })
    );
  });
}, 20_000);

The ASL for the TransformDataStep is extracted via the toStateJson method. The rest of the test is similar to how we did it previously. The only difference is how we make the assertion.

This testing method is analogous to the one described by the By decomposing AWS Step Function Tasks section.

Closing words

I hope you find this blog post helpful regarding AWS Step Functions testing.

Consider following me on Twitter for more serverless content - @wm_matuszewski.

Thank you for your valuable time.

Forem: Wojciech Matuszewski

Automatic AWS CloudFormation rollbacks upon a test failure in your CI pipelines

Many ways to test the AWS serverless stack

Using CodeDeploy deployment group

Using the IntrinsicValidator package

Using a AWS CloudFormation Custom resource

Using the cdk-triggers package

The bottom line

Closing words

AWS CDK and Amazon DynamoDB global tables

Traditional way of creating global tables

The problem with the custom resource

The DeletionPolicy attribute

Enabling point-in-time recovery on the replica tables

A different way of creating global tables

Migration

Closing words

Strategies to test AWS AppSync VTL templates

Testing VTL templates using the AWS SDK

Testing VTL templates using VTL parsers

Testing VTL templates using AWS AppSync simulator

Testing VTL templates by making GraphQL requests

Closing words

Two ways to directly integrate AWS Lambda function with Amazon API Gateway

The proxy integration

The non-proxy integration type

My recommendation

Closing worlds

Better error messages for non-existing API resources with Amazon API Gateway

The problem

Unknown resource response in the context of Amazon API Gateway REST API

Modifying gateway responses

Deploying a "greedy" route with ANY method

Greedy route backed by AWS Lambda function

Greedy route backend by Mock Integration

Unknown resource response in the context of Amazon API Gateway HTTP API

Greedy route backed by AWS Lambda function

Closing words

Amazon EventBridge archive and ordered replay of events

About the event archive

The archive event replay

Ordered replay with the help of AWS Step Functions

Cost considerations

Closing words

Processing large payloads with Amazon API Gateway asynchronously

Problem refresher

The starting line - pushing the data to the storage layer

I have the data in Amazon S3. Now what?

S3 Event Notifications

EventBridge events

Munching on the data

Processing with a dedicated AWS Lambda

Processing with Amazon StepFunctions

Shared AWS Lambda storage

Yielding back to the user

Keeping track of the work status

Notifying the user about the results

Closing words

Synchronous AWS Lambda & Amazon API Gateway limits and what to do about them

The typical AWS Lambda Amazon API Gateway synchronous architecture

The payload size difference problem

I need to process bigger payloads synchronously

I need more time to process my synchronous request

AWS Lambda function URL

AWS Lambda fronted with Application Load Balancer

The ultimate solution

Closing words

Minimal AWS SSO setup for personal AWS development

The why of AWS SSO for AWS development

Setting the stage

The end goal

The configuration

The prerequisites

The steps

Testing

AWS console access through AWS SSO

AWS CLI access

Possible improvements

Closing words

Curious case of AWS mapping template built-in variables

The `DeletionPolicy` attribute

Deploying a "greedy" route with `ANY` method