Forem: Alex Kates

How to Trigger an AWS Lambda from Kinesis

Alex Kates — Wed, 14 Jul 2021 13:03:17 +0000

In this post we are going to use the AWS CDK to build an AWS Lambda Function that triggers from an AWS Kinesis Stream.

All of the code can be found in this repository.

Setup

We need to run a few commands to setup our CDK app.

mkdir how-to-trigger-lambda-from-kinesis
cd how-to-trigger-lambda-from-kinesis
npx cdk init app --language typescript

This should give you the following directory structure.

Also make sure you have your AWS CLI configured. For more information follow the AWS CLI quickstart guide.

Create a Kinesis Stream

Install the Kinesis CDK package.

npm i @aws-cdk/aws-kinesis

Open lib/how-to-trigger-lambda-from-kinesis-stack.ts, add a new Kinesis stream, and deploy.

import * as cdk from '@aws-cdk/core';
import * as kinesis from '@aws-cdk/aws-kinesis';

export class HowToTriggerLambdaFromKinesisStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const stream = new kinesis.Stream(this, 'MyKinesisStream', {
      streamName: 'MyKinesisStream',
    });
  }
}

Nice! Let's deploy the stack.

npm run cdk deploy

Create a Lambda

Install the Lambda CDK package.

npm i @aws-cdk/aws-lambda

Deploying a Lambda function requires bootstrapping your CDK app which gives us an S3 bucket where our Lambda's source code will live. This is a one time operation.

npm run cdk bootstrap

Create src/index.js and paste the following code

exports.handler = async (event) => {
  event.Records.forEach((record) => {
    console.log('Record: %j', record);
  });
};

Open lib/how-to-trigger-lambda-from-kinesis-stack.ts, add a new Lambda function, and deploy.

import * as cdk from '@aws-cdk/core';
import * as kinesis from '@aws-cdk/aws-kinesis';
import * as lambda from '@aws-cdk/aws-lambda';

export class HowToTriggerLambdaFromKinesisStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const stream = new kinesis.Stream(this, 'MyKinesisStream', {
      streamName: 'MyKinesisStream',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'KinesisMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });
  }
}

Deploy again ...

npm run cdk deploy

You may need to confirm some IAM changes.

Create the Event Source

Install the Lambda Event Sources CDK package.

npm i @aws-cdk/aws-lambda-event-sources

Open lib/how-to-trigger-lambda-from-kinesis-stack.ts, add a new KinesisEventSource to the Lambda Function.

import * as cdk from '@aws-cdk/core';
import * as kinesis from '@aws-cdk/aws-kinesis';
import * as lambda from '@aws-cdk/aws-lambda';
import * as lambdaEventSources from '@aws-cdk/aws-lambda-event-sources';

export class HowToTriggerLambdaFromKinesisStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const stream = new kinesis.Stream(this, 'MyKinesisStream', {
      streamName: 'MyKinesisStream',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'KinesisMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });

    const eventSource = new lambdaEventSources.KinesisEventSource(stream, {
      startingPosition: lambda.StartingPosition.TRIM_HORIZON,
    });

    lambdaFunction.addEventSource(eventSource);
  }
}

One more deployment ...

npm run cdk deploy

One last time, you may need to approve IAM changes.

Testing

We are going to make use of the AWS CLI to test our stack.

First, we need the Stream name of our Kinesis queue, which you can get using the following command

aws kinesis list-streams

Next, using the Stream name from the previous command, use the AWS CLI to send a new message to MyKinesisStream. Please refer to the Kinesis docs for information about these arguments.

aws kinesis put-record \
  --data "aGVsbG8sIHdvcmxk" \
  --stream-name MyKinesisStream \
  --partition-key pk1

Verify that the Lambda executed by looking in CloudWatch. Find the LogGroup named /aws/lambda/KinesisMessageHandler and open up the latest LogStream. You should see some log messages that look similar to this.

Clean Up

Don't forget to delete your stack when you are finished!

npm run cdk destroy

Thanks for reading! If you found this useful, please follow me here
https://dev.to/thealexkates
https://twitter.com/thealexkates

How to Trigger an AWS Lambda from SQS

Alex Kates — Fri, 02 Jul 2021 11:55:34 +0000

In this post we are going to use the AWS CDK to build an AWS Lambda Function that triggers from an AWS SQS message.

All of the code can be found in this repository.

Setup

We need to run a few commands to setup our CDK app.

mkdir how-to-trigger-lambda-from-sqs
cd how-to-trigger-lambda-from-sqs
npx cdk init app --language typescript

This should give you the following directory structure.

Also make sure you have your AWS CLI configured. For more information follow the AWS CLI quickstart guide.

Create an SQS Queue

Install the SQS CDK package.

npm i @aws-cdk/aws-sqs

Open lib/how-to-trigger-lambda-from-sqs-stack.ts, add a new SQS queue, and deploy.

import * as cdk from '@aws-cdk/core';
import * as sqs from '@aws-cdk/aws-sqs';

export class HowToTriggerLambdaFromSqsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const queue = new sqs.Queue(this, 'OurSqsQueue', {
      queueName: 'OurSQSQueue',
    });
  }
}

Nice! Let's deploy the stack.

npm run cdk deploy

Create a Lambda

Install the Lambda CDK package.

npm i @aws-cdk/aws-lambda

Deploying a Lambda function requires bootstrapping your CDK app which gives us an S3 bucket where our Lambda's source code will live. This is a one time operation.

npm run cdk bootstrap

Create src/index.js and paste the following code

exports.handler = async (event) => {
  event.Records.forEach((record) => {
    console.log('Record: %j', record);
  });
};

Open lib/how-to-trigger-lambda-from-sqs-stack.ts, add a new Lambda function, and deploy.

import * as cdk from '@aws-cdk/core';
import * as sqs from '@aws-cdk/aws-sqs';
import * as lambda from '@aws-cdk/aws-lambda';

export class HowToTriggerLambdaFromSqsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const queue = new sqs.Queue(this, 'OurSqsQueue', {
      queueName: 'OurSQSQueue',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'SqsMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });
  }
}

Deploy again ...

npm run cdk deploy

You may need to confirm some IAM changes.

Create the Event Source

Install the Lambda Event Sources CDK package.

npm i @aws-cdk/aws-lambda-event-sources

Open lib/how-to-trigger-lambda-from-sqs-stack.ts, add a new SqsEventSource to the Lambda Function.

import * as cdk from '@aws-cdk/core';
import * as sqs from '@aws-cdk/aws-sqs';
import * as lambda from '@aws-cdk/aws-lambda';
import * as lambdaEventSources from '@aws-cdk/aws-lambda-event-sources';

export class HowToTriggerLambdaFromSqsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const queue = new sqs.Queue(this, 'OurSqsQueue', {
      queueName: 'OurSQSQueue',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'SqsMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });

    const eventSource = new lambdaEventSources.SqsEventSource(queue);

    lambdaFunction.addEventSource(eventSource);
  }
}

One more deployment ...

npm run cdk deploy

One last time, you may need to approve IAM changes.

Testing

We are going to make use of the AWS CLI to test our stack.

First, we need the Queue URL of our SQS queue, which you can get using the following command

aws sqs get-queue-url --queue-name OurSQSQueue

Next, using the Queue URL from the previous command, use the AWS CLI to send a new message to OurSQSQueue. Your Queue URL may vary, but for me this command looks like this.

aws sqs send-message \
--message-body "Hello, world" \
--queue-url https://sqs.us-east-2.amazonaws.com/472331918655/OurSQSQueue

Verify that the Lambda executed by looking in CloudWatch. Find the LogGroup named /aws/lambda/SqsMessageHandler and open up the latest LogStream. You should see some log messages that look similar to this.

Clean Up

Don't forget to delete your stack when you are finished!

npm run cdk destroy

Thanks for reading! If you found this useful, please follow me here
https://dev.to/thealexkates
https://twitter.com/thealexkates

How to Trigger an AWS Lambda from SNS

Alex Kates — Fri, 25 Jun 2021 14:49:21 +0000

In this post we are going to use the AWS CDK to build an AWS Lambda Function that triggers from an AWS SNS message.

All of the code can be found in this repository.

Setup

We need to run a few commands to setup our CDK app.

mkdir how-to-trigger-lambda-from-sns
cd how-to-trigger-lambda-from-sns
npx cdk init app --language typescript

This should give you the following directory structure.

Also make sure you have your AWS CLI configured. For more information follow the AWS CLI quickstart guide.

Create an SNS Topic

Install the SNS CDK package.

npm i @aws-cdk/aws-sns

Open lib/how-to-trigger-lambda-from-sns-stack.ts, add a new SNS topic, and deploy.

import * as cdk from '@aws-cdk/core';
import * as sns from '@aws-cdk/aws-sns';

export class HowToTriggerLambdaFromSnsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const topic = new sns.Topic(this, 'OurSnsTopic', {
      displayName: 'Our SNS Topic',
    });
  }
}

Nice! Let's deploy the stack.

npm run cdk deploy

Create a Lambda

Install the Lambda CDK package.

npm i @aws-cdk/aws-lambda

Deploying a Lambda function requires bootstrapping your CDK app which gives us an S3 bucket where our Lambda's source code will live. This is a one time operation.

npm run cdk bootstrap

Create src/index.js and paste the following code

exports.handler = async (event) => {
  event.Records.forEach((record) => {
    console.log('Record: %j', record);
  });
};

Open lib/how-to-trigger-lambda-from-sns-stack.ts, add a new Lambda function, and deploy.

import * as cdk from '@aws-cdk/core';
import * as sns from '@aws-cdk/aws-sns';
import * as lambda from '@aws-cdk/aws-lambda';

export class HowToTriggerLambdaFromSnsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const topic = new sns.Topic(this, 'OurSnsTopic', {
      displayName: 'Our SNS Topic',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'SnsMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });
  }
}

Deploy again ...

npm run cdk deploy

You may need to confirm some IAM changes.

Create the Event Source

Install the Lambda Event Sources CDK package.

npm i @aws-cdk/aws-lambda-event-sources

Open lib/how-to-trigger-lambda-from-s3-stack.ts, add a new S3EventSource to the Lambda Function.

import * as cdk from '@aws-cdk/core';
import * as sns from '@aws-cdk/aws-sns';
import * as lambda from '@aws-cdk/aws-lambda';
import * as lambdaEventSources from '@aws-cdk/aws-lambda-event-sources';

export class HowToTriggerLambdaFromSnsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const topic = new sns.Topic(this, 'OurSnsTopic', {
      displayName: 'Our SNS Topic',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'SnsMessageHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });

    const eventSource = new lambdaEventSources.SnsEventSource(topic);

    lambdaFunction.addEventSource(eventSource);
  }
}

One more deployment ...

npm run cdk deploy

One last time, you may need to approve IAM changes.

Testing

We are going to make use of the AWS CLI to test our stack.

First, we need the ARN of our SNS topic, which you can get using the following command

aws sns list-topics --query "Topics[?contains(TopicArn, 'OurSnsTopic')]"

Next, using the TopicArn from the previous command, use the AWS CLI to publish a new message to Our SNS Topic. Your topic ARN may vary, but for me this command looks like this.

aws sns publish \
    --topic-arn "arn:aws:sns:us-east-2:472331918655:HowToTriggerLambdaFromSnsStack-OurSnsTopic2ED19057-1GAA1XT2U6XXH" \
    --message "Hello from SNS"

Verify that the Lambda executed by looking in CloudWatch. Find the LogGroup named /aws/lambda/SnsMessageHandler and open up the latest LogStream. You should see some log messages that look similar to this.

Clean Up

Don't forget to delete your stack when you are finished!

npm run cdk destroy

Thanks for reading! If you found this useful, please follow me here
https://dev.to/thealexkates
https://twitter.com/thealexkates

How to Trigger an AWS Lambda from S3

Alex Kates — Wed, 16 Jun 2021 12:42:11 +0000

In this post we are going to use the AWS CDK to build an AWS Lambda Function that triggers from AWS S3 Put requests.

All of the code can be found in this repository.

Setup

We need to run a few commands to setup our CDK app.

mkdir how-to-trigger-lambda-from-s3
cd how-to-trigger-lambda-from-s3
npx cdk init app --language typescript

This should give you the following directory structure.

Also make sure you have your AWS CLI configured. For more information follow the AWS CLI quickstart guide.

Create an S3 Bucket

Install the S3 CDK package.

npm i @aws-cdk/aws-s3

Open lib/how-to-trigger-lambda-from-s3-stack.ts, add a new S3 Bucket, and deploy.

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';

export class HowToTriggerLambdaFromS3Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'OurBucket', {
      /**
       * The following properties ensure the bucket is properly 
       * deleted when we run cdk destroy */
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    })
  }
}

npm run cdk deploy

You may need to confirm some IAM changes.

Create a Lambda

Install the Lambda CDK package.

npm i @aws-cdk/aws-lambda

Deploying a Lambda function requires bootstrapping your CDK app which gives us an S3 bucket where our Lambda's source code will live. This is a one time operation.

npm run cdk bootstrap

Create src/index.js and paste the following code

exports.handler = async (event) => {
  event.Records.forEach((record) => {
    console.log('Event Name: %s', record.eventName);
    console.log('S3 Request: %j', record.s3);
  });
};

Open lib/how-to-trigger-lambda-from-s3-stack.ts, add a new Lambda function, and deploy.

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';

export class HowToTriggerLambdaFromS3Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'OurBucket', {
      /**
       * The following properties ensure the bucket is properly 
       * deleted when we run cdk destroy */
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'BucketPutHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });
  }
}

Deploy again ...

npm run cdk deploy

You may need to confirm more IAM changes.

Create the Event Source

Install the Lambda Event Sources CDK package.

npm i @aws-cdk/aws-lambda-event-sources

Open lib/how-to-trigger-lambda-from-s3-stack.ts, add a new S3EventSource to the Lambda Function.

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';
import * as lambdaEventSources from '@aws-cdk/aws-lambda-event-sources';

export class HowToTriggerLambdaFromS3Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.Bucket(this, 'OurBucket', {
      /**
       * The following properties ensure the bucket is properly 
       * deleted when we run cdk destroy */
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'BucketPutHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });

    const s3PutEventSource = new lambdaEventSources.S3EventSource(bucket, {
      events: [
        s3.EventType.OBJECT_CREATED_PUT
      ]
    });

    lambdaFunction.addEventSource(s3PutEventSource);
  }
}

One more deployment ...

npm run cdk deploy

One last time, you may need to approve IAM changes.

Testing

Create a text file that we will use to upload to the S3 bucket.

echo "This is a test" > testFile.txt

Use the AWS CLI to upload this file into our S3 bucket from earlier. Your bucket name will vary, but for me this command looks like this.

aws s3 cp ./testFile.txt s3://howtotriggerlambdafroms3stack-ourbucket48caf57c-1p9g74rol1cza/testFile.txt

Verify that the Lambda executed by looking in CloudWatch. Find the LogGroup named /aws/lambda/BucketPutHandler and open up the latest LogStream. You should see some log messages that look similar to this.

Clean Up

Don't forget to delete your stack when you are finished!

npm run cdk destroy

Thanks for reading! If you found this useful, please follow me here
https://dev.to/thealexkates
https://twitter.com/thealexkates

How We Scaled Our Serverless SaaS Platform in AWS

Alex Kates — Fri, 11 Jun 2021 00:49:13 +0000

I was recently interviewed by Elise Greve of AWS about how we scaled Credit Genie from MVP to a full-scale platform with over 20 AWS Amplify applications. You can find the original article here. In this post I want to expand on some of the topics covered in that interview.

What is Credit Genie?

Credit Genie is a Fintech company focused on helping 10 million consumers get out of credit card debt. We've partnered with one of the largest non profit credit counseling organizations in the United States to provide a platform for Credit Counseling Agencies to provide debt relief to consumers struggling with credit card debt.

The Product

Credit Genie provides a SaaS based customer servicing platform that enables Credit Counseling agencies to collect consumer financial data including income, expenses and debt history in order to evaluate consumer financial health and offer needs-based debt management plans.

To achieve this, Credit Genie offers an online web application that allows the consumer to securely submit their financial information. Also, several web portals exist where the credit counselor and consumer can login and view various financial related information. The Credit Genie platform also integrates with several third party system such as TransUnion and Plaid to accurately confirm consumer financial information.

Our Tech

Credit Genie is Serverless-First, reaching for managed services to solve our business and technical problems. We believe that the future of software development and architecture is a cloud-based, serverless-first world where elasticity and pay-as-you-go are at the forefront.

We've also picked AWS to be our cloud provider and were early adopters of AWS Amplify to facilitate our web application development and cloud infrastructure provisioning needs. Leveraging Amplify has allowed us to go from MVP to a multi-account, single tenant platform serving millions of requests per month in under a year. Amplify is used to orchestrate our AWS serverless footprint including AppSync, DynamoDB, Lambda, APIGW, S3, SQS, Cognito, Redshift and more.

Guiding Principles

Building a SaaS based solution within FinTech offers its own set of unique problems. At the forefront are security, scalability and cost-effectiveness. We believe a shift-left approach on these 3 concerns, along with adoption of serverless technologies is what defines a modern, well-architected FinTech platform.

Security

Security within FinTech is a hot-button issue these days. A quick google search shows several high profile data breaches over the past 10 years.

Here at Credit Genie, we have 5 guiding principles for security.

Customer data must always be segregated. We have several counseling agencies servicing their customers on our platform, and it's important that each agency's customer data is isolated. The core idea here is to limit the blast radius if an incident were to occur.
Customer data must always be encrypted. Whenever customer data is in transit or at rest, it must be encrypted. Credit Genie leverages AWS KMS and TLS to ensure our customer's data is encrypted.
Humans and services must adhere to the principle of least privilege. Humans and services will always need permissions to do things; there's no way around that. The principle of least privilege means that these actors only receive as little privilege as needed to fulfill the task.
Tasks performed by humans and services must be observable and auditable. Credit Genie leverages CloudWatch and CloudTrail to ensure that all compute services are monitorable and all AWS API requests are auditable.
All permission grants must be time boxed. Whether a human or a service needs permissions to perform a task, they must re-authenticate within the context of each unit of work. In other words, no permanent authorization.

Scalability

At Credit Genie, we “build today with tomorrow in mind,” meaning everything we build is designed to cater to current scale requirements as well as anticipated growth.

This is why we love AWS Amplify and Serverless in general. Every managed service we leverage in AWS has this same mantra baked into its design. AWS calls it this elasticity, but the general idea is that your services should be able to scale up/down/in/out as the load on the system changes over time.

Cost-Effectiveness

Growth is fundamental to the success of a startup. Technology costs can be a wet blanket on growth if left unchecked. We design our systems with cost in mind from the start.

By always reaching for serverless and managed services in AWS first, we are getting elasticity in both scale and cost. AppSync, Lambda, S3, DynamoDB, Amplify and CloudFront all follow on-demand, pay as you go pricing models.

The Architecture

At its core, the Credit Genie platform is comprised of an Online Transactional Platform (OLTP) and an Online Analytical Platform (OLAP) and is all managed under a single AWS Organization.

The OLTP is a collection of single-tenant AWS accounts for each participating agency, with a few extra accounts to solve for cross-cutting concerns like IAM and DNS. These environments service the day to day operations of the consumer and credit counselor like collecting and storing consumer data and facilitating credit counselor daily activities.

The OLAP consists of a data warehouse in RedShift and S3 via Spectrum and an Airflow cluster managing numerous ETL workflows. The OLAP also enables our data scientists, analytics and business intelligence experts to do what they what they do best.

Let's step through this architecture diagram.

Customers are funneled to a single URL, which is a CloudFront distribution with a Viewer Request Lambda@Edge.
The Viewer Request Lambda@Edge intercepts every URL visit and determines which participating agency will receive the customer; ultimately redirecting the customer to the agencies Amplify application URL. This of this as a traffic router. From this point, the customer completes the online application within the agencies isolated AWS Amplify application, allowing for physical separation of customer data across member agencies.
As customers are routed into isolated Amplify applications and submit their financial information, subsets of that data is streamed into a centralized data warehouse in AWS RedShift. This enables our Reporting, Analytics and Data Science teams to perform analysis and business intelligence on all sorts of dimensional data sets.
DNS management with a multi-account single-tenant architecture can be tricky. Our solution involves having a dedicated AWS account with the single responsibility of managing all DNS records via Route53.
Identity and Access Management is also a complex problem to solve in this type of architecture. We lean heavily on AWS Organizations and SSO to manage all of our internal AWS Users, Roles and Permission Sets.
How do new features get introduce in an architecture like this? We have a single repository with several branches. Features are merged into our main branch which triggers the development environment's build, test and deploy steps. If successful, main is merged into a pre-production environment where all the same checks are run one more time. Finally, changes are merged into several customer environments and deployed into production.

Our Scale So Far

The following charts span the 10 months between August 2020 to May 2021 during which our platform scaled from 3 to 15+ environments. In general, we've seen a steady growth over the past 10 months as we've onboarded more and more Credit Counseling Agencies.

AppSync

AppSync requests/month increased from ~1.8 to ~6.2 million requests per month.

Lambda

Lambda GB-seconds increased from ~0.5 to ~2 million GB-seconds per month.

Amplify

Amplify Build usage increased from ~2 to ~4 thousand minutes per month.

S3

S3 storage increased from ~400 to ~900 GB per month

The Team

AWS operates under the Shared Responsibility Model. At a high level, this means that AWS guarantees the infrastructure, but it's up to you to responsibly drive and navigate it.

Our growth would not be possible without our amazing team of drivers, navigating AWS like the NASCAR champs that they are (sorry for the cheesy metaphor gang). I would be remiss to not mention each one of them here. Connect with all of these amazing engineers!

Wrapping Things Up

Thank you so much for reading through our technology journey over the past 10 months. AWS and Serverless technologies are very powerful tools for solving your security, scaling and cost problems and I hope this article helps you navigate your own AWS adventures!

Please consider following me here on dev.to and on twitter
https://dev.to/thealexkates
https://twitter.com/thealexkates

How to Mock Luxon's DateTime.now When Using TDD

Alex Kates — Mon, 07 Jun 2021 19:12:24 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1400555383391989763-350'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1400555383391989763&theme=dark" }

When is this relevant?

Ok, here's the situation. You are maintaining a NodeJS application that depends on Luxon and you practice test-driven development. Along comes a requirement that is dependent on the current date and time.

If you tried to use Luxon's default DateTime.now() function in your tests, you will quickly find that it's difficult to set up proper test data and assertions. This is due to the coupling between the tests and the actual current time.

What's the solution?

As TDD practitioners, we know it's always a good thing to fake integration points. This is true for network calls, database read/writes, and even the system clock!

Let's dive deeper into how you can leverage Jest to fake Luxon's default DateTime.now() function so we can assert on an expected current time.

First, let's take a look at a very simple system under test (SUT).

import { DateTime } from 'luxon';

export const doSomethingThatDependsOnDateTimeNow = () => DateTime.now();

Nothing special here, just a simple function that returns Luxon's DateTime.now() result.

Now let's dive into the unit test.

import { DateTime, Settings } from 'luxon';
import { doSomethingThatDependsOnDateTimeNow } from './doSomethingThatDependsOnDateTimeNow';

describe('Mocking DateTime.now with Luxon', () => {
  it('should return the mock DateTime', () => {
    // Arrange
    const expectedNow = DateTime.local(2021, 6, 1, 23, 0, 0);
    Settings.now = () => expectedNow.toMillis();

    // Act 
    const actual = doSomethingThatDependsOnDateTimeNow();

    // Assert
    expect(actual).toEqual(expectedNow);
  });
});

A lot going on here, so let's break it down.

First, we need to import Settings from Luxon, which is used to configure Luxon. We also import DateTime and our system under test (SUT), doSomethingThatDependsOnDateTimeNow.

Next, in the arrange section of the test, we create a new DateTime for the date 6/1/2021 at 11 pm EST. Then we override the DateTime.Now() function by assigning a new function Settings.now that returns our DateTime instead of the current time.

Finally, we invoke our SUT, capture the actual result, and assert that it is equal to our DateTime value.

Wrapping things up

Using TDD and integrating with Luxon's DateTime.now() can be a bit complex at first, but I hope this guide helps you red-green-refactor your way through your next time-based application feature.

If you found this useful, please consider following me on Twitter https://twitter.com/thealexkates

How to Animate a Waving Hand Emoji in TailwindCSS

Alex Kates — Thu, 03 Jun 2021 12:22:34 +0000

I recently spent time polishing the mobile reponsivness for my personal site, https://alexkates.dev, and I wanted to add a little animation flair to some empty space next to my Hero introduction.

👋 TL;DR 👋

I made the waving hand emoji actually wave using TailwindCSS.
You can find the source code here.

Setup

The majority of the Setup is taken from the TailwindCSS setup guide. The most jarring part is the use of Craco instead of the standard react-scripts.

Scaffold CRA, TailwindCSS and Craco

npx create-react-app waving-hand-tailwind

cd waving-hand-tailwind

npm install -D tailwindcss@npm:@tailwindcss/postcss7-compat postcss@^7 autoprefixer@^9

npm install @craco/craco

Update package.json

Configure craco.config.js

Configure tailwind.config.js

Include TailwindCSS in index.css

Build

Create the Wave Animation

Let's break down what is happening here.

We are extending the TailwindCSS configuration in order to bake in a new animation.

The keyframes section defines exactly how the animation transforms during the lifetime of the animation. At 0%, rotate 0 degrees; at 15%, rotate 14 degrees, etc.

Finally, the animation definition for wave to use the wave keyframes, transform for 1.5s, and loop infinitely.

Update App.js

Lastly, let's add a new span with the hand emoji.

Start Your App

npm start

Thanks for reading! If you found this useful feel free to follow me on twitter.

On the Topic of Writing

Alex Kates — Fri, 28 May 2021 17:52:18 +0000

I've consistently been writing for a few weeks now intending to publish a new post every Friday. This week, I want to pause the technical posts and reflect on my writing process and what I've learned with the hope that it resonates with other writers that may just be starting their journey.

Perfection Isn't Perfect

Perfection is not attainable, but if we chase perfection we can catch excellence.

Vince Lombardi
Winning coach of the first-ever NFL Super Bowl

A key metric of a SMART goal is that it needs to be attainable. That is, given a finite amount of time, the goal can be realistically achieved. This concept is really important when thinking about knowledge work. After several weeks of writing software development content, I realized that my biggest flaw in writing has been setting perfection as my goal.

But what is perfection? To me, perfection is when a thing reaches a state such that any new changes would no longer improve its quality. Working with computers and software has taught me that this state isn't obtainable. Given infinite resources, systems can always be improved just a bit more. Just like limits in calculus, you can approach perfection, but you'll never quite reach it. This was a big eureka; perfection can't be the goal because it isn't attainable!

So what does this mean for writing? If perfection isn't the goal, then what is? I realized over the past few weeks that I was chasing something more fulfilling; I was chasing consistency. My goal changed from writing perfect articles to publishing something new every Friday; to making progress. While this may now appear obvious, it took quite a bit of reflection to get here.

The Mysterious Quality

It is not the mountain we conquer but ourselves.

Edmund Hillary
The first climber to summit Mount Everest

Knowledge work, including writing, isn't wholly about the destination. For me, it's about the journey; about the progress. There is something special in the progress that is hard to qualify. This mysterious quality lives in the grind; in the iteration; in the revision and editing; in the trial and error. It feels like quality. That is, the process of writing allows for a self-reflection to happen where I can measure my quality improvements.

This feedback loop is powerful. I've found that as I write, I read and revise in the same cycle. Each cycle shows some tiny improvement from the last cycle. It's addicting; like a shot of dopamine over and over until the piece is finished. This also feels very similar to the way I write software. I use test-driven development, which has a similar micro-feedback loop.

I've also found that I feel a bit empty when I finish a piece. Almost like a mild withdrawal from the dopamine cycles. Maybe this is a bit of an exaggeration, I'll admit. But there is a change that occurs with each write/revise cycle that culminates into a feeling of relief and emptiness when finished. Don't get me wrong, I'm excited to publish every week and see the feedback and engagement, but there is something sad about finishing.

Fear of the Unknown

The oldest and strongest emotion of mankind is fear, and the oldest and strongest kind of fear is fear of the unknown.

H. P. Lovecraft
Writer and creator of Cthulhu Mythos

Fear is debilitating, no matter the manifestation. It can manifest in things as extreme as staring down a Grizzly on its hind legs, or as simple as tasting seared octopus for the first time. Regardless of the manifestation, the effect is always the same.

This isn't any different for writing. Especially as a new writer, the process can bring feelings of dread. For me, this fear manifested from the thought of putting my thoughts out for the public to potentially criticize. It's the fear of being ridiculed that stopped me from writing all these years.

I don't have any profound realizations on this one. I still have this fear, even as writing this. I don't think fear is meant to be dismissed. Fear is a sieve that separates recklessness from intentionality. I've learned to recognize when fear comes and use it for a moment of reflection.

Until Next Time

I'll admit this got a bit philosophical, but I think it's good to stop and reflect especially when starting something new. I'm hooked on writing, that's for sure. I'm excited to come back to this post in a year and see how I've changed.

I hope you found this insightful or at the very least entertaining. If so, please like and follow!

https://twitter.com/thealexkates
https://www.alexkates.dev

5 Essential Extreme Programming (XP) Resources

Alex Kates — Sat, 22 May 2021 13:54:31 +0000

Extreme Programming (XP) is a software development methodology that promotes communication, simplicity, feedback, courage and respect in order to create higher quality software and maximize quality of life for developers.

The following resource list is not comprehensive by any means. It does however serve as an essential starting point for your XP adventures.

1. Extreme Programming Explained

Extreme Programming (XP) is about social change. It is about letting go of habits and patterns that were adaptive in the past, but now get in the way of us doing our best work.

Kent Beck - Extreme Programming Explained

First on our list is Extreme Programming Explained by Kent Beck. Where better to start than the godfather of XP, Kent Beck? Beck's body of work, including Extreme Programming Explained, is considered by most to be the system of record for XP.

First written in 1999 and revised as a 2nd edition in 2004, Extreme Programming Explained is a comprehensive catalog of the what and why of XP. It includes deep dives into the values, principles, and practices that are hallmark of XP.

If you are just starting your XP journey, Extreme Programming Explained is a great place to start. Also, check out the story of the Chrysler Comprehensive Compensation System (C3) for a great origin story of XP.

2. Planning Extreme Programming

We don't plan so we can predict the future. Business and software are changing too rapidly for prediction to be possible. Even if it were possible to predict what we needed in three years, it wouldn't necessarily help because between now and then we need so many different things.

Kent Beck, Martin Fowler - Planning Extreme Programming

Our next essential resource is Planning Extreme Programming. Written in 2000, Kent pairs up with another XP legend Martin Fowler to dissect all things XP and planning, including:

Estimating scope, time, and effort for user stories
Prioritizing user stories
Balancing the business value and technical risk of user stories
What to do when you’re not going to make the date

While Planning Extreme Programming is a book written about planning, it isn't just for planners. XP teaches us about the value of whole-team and how important it is when it comes to planning. Planning Extreme Programming is a must-read for developers and project managers looking to adopt XP.

3. Extreme Programming Installed

Extreme Programming is a discipline of software development with values of simplicity, communication, feedback and courage. We focus on the roles of customer, manager, and programmer and accord key rights and responsibilities to those in those roles.

Ron Jeffries, Ann Anderson and Chet Henrickson - Extreme Programming Installed

Next up is Extreme Programming Installed by Ron Jeffries, Ann Anderson, and Chet Hendrickson

Extreme Programming Installed is a practical guide to XP with emphasis on roles and their rights/responsibilities with an XP team. It lays out in a bill of rights way, the Manager, Customer and Programmer roles and their respective rights. For example

Manager and Customer Rights

You have the right to an overall plan, to know what can be accomplished, when, and at what cost.
You have the right to get the most possible value out of every programming week.
You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify.

Programmer Rights

You have the right to know what is needed, with clear declarations of priority.
You have the right to produce quality work at all times.
You have the right to make and update your own estimates.

4. The Art of Agile Development: Pragmatic Guide to Agile Software Development

Have you ever heard a musician playing scales? That's an étude. An étude teaches mastery through precise and careful repetition. Eventually, the étude is abandoned, but the skills remain. Extreme Programming is our étude for software development.

James Shore and Shane Warden - The Art of Agile Development

If I could only recommend one XP resource, it would be The Art of Agile Development by James Shore and Shane Warden.

I'll admit, I'm a bit biased. James Shore was my first exposure to XP back in 2013 where I was lucky enough to join a team that had brought him in to coach them directly in XP.

Released in 2007, The Art of Agile Development is responsible for transforming countless developers into XP practitioners. For me, it's the book that is always on my desk and has 20+ sticky note bookmarks.

The Art of Agile Development emphasizes étude as a tool to mastery. Imagine an infield baseball player taking fielding practice, or an archer shooting arrow after arrow at a target, études offer a way for developers to practice and master skills such as test-driven development, refactoring and iterative design.

5. Test-Driven Development: By Example

If courage is a core value of XP, then Test-Driven Development (TDD) is the garlic that makes you brave enough to fight off the proverbial software vampire known as change.

Rounding out our list, Kent Beck returns to deliver Test-Driven Development, the quintessential guide to practicing TDD. Beck guides us through real world domain problems and how TDD can be used to fearlessly iterate and refactor. While the tools and technologies may be a bit dated, the core tenets still hold, even in the 2021 ever changing web development world.

Bonus Material

While the following resources didn't make the essential list, they are fantastic in their own rights. I highly encourage you to check them out!

Also, if you are now hooked on XP and are near the Philadelphia area, please check out https://www.meetup.com/PhillyXP/

That's it folks! Hope this article sparked an interest in XP for you.

Feel free to follow me https://twitter.com/thealexkates

How to Trigger an AWS Lambda from DynamoDB

Alex Kates — Fri, 14 May 2021 15:43:20 +0000

In this post we are going to use the AWS CDK to build an AWS Lambda Function that triggers from DynamoDB Stream Events.

All of the code can be found in this repository.

Setup

We need to run a few commands to setup our CDK app.

mkdir how-to-trigger-lambda-from-ddb-stream
cd how-to-trigger-lambda-from-ddb-stream
npx cdk init app --language typescript

This should give you the following directory structure.

Also make sure you have your AWS CLI configured. For more information follow the AWS CLI quickstart guide.

DynamoDB

Install the DynamoDB CDK package.

npm i @aws-cdk/aws-dynamodb

Open lib/how-to-trigger-lambda-from-ddb-stream-stack.ts, add a new DynamoDB table and deploy.

import * as cdk from '@aws-cdk/core';
import * as dynamodb from '@aws-cdk/aws-dynamodb';

export class HowToTriggerLambdaFromDdbStreamStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const table = new dynamodb.Table(this, 'Table', {
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
      stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
      tableName: 'Table',
    });
  }
}

npm run cdk deploy

Lambda

Install the Lambda CDK package.

npm i @aws-cdk/aws-lambda

Deploying Lambda functions requires bootstrapping your CDK app which gives us an S3 bucket where our Lambda's source code will live.

npm run cdk bootstrap

Create src/index.js and paste the following code

exports.handler = async (event) => {
    event.Records.forEach((record) => {
        console.log('Event Id: %s', record.eventID);
        console.log('Event Id: %s', record.eventName);
        console.log('DynamoDB Record: %j', record.dynamodb);
    });
};

Open lib/how-to-trigger-lambda-from-ddb-stream-stack.ts, add a new Lambda function and deploy.

import * as cdk from '@aws-cdk/core';
import * as dynamodb from '@aws-cdk/aws-dynamodb';
import * as lambda from '@aws-cdk/aws-lambda';

export class HowToTriggerLambdaFromDdbStreamStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const table = new dynamodb.Table(this, 'Table', {
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
      tableName: 'Table',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'TableStreamHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });
  }
}

npm run cdk deploy

Event Source

Install the Lambda Event Sources CDK package.

npm i @aws-cdk/aws-lambda-event-sources

Open lib/how-to-trigger-lambda-from-ddb-stream-stack.ts, add a new DynamoEventSource to the Lambda Function.

import * as cdk from '@aws-cdk/core';
import * as dynamodb from '@aws-cdk/aws-dynamodb';
import * as lambda from '@aws-cdk/aws-lambda';
import { DynamoEventSource } from '@aws-cdk/aws-lambda-event-sources';

export class HowToTriggerLambdaFromDdbStreamStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const table = new dynamodb.Table(this, 'Table', {
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
      stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
      tableName: 'Table',
    });

    const lambdaFunction = new lambda.Function(this, 'Function', {
      code: lambda.Code.fromAsset('src'),
      handler: 'index.handler',
      functionName: 'TableStreamHandler',
      runtime: lambda.Runtime.NODEJS_12_X,
    });

    lambdaFunction.addEventSource(new DynamoEventSource(table, {
      startingPosition: lambda.StartingPosition.LATEST,
    }));
  }
}

npm run cdk deploy

Testing

Using the AWS CLI, insert a new item into the DynamoDB table.

aws dynamodb put-item \
    --table-name Table \
    --item '{
        "id": {"S": "test123"}
      }'

Verify that the Lambda executed by looking in CloudWatch. Find the LogGroup named /aws/lambda/TableStreamHandler and open up the latest LogStream. You should see some log messages that look similar to this.

2021-05-14T19:08:55.487Z    ffba16f2-846b-4c6e-9ff5-189c3293116e    INFO    Event Id: e6db4f872fd1997f0f459bcebb8163e8

2021-05-14T19:08:55.527Z    ffba16f2-846b-4c6e-9ff5-189c3293116e    INFO    Event Id: INSERT

2021-05-14T19:08:55.527Z    ffba16f2-846b-4c6e-9ff5-189c3293116e    INFO    DynamoDB Record: {
    "ApproximateCreationDateTime": 1621019335,
    "Keys": {
        "id": {
            "S": "test123"
        }
    },
    "NewImage": {
        "id": {
            "S": "test123"
        }
    },
    "SequenceNumber": "733400000000008471956488",
    "SizeBytes": 18,
    "StreamViewType": "NEW_AND_OLD_IMAGES"
}

Summary

Congratulations. If you've made it this far then you successfully ...

Scaffolded a new CDK app
Created a DynamoDB table
Created a Lambda Function
Created a DynamoEventSource for the Lambda Function