Forem: Michael Mather

AWS Cloud Map service discovery for ECS Fargate containers

Michael Mather — Wed, 08 Mar 2023 22:34:27 +0000

AWS ECS Fargate is a serverless container provider that lets you run containers in the cloud without managing the underlying servers. They're often used as easy-to-manage and cost-effective methods of hosting web APIs. Fargate as it is works perfectly for hosting, say, your frontend on one container and your API on another. But as the complexity of an application grows you might find that you need to split up your code into multiple services. Container orchestration is perfect for this as it lets your services scale up and down separately depending on their requirements, and keeps code confined into smaller, more manageable pieces. I won't go into the details of microservice architecture in this post but using ECS is a common method of developing a microservice architecture.

If you're developing a microservice architecture on ECS, you might reach the point where you need to allow one container to speak to another in the same cluster. This sounds simple but is fairly complex in practice. For one, each service can scale up and down independently, so if one container sends a request to another, how does that container know the actual IP of the container to send the request to? There may be multiple IPs available, and the sender would need to figure out which IP to send it to. The process of a container figuring out how to send request to another is called service discovery. There are many, many ways to handle service discovery that go beyond the scope of this post. I'll be discussing a method that's relevant to ECS and the AWS ecosystem: AWS Cloud Map.

AWS Cloud Map

Cloud Map is AWS' service discovery offering, that allows you to register IPs that the service discovery service can then provide that information to other services who need to know. Cloud Map offers three namespace types:

Http Namespaces

HTTP Namespaces allow your services to use the DiscoverInstances API call to retrieve a list of IPs of your services. Your service can then send a request to any of the IPs however you feel is most appropriate.

Public DNS Namespaces

Private DNS namespaces create Route53 records for each registered service instance. DNS queries can then be made within your VPC to discover services. You can choose between a WEIGHTED routing policy, in which Route53 responds with the IP of a random instance, or MULTIVALUE in which the DNS query will respond with the IPs of up to 8 healthy instances. You can also still access the DiscoverInstances API call when you're using private namespaces.

Public DNS Namespaces

Public DNS namespaces are similar to the private namespace, except it will obviously use a public DNS to respond to instance requests. Cloud Map also doesn't support making API calls the get registered instances when using public namespaces.

In this post, I'll be going over using Private DNS Namespaces and using Route53's weighted routing policy to respond to DNS queries.

Setting up the namespace

Let's say we have two services running in ECS, where service A needs to be able to request some data from service B. We want service A to be able to find out where it should be sending the request to, and the system should be able to automatically register new instances as the containers scale up and down. We'll be using a Cloud Map Private DNS Namespace to do this. To demonstrate, I'll be using some CDK code.

We'll start by setting up our Fargate cluster

import * as servicediscovery from '@aws-cdk/aws-servicediscovery';
import * as ecs from "@aws-cdk/aws-ecs";

const taskDefinition = new ecs.FargateTaskDefinition(this, "MyTaskDefinition", {
  cpu: 512,
  memoryLimitMiB: 2048
});

taskDefinition.addContainer("FargateContainer", {
  image: "<your container definition>",
  logging: ecs.LogDriver.awsLogs({ streamPrefix: "myService" }),
  portMappings: [
    { containerPort: 80 } // important. Default is no port mapping
  ]
});

const ecsService = new ecs.FargateService(this, 'MyService', {
   cluster,
   taskDefinition
});

  const cloudMapNamespace = new servicediscovery.PrivateDnsNamespace(this, `ServiceDiscoveryNamespace`, {
    name: 'mydomain.com', // The domain your want to use in the DNS lookup
    vpc: yourVpc
  });

const cloudMapService = new servicediscovery.Service(this, `ServiceDiscovery`, {
  namespace: cloudMapNamespace,
  dnsRecordType: servicediscovery.DnsRecordType.A,
  dnsTtl: cdk.Duration.seconds(300),
  name: 'service-name', // will be used as a subdomain of the domain set in the namespace
  routingPolicy: servicediscovery.RoutingPolicy.WEIGHTED,
  loadBalancer: true // Important! If you choose WEIGHTED but don't set this, the routing policy will default to MULTIVALUE instead
 })

ecsService.associateCloudMapService({
  service: cloudMapService
 })

Let's go through this step by step:

Create a task definition, and add a container to it.
Create a Fargate service and set the task definition
Create the Cloud Map namespace
Create the CloudMap service and set the record type to A, and the routing policy to WEIGHTED. Note: it's important that you set the loadBalancer property to true if you want to use weighted routing.
Associate the ECS service with the Cloud Map service.

Once this is deployed, go to Route53 and have a look for the hosted zone created by Cloud Map. You should see that it has 1 A record with a routing policy set to weighted and the value as the private IP of your container. If your containers are set up and working correctly, you should be able to launch another one, and send a request to . to reach your service container.

Deploying an Elastic Beanstalk Application Using the AWS CDK

Michael Mather — Wed, 24 Mar 2021 14:40:28 +0000

The AWS CDK is AWS' "infrastructure as code" toolkit that allows you to build AWS resources using code. I've used CloudFormation extensively, and although I find the documentation to be fairly easy to follow, the developer experience just isn't there. The ability to write infrastructure in the same language you're developing your backend on is a huge advantage, and it makes AWS infrastructure much more accessible to those without extensive experience in the AWS ecosystem. I am also a huge Typescript fan so the fact that the CDK comes with Typescript out of the box is amazing. Though the AWS CDK is growing in popularity (and rightly so), the documentation is sorely lacking and very hard to read. I find I end up spending most of the time looking at the CloudFormation documentation and mapping it back to the CDK instead.

I recently setup an Elastic Beanstalk application using the CDK and ran into a couple of issues that I hope I can clear up for others. In this post I'll describe how to create a simple Elastic Beanstalk application using the AWS CDK that you can use to deploy a compatible web application.

Elastic Beanstalk

If you're familiar with Heroku or DigitalOcean's new App Platform, you should be able to get up and running quickly. EB is a simple-to-use service for deploying web applications that supports most web application frameworks. Using the EB CLI, you can deploy NodeJS, Django and many more applications with a simple eb deploy command. EB comes with load balancing out the box and allows you to use auto-scaling configurations to handle scaling. We'll be setting up a NodeJs web application and deploying it to EB using the EB CLI.

Here's an example of what our final architecture will look like:

Prerequisites

To create and deploy the app, you'll need both the AWS CDK installed, and the Elastic Beanstalk CLI. The CDK can be installed through npm -g install aws-cdk and the EB CLI can be installed through apt or brew depending on your system. We'll use the CDK to create and bootstrap our project, then we'll use the EB CLI to deploy our code. You'll also need the AWS CLI installed and your profile configured. See here for how to do that.

Getting Set Up

First, you'll want to initialize your CDK project.The CLI comes with a handy way to start new projects. I'll be using typescript for this but you can use whatever language you prefer. In a new directory, run:

cdk init --app myApp --language typescript

You should notice the CDK has created a bunch of folders: lib/, bin/ and test. You'll create all of your stacks in the lib folder, write your tests in the test folder and initialize your stacks for deployment in the bin folder when you're ready to deploy.

Installing the CDK Libraries

The CDK packages its libraries for various resources separately. Rather than having one massive library, we can just include only what's necessary for our project. We'll be using the IAM and Elastic Beanstalk libraries which you can install by doing:

npm install @aws-cdk/aws-elasticbeanstalk @aws-cdk/aws-iam

Note: When using multiple libraries, you need to make sure the libraries you're using are on the same version. I ran into a compatibility issue when the @aws-cdk/core package was using V1.94.0 and the @aws-cdk/aws-iam library was using V1.85.0. Updating all packages to use the same version (E.g. V1.94.0 or V1.85.0) resolved this issue.

Creating your Stack

Next, we'll want to take a look at what's in the lib/ folder. You should see a file named after the app name you specified when you initialized the project. You should see something similar to this:

import * as cdk from '@aws-cdk/core';
import * as elasticbeanstalk from '@aws-cdk/aws-elasticbeanstalk';
import * as iam from '@aws-cdk/aws-iam';

export class AppStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
  }
}

const app = new cdk.App();

new AppStack(app, 'Production');

app.synth();

The CDK works by creating "stacks". If you're not familiar with CloudFormation, a stack is a collection of AWS resource configurations (by resources, I mean databases, servers, containers etc.). The stack itself doesn't contain any resources, it is just a collection of resource configurations. When you deploy your stack, AWS then uses the configurations to create the resources. You don't pay anything for CloudFormation, but you will have to pay for anything CloudFormation creates.

To get our app up and running, we'll want to create 3 resources:

An IAM role and an Instance Profile to give EB the permissions to manage our server
An EB Environment
An EB App

Let's start with our IAM permissions:

// EBS IAM Roles
const EbInstanceRole = new iam.Role(this, `${appName}-aws-elasticbeanstalk-ec2-role`, {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
});

const managedPolicy = iam.ManagedPolicy.fromAwsManagedPolicyName('AWSElasticBeanstalkWebTier')
EbInstanceRole.addManagedPolicy(managedPolicy);

const profileName = `${appName}-InstanceProfile`
const instanceProfile = new iam.CfnInstanceProfile(this, profileName, {
  instanceProfileName: profileName,
  roles: [
    EbInstanceRole.roleName
  ]
});

This creates an IAM role with the AWSElasticBeanstalkWebTier managed IAM policy attached. Next, we create the instance profile and use the role we've just created. This gives the server the permissions it needs to run as an EB server.

Next, we'll create the EB application and environment:


const node = this.node;
const platform = node.tryGetContext("platform");

const optionSettingProperties: elasticbeanstalk.CfnEnvironment.OptionSettingProperty[] = [
  {
    namespace: 'aws:autoscaling:launchconfiguration',
    optionName: 'InstanceType',
    value: 't3.small',
  },
  {
    namespace: 'aws:autoscaling:launchconfiguration',
    optionName: 'IamInstanceProfile',
    value: profileName
  }
];

// EBS Application and Environment
const app = new elasticbeanstalk.CfnApplication(this, 'Application', {
  applicationName: `${appName}-EB-App`
});

const env = new elasticbeanstalk.CfnEnvironment(this, 'Environment', {
  environmentName: `${appName}-EB-Env`,
  applicationName: `${appName}-EB-App`,
  platformArn: platform,
  solutionStackName: '64bit Amazon Linux 2 v5.3.0 running Node.js 14',
  optionSettings: optionSettingProperties
});

env.addDependsOn(app);

In our option settings, we specify the instance profile we created earlier and the instance type we'll be running. I've just selected the t3 small but you might want something bigger if you're expecting higher workloads. Setting up the application and the environment then is quite simple. Note that you may want to use a different solutionStackName. I'll be running a NodeJs server so I've selected Amazon Linux with Node.js 14. You can view all of the available solution stacks here.

Deploying Our Stack

To deploy our stack you'll first want to run the synth command to ensure that there's no issues translating our Typescript code into CloudFormation templates. Next, we'll call the bootstrap command to get the CDK to initialize our Stack in AWS. This will create some additional resources that the CDK uses to manage the state of your stack. You'll likely see a stack created in CloudFormation called "CDK Toolkit". Lastly, we'll deploy our stack. You might see a warning about IAM permissions. The CDK will warn you if your stack is creating or changing IAM permissions as a security measure.

cdk synth
cdk bootstrap
cdk deploy

Now we wait. You can navigate to the CloudFormation page in the AWS Console if you want to see more details about what the stack is creating, and if any failures have occurred.

Once the stack is complete, you should be able to see your EB Environment and Application in the EB console and your server health should be "healthy". Next, we'll look at deploying some code the the server.

Deploying our Server

In order for us to deploy our code to our newly created server, we first need to initialize our project with EB. I'm using a simple Express app which I won't detail here as there's plenty of resources online for how to set that up. To start, run:

eb init

If you're using the same profile you used to deploy your CDK stack, you should see your EB environment show as a selectable option in your terminal. Select your environment and wait for the project to initialize. Once that's done, it should be as simple as:

eb deploy

Debugging EB Deployment Issues

If you're running into issues with your deployment, you can view the most recent logs or SSH into your server by running some useful commands:

eb logs
eb ssh

Testing and Conclusion

You should now be able to find the URL for your application in the AWS console. From here on, you may want to build on your stack by creating databases, S3 buckets, networks or Lambdas. You can continue to add resources to your stack, or break parts of it out into separate stacks for modularity. I would also recommend creating 2 instances of your stack, one for development/staging, and one for production. If you're setting up more than a hobby project, I'd recommend you do this in separate accounts and add your application into a VPC with security groups, and a web application firewall for increased security. But if this is a small project, this should be perfectly fine and can scale to thousands of users easily.

View the original post and more here

Using AWS RDS Proxy to Pool Database Connections from Lambda

Michael Mather — Fri, 05 Mar 2021 21:27:38 +0000

The title sounds like a bunch of random nonsensical words strung together, but bear with me. AWS announced general availability for RDS Proxy at re:Invent 2019 - a highly available proxy for (you guessed it) RDS that pools database connections for your serverless functions to improve performance and decrease load on your databases. In this post I’ll be discussing more about the purpose of RDS and how to set it up for your Lambda functions.

Serverless has gained so much popularity in the last few years due to the scalable nature of running code in ephemeral functions. This is great because you only need to run code when you absolutely need to rather than having an always-on server sitting idle when it’s not receiving any traffic. However, one of the things that has taken some time to catch up is making database connections from a serverless function.

When running code on an always-on server, if you’re connecting to a database (E.g. MySQL or Postgres) usually your application will automatically handle database connection pooling.

Connection pooling is when a certain number of database connections are kept alive so that when a request comes in that needs to query the database it doesn’t need to create a new connection every time, it can borrow one of the existing connections from the connection pool. This saves on CPU and memory usage for the database since the only calls to the db are the ones that query data, rather than creating/closing connections concurrently. In Lambda functions we don’t have that luxury, and this has been a problem for some time.

Since Lambda functions are stateless and asynchronous, meaning that state from one function doesn’t carry over to any subsequent Lambda calls, database connections can’t be kept alive. Every database connection must be opened and closed during the runtime of the function. This has been a problem for many and the only real way to mitigate this issue was to control the concurrency of your lambdas to prevent them from flooding the database when they scale. There were some solutions that managed to reuse connections while the lambda was still warm but this was far from ideal and seemed more of a hack than a solution to a very clear problem in serverless architecture.

Introducing RDS Proxy

To solve this problem, AWS introduced RDS Proxy. When using RDS Proxy in front of your database, the Proxy will manage connection pools for you and allow your Lambda functions to borrow existing connections rather than create new ones. Then your Proxy will forward your request to RDS and return the result. Besides offering speed and scalability to your Lambda functions, Proxy also automatically fails over to standby databases if your primary database fails while preserving application connections. Proxy helps to mitigate database failures due to too many connections by allowing you to control the number of connections Proxy will allow. This means you could set it to only use 80% of the allowed maximum connections to prevent overloading your database. While it does help to scale, it is still limited by the maximum number of connections your database can allow.

RDS Proxy can also add additional security to your databases. It uses database credentials stored in Secrets Manager to connect to the database, but your Lambda can use IAM authentication by using the AWS SDK to generate a signed password to connect to the Proxy

This is a good summary of the authentication process that occurs when your application queries a database through RDS Proxy.

Credit: https://aws.amazon.com/blogs/compute/introducing-the-serverless-lamp-stack-part-2-relational-databases/

How to use RDS Proxy for MySQL RDS

To use RDS Proxy, you'll need to have a couple of core components:

RDS Proxy (obviously)
An RDS Proxy Target Group
A compatible RDS database instance
Database credentials in Secrets Manager

I'll be using CloudFormation templates to demonstrate the RDS Proxy configuration, but will assume you already have a database instance set up. Here is a very simple RDS Proxy configuration:

DbProxy:
  Type: AWS::RDS::DBProxy
  Properties:
    Auth:
      - AuthScheme: SECRETS # Proxy will use credentials in Secrets Manager to authenticate with the DB
        SecretArn: !Ref: DbUserSecret
        IAMAuth: REQUIRED # Connections to the Proxy must use IAM auth
    DebugLogging: true
    DBProxyName: my-db-proxy
    EngineFamily: MYSQL
    IdleClientTimeout: 1800
    RequireTLS: true
    VpcSubnetIds:
      - !Ref: PrivateSubnet
    VpcSecurityGroupIds:
      - !Ref: ProxySG
    RoleArn: !GetAtt: [ProxyRole, Arn]

Something to note here is that in order for RDS to authenticate with your database you need to reference credentials stored in Secrets Manager. This also means that you must create a database user that uses those credentials. What I would recommend is to make one database user for regular querying of the database with limited SELECT, UPDATE, INSERT, DELETE permissions, and another to run database migrations with permissions to change tables and such. The credentials for both of those users would need to be stored in Secrets Manager and referenced here in the Proxy configuration.

Here's a template for creating a secret in Secrets Manager that you can reference in the above config

AppDbUser:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: app-db-user
      Description: Application user for serverless app
      GenerateSecretString:
        SecretStringTemplate: '{ "username": "appUser"}'
        GenerateStringKey: "password"
        PasswordLength: 30
        ExcludeCharacters: '"@/\' # MySQL doesn't like these characters

Next, you'll need to tell your proxy which database you'd like to forward requests to. To do this you need to create a Proxy Target Group:

ProxyTargetGroup:
  Type: AWS::RDS::DBProxyTargetGroup
  Properties:
    DBProxyName: !Ref: DbProxy
    DBInstanceIdentifiers:
      - !Ref: MasterDbInstance
    TargetGroupName: default
    ConnectionPoolConfigurationInfo:
        MaxConnectionsPercent: 50
        MaxIdleConnectionsPercent: 12
        ConnectionBorrowTimeout: 30

Target groups are what connects your Proxy to your databases. You can have multiple target groups with different configurations.

Now, all you need to do is connect to the proxy endpoint the same way you would connect to a regular database instance. The only difference is that rather than using regular database credentials, you'll use the AWS sdk to generate a signed password:


const { DB_HOST, DB_USERNAME, DB_NAME } = process.env;

const signer = new AWS.RDS.Signer({
    region: '[REGION]',
    hostname: DB_HOST, // The proxy endpoint
    port: 3306,
    username: DB_USERNAME
});

const token = signer.getAuthToken({
      username: DB_USERNAME
});

const connectionConfig = {
    host: DB_HOST
    user: DB_USERNAME,
    database: DB_NAME
    ssl: { rejectUnauthorized: false },
    password: token,
    authSwitchHandler: function ({pluginName, pluginData}, cb) 
        // See here: https://dev.mysql.com/doc/internals/en/clear-text-authentication.html
        console.log("Setting new auth handler.");
    }
};

You can now use this connection config (or a similar version) in the database library of your choosing.

Your Lambdas are now ready to see very real decreases in their database connection times and you can now sleep soundly. This was a very brief and shallow introduction to RDS Proxy, but I hope I was able to demonstrate it's advantages and clear up some of the gotchas I ran into when I first started using it. For anyone interested in a much deeper dive into RDS Proxy, here's a fantastic resource that I hope helps:

Adding a Custom Domain Name to AWS CloudFront

Michael Mather — Sun, 28 Feb 2021 22:18:09 +0000

This site is built using AWS S3 for static website hosting, behind an AWS CloudFront distribution. To get the site to be served on the domain we want, we have to setup CloudFront to use a custom CNAME, and enable HTTPS traffic by provisioning an SSL certificate through AWS ACM. In this post, I'll explain how you can setup an existing CloudFront distribution with a custom domain name.

Getting our SSL Certificate

For this post I'll assume you already have an S3 site with a CloudFront distribution. If not, take a look at the documentation here and here.

In order for us to safely server traffic on our website using TLS, we need to get a certificate from Amazon Certificate Manager. The default CloudFront distribution domain has it's own TLS certificates that it uses. If we want to serve the site on our own domain, AWS will need to validate that we own the domain. To do this, search for ACM in the AWS console and click on Certificate Manager.

Next, click Request a Public Certificate and enter the domain name you want to use. For this site, I used "mikemather.cintsa-cms.com". Once that's done, you should see the fields that you'll need to add to your DNS records to validate the domain.

Now we can go into our DNS provider (I use NameCheap) and create a new DNS record. You'll use CNAME as the record type and use the name (or host in my name) and value fields provided by AWS. AWS uses this to validate that you own the domain you're requesting the certificate for.

When that's completed, you'll see that your certificate is pending validation. Depending on your DNS provider this could take anywhere from a few minutes to a few hours. You'll just have to sit back and wait while they work their magic.

Note: This is specifically for subdomains, not root domains. For root domains you'll need to add a hosted zone in AWS Route53 and add an A record with the CloudFront distribution as the alias. See here for more details.

For me, it took about 5 minutes to validate.

Nice! Now we can apply the domain to our CloudFront distribution.

Applying the Domain to CloudFront

To apply our domain to the distribution, all we need to do is tell CloudFront which domain to use and which certificate to use to serve HTTPS traffic. To do this, go to the CloudFront console in AWS. You should see your distribution listed. Select your distribution and click Distribution Settings and then Edit.

In the "Alternate domain names" field enter the domain name that you created the certificate for. You can use any number of domain names here as long as they match the domain names you used when you created your SSL certificate. Then in the SSL certificate field, select "Custom SSL Certificate". When you click into the input field, you should see the certificate that you created listed, so you can just select it there. Now all you have to do is save your changes and you're good to go. You should see your changes reflected relatively quickly.

Updating your DNS records

There are two ways that you may need to update your DNS records. If you're using a subdomain instead of a root domain, you can just create a CNAME record in your DNS registrar that points to your CloudFront distribution domain. E.g. if you want to host a site on myapp.example.com then you set the Name of the DNS record to myapp, the value to your CloudFront domain (xxxxx.cloudfront.net) and the record type to CNAME.

If you're intending to host the site on a root domain (e.g. example.com) then you'll have to use a Route53 hosted zone. You can still use a Route53 hosted zone if you have your domain registered with another registrar (see here). In short, you can create a hosted zone and set the NS records in you DNS registrar to point to your hosted zone. Then in Route53 you can go into your domain and create an A record, and select the CloudFront distribution from the "Alias" field. If your domain is already registered through Route53, then you'll just need to create an A record with the CloudFront distribution as the alias.

Conclusion

In this post I used CloudFront in front of an S3 website, but you can use CloudFront for serving just about any static content. For example, you could host an app on Heroku with a CloudFront distribution to speed-up content delivery. Whatever you use CloudFront for, the process for adding custom domains is still the same.

Securely Serving a Static Site on AWS S3 with CloudFront

Michael Mather — Wed, 13 Jan 2021 19:06:28 +0000

For any sadists out there who enjoy the quirkiness of the the AWS permissions ecosystem, you'll love this one.

I've been working on a project recently that involves hosting a static site on S3 and serving it via CloudFront CDN. AWS has some good documentation on the process but I ended up running into a problem that I found to be incredibly frustrating. Since I spent about a full day getting way too deep into the docs on this issue, I thought I'd spell it out for any other poor souls that run into the same trouble:

Public Read Access to a Bucket Makes me Nervous

When you setup your bucket with a web-hosting configuration the docs instruct to use a bucket policy that allows public read access such as:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::example.com/*"
            ]
        }
    ]
}

Then your content gets served via a website similar to mybucket.com.s3-website-.amazonaws.com.

Now, we've just given public read access to our bucket, but we want to serve the site through CloudFront only. We don't want people using the direct S3 website endpoints, we want to restrict it to our CloudFront configuration. Usually what AWS suggests is to use an Origin Access Identity and update the bucket policy to restrict to that OAI.

But if there's one thing I've learned while using AWS, it's that there's always one little info box that can suddenly end days of failed deployments and poring through docs. In this case, it's this one in these docs:

These are the docs on configuring CloudFront for S3 content, and they explicitly state that you can't restrict bucket access when using a website configuration. What they suggest is using a custom header in your CloudFront configuration instead and conditionally requiring that header in the bucket policy.

Using a Custom Header to Secure Bucket Access

To implement this, you can go into your distribution's settings, and add a custom header:

For the value field, use a value that's unique, or something that only you know. I just used the Python secrets.token_urlsafe() function

Then we can update the bucket policy to read this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudFrontGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::example.com/*"
            ],
            "Condition": {
             "StringLike": {
                "aws:Referer": "gQye0hsIAIsnEnX1EIwUNVLa5Tnly5UFthZrx6n7-gI"
              }
            }
        }
    ]
}

Notice the last key "Condition". We use a "StringLike" match on the Referer header on the request to ensure that the we only allow access to requests that match our unique key. Now, when we try to access our S3 website via the object URLs instead of through CloudFront we'll get an Access Denied error and we've successfully secured our bucket content.

In the event that this post didn't help at all, here's a couple of articles that should point you in the right direction:

Middi - Digital Ocean Hackathon Submission

Michael Mather — Wed, 06 Jan 2021 01:36:31 +0000

What I built

Middi is a fun and wholesome habit tracker that makes you feel good for completing habits. The Middi app takes place in a chat-like interface where users get support and encouragement every time they complete a habit. Manage your habits, complete your goals and feel good doing it.

Category Submission:

Program for the People: Build an application related to mental health, a public service (like a COVID-19 spread tracker), or a feel-good app to ring in the New Year right

App Link

Middi App

Screenshots

Description

Middi lets you create and track habits with weekly goals. You can then monitor your habit progress and get encouragement (and confetti parties) every time you complete a goal.

Link to Source Code!

Middi Repo

Permissive License

MIT

Background

I've tried to get into habit tracking a couple of times but I always find that the material design "spreadsheet" approach never really appealed to me. I liked the idea of a habit tracker that made you feel good every time you completed a habit. I tried to build Middi to be friendly and supportive so that habit tracking didn't feel like making a grocery list.

How I built it

This project really shows how easy it is to make a full-stack project on Digital Ocean App Platform. I use one app to host both the backend and the frontend, and an App Platform component to create a PostgreSQL database. Linking the two up was as simple as a few clicks with DO doing the heavy lifting, including managing database users.

I use Heroku regularly so It's really great to see competitors in the same space.

I used NodeJS with Express, Passport and Sequelize to handle the authentication and database models. I then used React with TypeScript on the frontend to handle the client side of things.

Additional Resources/Info

Huge thanks to DEV and Digital Ocean for organising this event. Looking forward to seeing the rest of the submissions!

DO Hackathon - The Finished Result

Michael Mather — Wed, 06 Jan 2021 00:59:04 +0000

Unfortunately, I didn't have as much time over the Christmas break as I was hoping so I didn't get a chance to go further in depth into the client-side part of the app. With that said, I did want to write a post about the final features that I managed to fit in.

Feature Overview

The Chat Interface

All interactions with the app are done through a chat interface. That means we load components into view dynamically within the dialog.

Adding Habits

You can add new habits and specify a custom colour, and the number of times per week you'd like to aim to complete the habit.

Viewing Habit Progress

You can view your individual habit progress, or an overview of all of your habits combined.

Celebrations

The best part of the app in my opinion is the celebration animations. Any time you do a habit or complete a goal a celebration animation is shown. It feels good when a mini confetti party is thrown every time you complete a goal.

Lessons Learned

I had a ton of fun developing this app and building everything from the ground up. I thought I'd also mention a few lessons I learned along the way:

Chat dialog is hard! I regret not looking more into good chat dialog methods (thinking of gaming systems in particular). Getting the chat sequences working correctly was challenging and I think there are good tools out there that I could have used.
Design for extensibility first. While it's possible to write your own chat sequences into the app, it would be nice to have an easily extensible system that allows anyone to define their own sequences with custom components.
Start simple first. I spent too much time in the beginning trying to figure out how I was going to build a system that would match the complex tracking systems of other habit trackers. Eventually, I realized that I've spent too much time designing things that are clearly out of scope. I settled on a very simple goal system that works well with the simplicity of the app.

DO Hackathon - Local Authentication Using PassportJS

Michael Mather — Mon, 21 Dec 2020 16:46:30 +0000

Now that we've got our database models, we need to setup some authentication routes to be able to query our database. I decided to use the popular PassportJS library to handle authentication.

The most important thing to understand about Passport is that it's incredibly modular. That means that handling user login, and handling JWT authentication requires two different libraries with separate Passport "strategies". Strategies are middleware functions that allow us to apply some verification to the incoming request.

Username/Password Login vs Federated

I've followed a number of other posts which are using things like Firebase or other federated sign-in authentication tools. My reasoning for using local username/password sign in is that I want this app to be a one-click deploy to anyone that wants to try it out. From the start, I've tried to ensure that there's minimal setup to get it running. Meaning that I didn't want you to have to have a Firebase or Auth0 account to get it up and running.

I decided that I'd use a regular email/password authentication to authorize users, and then use JWT stored in a cookie for subsequent requests to the backend.

Passport Strategies

In Passport terms, it means we'll be creating a Local strategy, and a JWT strategy. Here's what that looks like:

const bcrypt = require('bcrypt');
const { User } = require('../database/models');
const passport = require('passport');
const localStrategy = require('passport-local').Strategy;

passport.use(
    'login',
    new localStrategy({
        usernameField: 'email',
        passwordField: 'password',
        session: false
    }, (username, password, done) => {
        try {
            User.findOne({
                where: {
                    email: username
                }
            }).then(user => {
                if (!user) {
                    return done({ message: 'incorrect email or password' }, false)
                }
                else {
                    bcrypt.compare(password, user.password).then(response => {
                        if (!response) {
                            return done({ message: 'incorrect email or password' }, false)
                        }
                        return done(null, user)
                    })
                }
            })
        }
        catch (err) {
            done(err);
        }
    })
)

This creates a passport strategy that allows us to verify a user when they attempt to login using our /login route. When we create the login route, we'll specify this strategy as the middleware to check the users' credentials and create the token that will be stored in the browser cookie.
We're using the bcrypt library to encrypt passwords before storing them.

Next, we'll create a JWT strategy that will verify a JWT token from the cookie.

const jwtSecret = require('./jwtConfig');
const passportJWT = require('passport-jwt');
const JWTStrategy =passportJWT.Strategy;

passport.use(
    'jwt',
    new JWTStrategy({
        jwtFromRequest: req =>  req.cookies ? req.cookies.jwt : null,
        secretOrKey: jwtSecret.secret
    },
    (jwtPayload, done) => {
        if (!jwtPayload) {
            console.log('No token')
            return done({ error: 'JWT missing' }, false);
        }
        else if (Date.now() > jwtPayload.expires) {
            console.log('Token expired')
            return done({ error: 'JWT expired' }, false);
        }
        return done(null, jwtPayload);
    }
    )
)

This strategy takes the token from the cookie in the request, verifies it, and returns the payload. This means if we use the token with the user ID as the payload, every request that uses this strategy (i.e. all of our protected routes) will have access to the user ID if the token is valid.
You'll notice we're importing a secret key from a file called ./jwtConfig.js. We won't actually be storing the key itself in that file as it would be incredibly bad practice. Instead the file will be using a secret key stored in an environment variable. Digital Ocean's App Platform allows us to configure and encrypt environment variables which will make sure our secret key stays secret.

Applying the Strategies to Routes

At the moment, our strategies are good but aren't implemented anywhere in our app. We have to apply the strategies to a route in order for them to work. We'll create some auth controllers to handle authentication:

const jwt = require('jsonwebtoken');
const passport = require('passport');
const db = require('../database/models');
const express = require('express');
const router = express.Router();

router.post('/login', async (req, res) => {
  await passport.authenticate(
    'login',
    { session: false },
    (err, user) => {
      if (err || !user) {
        console.log('Error authenticating user: ', err);
        res.status(400).json({ err });
      }

      const expires = Date.now() + 24 * 60 * 60 * 7 // One week
      const payload = {
        id: user.id,
        name: user.name,
        email: user.email,
        expires
      }
      req.login(payload, { session: false }, error => {
        if (error) {
          console.log('Error logging in: ', err);
          res.status(400).send({ error })
        }
        const token = jwt.sign(JSON.stringify(payload), secret);
        res.cookie('jwt', token, { 
          httpOnly: true, 
          secure: process.env.NODE_ENV === 'production',
          maxAge: expires
        });
        const { name, email } = user;
        res.status(200).send({ name, email });
      });
    }
  )(req, res)
})

Here, we've defined our /login route. The login process involves:

Pass the request/response object to our passport.authenticate function
Check if the authentication strategy threw an error and return the error to the frontend if so
Create the JWT payload and sign it
Apply the token to a secure cookie on the response
Return the response to the frontend

Storing the token in a cookie with the httpOnly option means that the token can only be accessed on the server, not through JavaScript on the browser. This will help protect our app from cross-site scripting attacks and is much more secure that storing the token directly in local storage on the browser. When this cookie is attached, the browser will automatically attach it to subsequent requests to our backend.

Next, we'll take a look at our JWT strategy implementation. After a user has successfully logged in, we'll receive a token in a cookie for all subsequent requests. We'll use our JWT strategy to verify this token before allowing access to our database. Here's an example route:

router.get(
  '/auth-user', 
  (req, res) => {
    passport.authenticate(
      'jwt', 
      { session: false },
      (err, user) => {
        if (err) {
          res.status(401).send({ message: err });
        }
        else {
          res.status(200).send({ user });
        }
      })(req, res)
});

This is a simple route that checks if the user is logged in. We can use this route to decide on the frontend if we'll show them the logged-in view or the default public view. It simply applies our JWT strategy, which will give us back the user if it is valid, or an error if not. All of our private routes will use this format which will ensure the user is authenticated before we pass any data from our database.

Next, we'll get started taking a look at the meat of our application - our React frontend

DO Hackathon - Postgres and Sequelize models

Michael Mather — Fri, 18 Dec 2020 13:21:31 +0000

As mentioned in a previous post, our app will be using a PostgreSQL database. To consolidate our database tables with data models in our code, we'll be using Sequelize.

I followed this brilliant post which outlines everything you'll need to know about connecting to the database and running migrations. I won't reiterate anything that's already in the post so instead I'll get right into our models and their relationships.

As you might remember, the purpose of our app is to help users keep track of when they complete habits and allow them to track their goal progression. We also want users to have the ability to login so that we can protect their data better. This means we'll likely be looking at models that look like this:

User:
  id: primary key
  name: string
  email: string
  password: string
  createdAt: timestamp

Habit:
  id: primary key
  userId: foreign key - users
  name: string
  description: string
  color: string
  weeklyGoal: number
  createdAt: timestamp

CompletedTask:
  id: primary key
  habitId: foreign key - inhalers
  dateCompleted: datetime
  createdAt: timestamp

CompletedGoal:
  id: primary key
  habitId: foreign key - inhalers
  weekStartDate: datetime
  createdAt: timestamp

I found the Sequelize documentation a little confusing, because there's multiple ways to run migrations. Coming from a Django/Flask background I was used to the process of modifying your models, which would automatically create migrations. Sequelize allows us to do something similar using the sync() method. Here's how the models turned out:

Our User model:

// models/users.js
const { DataTypes } = require('sequelize')

module.exports = {
    id: {
      type: DataTypes.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    name: {
      type: DataTypes.STRING,
      allowNull: false
    },
    email: {
      type: DataTypes.STRING,
      allowNull: false
    },
    password: {
      type: DataTypes.STRING,
      allowNull: false
    },
    createdAt: {
      allowNull: false,
      type: DataTypes.DATE
    },
    updatedAt: {
      allowNull: false,
      type: DataTypes.DATE
    }
  }

Our Habit model:

// models/habits.js
const { DataTypes } = require('sequelize')
module.exports = {
    id: {
      type: DataTypes.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    name: {
      type: DataTypes.STRING,
      allowNull: false
    },
    description: {
      type: DataTypes.STRING,
      allowNull: false
    },
    color: {
      type: DataTypes.STRING,
      allowNull: false
    },
    weeklyGoal: {
      type: DataTypes.INTEGER,
      allowNull: false
    },
    createdAt: {
      allowNull: false,
      type: DataTypes.DATE
    },
    updatedAt: {
      allowNull: false,
      type: DataTypes.DATE
    }
}

Our CompletedTask model:

// models/completedTask
const { DataTypes } = require('sequelize')

module.exports = {
    id: {
      type: DataTypes.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    dateCompleted: {
      allowNull: false,
      type: DataTypes.DATE
    },
    createdAt: {
      allowNull: false,
      type: DataTypes.DATE
    },
    updatedAt: {
      allowNull: false,
      type: DataTypes.DATE
    }
}

And lastly:

// models/completedGoals
const { DataTypes } = require('sequelize')

module.exports = {
    id: {
      type: DataTypes.INTEGER,
      primaryKey: true,
      autoIncrement: true
    },
    weekStartDate: {
      allowNull: false,
      type: DataTypes.DATE
    },
    createdAt: {
      allowNull: false,
      type: DataTypes.DATE
    },
    updatedAt: {
      allowNull: false,
      type: DataTypes.DATE
    }
}

You'll notice that there's no foreign keys in these models. That's because sequelize allows us to define relationships using the .hasMany(), .haseOne() and .belongsTo() model functions.

Now that we've got our definitions, we need to initialize our models and sync up any changes with the database. I created an index.js file in the /models directory to do this:

const fs = require('fs');
const path = require('path');
const Sequelize = require('sequelize');
const envConfigs =  require('../config/config');
const userDef = require('./user');
const habitDef = require('./habit');
const completedTaskDef = require('./completedTask');
const completedGoalDef = require('./completedGoal');

const env = process.env.NODE_ENV || 'development';
const config = envConfigs[env];
const db = {};

// Establish the connection
const sequelize = new Sequelize(config.url, config);
db.sequelize = sequelize;
db.Sequelize = Sequelize;

// Define the models
const User = sequelize.define('User', userDef);
const Habit = sequelize.define('Habit', habitDef);
const CompletedTask = sequelize.define('CompletedTask', completedTaskDef);
const CompletedGoal = sequelize.define('CompletedGoal', completedGoalDef);

// Define the relationships between models
Habit.hasMany(CompletedTask, {
  onDelete: 'CASCADE'
});
Habit.hasMany(CompletedGoal, {
  onDelete: 'CASCADE'
});
CompletedTask.belongsTo(Habit);
CompletedGoal.belongsTo(Habit);
User.hasMany(Habit, {
  onDelete: 'CASCADE'
});
Habit.belongsTo(User);

// Sync any changes to the database
sequelize.sync({ alter: true })
db.Habit = Habit;
db.CompletedTask = CompletedTask;
db.CompletedGoal = CompletedGoal;
db.User = User;

module.exports = db;

When we import this file into our main index.js, it'll connect to the database, initialize our models and sync any changes in our model definitions to our database tables. We can then import the db object and run a query like so:

const user = await db.User.findOne({
  where: { id: 1 }
 });

We'll be using queries like that in the next post to setup password authentication for our users.

DO Hackathon - The Middi Interface (Part 3)

Michael Mather — Thu, 17 Dec 2020 14:21:05 +0000

Before we can write a single line of code we have to have a good idea of what we're creating. We know our requirements and our stack from the previous posts, now it's time to mock up the UI.

Usually it's a good idea to start with a wireframe before getting straight into the colour mockups. Wireframes are better for iterating quickly on layout options. Since we're building a simple UI that is really only a chatbox, I figured we can move past the wireframe step and start with a full-colour mockup.

I would also recommend mapping out the user flows and how it relates back to our stories defined in the first post but since I don't expect that this app will be very groundbreaking I think it's best that we allocate our time elsewhere. There is plenty more to do.

Colours

I wanted to choose a colour palette that was welcoming, informal and warm to users. Since this will be a chatbot-like interaction, I wanted to give the app a friendly atmosphere.
I settled on a purple/pink palette:

I used the fantastic coolors.co to get this palette together

First draft

I went with a blurred background image to allow an interesting and fun background without taking focus away from the main chat window. I also used a sans-serif font that was playful to keep the "informal" aesthetic.

Here's a second design which includes a login form. This demonstrates my idea of keeping the entire app inside a chat window, which includes all forms and other components. To be honest, it's probably not amazing for user experience... but it's still an interesting proof of concept so I want to see what comes of it.

This next design includes a simple toggle for completing daily habits. It would be interesting to allow users to assign a color to a habit to personalize their experience.

Lastly, here's the overview component which shows a summary of a habit's progress. There is some flexibility here to show some interesting stats.

One thing not included in these designs that I wanted to keep in mind, is some fun animations/interactions to keep the app lighthearted and friendly. I want users to feel good when they complete a habit. So showing some sort of "celebration" animation when they complete a habit will be really important and there is a lot of opportunity to be creative.

Now that we've got a decent UI in mind, we can actually get into the code. In the next post we'll be looking at setting up the backend with authentication and database models.

DO Hackathon - Middi - The Stack (Part 2)

Michael Mather — Wed, 16 Dec 2020 13:25:58 +0000

Now that we know what we're going to build from my last post, we can figure out how we're going to do it. This is the fun part where we can imagine all the cool tools we're going to use without actually having to dig into any documentation.

The Frontend

One of our requirements is to make the user interface intuitive and fun to use. That means it needs to be responsive and fast. For this reason, I figured I'd be super original and go React - not sure if you've heard about it before.
I have past experience with React and know that I can get up and running quickly with create-react-app.

I'm not a huge fan of CSS frameworks like Tailwind and prefer to build the UI from the ground up. For this reason, I chose SASS to handle styling.

At first glance there doesn't seem to be any need for a state management system at the scale of Redux, so I think we'll stick with React's useContext and useReducer to handle state management.

The Backend

We already know we'll be using Digital Ocean's App Platform to host the application. For the API, I figured we'd keep things consistent end-to-end and use NodeJS and Express.

Since we'll be serving a frontend, and also hosting an API we have two options: serve the frontend on a separate App Platform, or use the API to serve the React bundle.
I wanted to keep all the code in one repo to make it easier for one-click-deployments in the event anyone else wanted to try it out. For this reason I decided to use one server to both serve the bundle and host the API.

The Database

Our requirements specify that a user should be able to sign up/log in and have their data persist between sessions. This means we'll need a database of some sorts. My intial thought was to use a MongoDB database because of how well it integrates with the rest of the stack we've discussed, but DO doesn't support Mongo in their database components for App Platform (although you can launch a Mongo droplet from the Marketplace). Instead, I decided to go with PostgreSQL. We can use Sequelize to create models and run migrations on our database from our backend.

Now that we've got a stack in mind, let's take a look at designing the user interface...

DO Hackathon - Middi Habit Tracker

Michael Mather — Tue, 15 Dec 2020 17:19:08 +0000

Idea Generation

In my mind, application development always boils down to two things: the What and the How. In this post, I'll be describing the former, which in my opinion, is way harder than building the thing itself.

Problem and Solution

One of the things I found difficult when shifting into working from home was adjusting the routine that I've been in for years. Granted, I'm very thankful to be able to work from home, so it's not the worst situation to be it, but it was clear that many people, including myself had trouble adjusting at first.

That's why for this hackathon I'll be building an ultra-simple habit tracker in the form of an intuitive, chatbot interface. The purpose of the application is to allow users to track habits in a way that isn't so "admin-dashboardy". Instead, the interface will be in the form of a chat app, with fun interactions and animations to make tracking habits feel good.

The Specifics - Requirements

Now that we have an idea of the purpose, let's get into the good stuff - requirements. A good starting point is to break it down into two categories: the needs and the nice-to-haves.

Needs

The web app needs a chatbot interface that feels natural to use
A user should be able to make an account
A user should be able to login/logout
A user should be able to create/edit habits
A user should be able to able to mark a habit as completed
A user should be able to view a summary of their habits

Nice-to-haves

Ability to set reminder notifications
Downloadable as a progressive web ap

Now that we've got a shortlist of features, we can start taking a look at the How: what we'll use to build the darn thing.