Forem: chester htoo

Deploying container applications on AWS with CI/CD pipelines

chester htoo — Fri, 10 Nov 2023 21:53:41 +0000

In this blog, we will be creating a cloud environment, specifically on Amazon Web Services, to deploy a web application, which is a simple Vite application. The Vite application will be containerised using Docker, and will be pushed into our Amazon ECR Registry, which will later be used by Amazon ECS task definition to run a service on ECS Fargate. We will also be setting up a CI/CD pipeline using Github actions so that whenever a change is committed to main branch (production), it will trigger an automatic docker image build process and update the ECS task to use the latest docker image from our ECR repository.

Architecture Overview

The architecture overview will look something like this.

Let's dive right into it.

Pre-requisites

Before we get into the good stuffs, first we need to make sure we have the required services on our local machine, which are:

Folder Structure

We will be using a (sort-of) monorepo approach for this project. We will have a terraform folder for our infrastructure, and an app folder for our web application. We will also have a .github/workflows folder for our Github Actions workflow files. So it will look something like this.

your-project
├── .github
│   └── workflows
├── app
└── terraform

Creating the Web Application

We don't need a fancy fully-functional web application for this project. We just need a simple web application that we can use to deploy, build docker image and make a few changes to test our CI/CD pipeline. So we will just be using a simple react application boilerplate created with Vite. You can create your own or use any other boilerplate you like. So let's go into our working directory (any folder you like) and create a new vite application. I will be using pnpm for this project and here is a link to their installation guide.

pnmp create vite app/
cd app/
pnpm install

We'll clean up a few changes in the App.tsx file and run pnpm run dev. If everything is working fine, you should be able to see the web application running on localhost:3000.

Resources: Don't worry about the codes, all the codes can be found on Github here. I will also be linking all the resouces either in the code commends or the end of the blog. Don't forget to star the repo and share this article if you find it useful 😄

Cool, great! Now we got the application up and running.

Dockerizing the Web Application

Now, let's create a dockerfile to create a docker image.

FROM --platform=linux/amd64 node:18-alpine
RUN ["npm" ,"install", "-g","pnpm"]
COPY package.json /vite-app/
COPY . /vite-app
WORKDIR /vite-app
RUN ["pnpm", "install"]
CMD ["pnpm", "dev"]
EXPOSE 8000

Let's go through the dockerfile line by line.

FROM --platform=linux/amd64 node:18-alpine: We are using the node:18-alpine image as our base image. We are also specifying the platform to be linux/amd64 because we will be using this docker image on ECS Fargate, which is a linux environment. If we don't specify the platform, it will default to my system's platform, which is MacOS M1 (darwin/arm64), and it will fail to run on ECS Fargate.
RUN ["npm" ,"install", "-g","pnpm"]: We are installing pnpm globally. We will be using pnpm to install our dependencies. You can use npm or yarn if you like.
COPY package.json /vite-app/: We are copying the package.json file to the /vite-app directory in our docker image.
COPY . /vite-app: We are copying the rest of the files to the /vite-app directory in our docker image.
WORKDIR /vite-app: We are setting the working directory to /vite-app.
RUN ["pnpm", "install"]: We are installing the dependencies.
CMD ["pnpm", "dev"]: We are running the dev script.
EXPOSE 8000: We are exposing the container port 8000.

We also don't want to include node_modules in our docker image, so we will add it to our .dockerignore file.

node_modules

Now, let's build our docker image using docker build -t vite-app:latest . (make sure you are in the app directory). You should be able to see the docker image when you run docker images.

REPOSITORY             TAG              IMAGE ID       CREATED        SIZE
vite-app               latest           4dd38de114b8   42 hours ago   390MB

Now, let's run the docker image using docker run -p 3000:8000 vite-app:latest. You should be able to see the web application running on localhost:3000 if you have setup everything correctly.

Setting up AWS Environment

Now, let's setup our AWS environment. We will be using Terraform to create our infrastructure. We will be creating the following main resources:

Amazon ECR private repository
Amazon ECS cluster
Amazon ECS task definition
Amazon ECS service
Some IAM roles and policies

Let's first create a couple of files that we plan to use in our terraform/ directory.

cd terraform/
touch main.tf providers.tf variables.tf outputs.tf main.tfvars iam.tf sg.tf vpc.tf

We'll be using Amazon Web Service provider for Terraform. So let's add the following to our providers.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.22.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

In our main.tf, we will be creating some of our core services, which are: Amazon ECR private repository, Amazon ECS cluster, Amazon ECS task definition, and Amazon ECS service.


module "vite_app_repository" {
  source  = "terraform-aws-modules/ecr/aws"
  version = "1.6.0"

  repository_name                 = "vite-app-repository"
  repository_type                 = "private"
  repository_image_tag_mutability = "IMMUTABLE"
  create_lifecycle_policy         = true

  # only keep the latest 5 images
  repository_lifecycle_policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Expire images by count"
        selection = {
          tagStatus   = "any"
          countType   = "imageCountMoreThan"
          countNumber = 5
        }
        action = {
          type = "expire"
        }
      }
    ]
  })

  tags = merge(var.common_tags)
}

resource "aws_ecs_cluster" "vite_app_cluster" {
  name = var.ecs_cluster_name
  tags = var.common_tags
}

resource "aws_ecs_task_definition" "vite_app_runner" {
  family                   = var.ecs_task_definition_name
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "512"
  memory                   = "1024"
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  container_definitions = jsonencode([
    {
      name   = var.ecs_container_name
      image  = "${module.vite_app_repository.repository_url}:latest"
      cpu    = 512
      memory = 1024
      portMappings = [
        {
          containerPort = 8000
          hostPort      = 8000
          protocol      = "tcp"
        }
      ]
      essential = true
    }
  ])
  tags = var.common_tags
}

resource "aws_ecs_service" "vite_app_service" {
  name            = var.ecs_service_name
  cluster         = aws_ecs_cluster.vite_app_cluster.id
  task_definition = aws_ecs_task_definition.vite_app_runner.arn
  launch_type     = "FARGATE"
  desired_count   = 1

  network_configuration {
    subnets          = module.vite_app_vpc.public_subnets
    security_groups  = [module.web_access_sg.security_group_id]
    assign_public_ip = true
  }

  tags = var.common_tags
}

So, let's go through our main.tf file.

First we are creating an ECR repository using the terraform-aws-modules/ecr/aws module. We are also creating a lifecycle policy to only keep the latest 5 images. We are also creating an ECS cluster, ECS task definition, and ECS service. We are also creating a security group for our ECS service to allow traffic from the internet (see below).

We also need to have some IAM permissions for us to allow ECS task to pull the docker image from our ECR repository. We will also need to create an IAM user for our Github Actions workflow to use to push the docker image to our ECR repository. We will be creating a couple of IAM roles and policies in our iam.tf file. I won't be putting the codes here on the blog but you can see the codes here on Github. We will also need to create a security group for our ECS service to allow traffic from the internet. We will be creating a sg.tf file for that. For the security group resource you can see here and for VPC resouce, you can get the terraform codes here.

We will also need to create a few variables, so let's create a variables.tf file and need to provide them depending on the different environments here. We will be using a main.tfvars file to store our variables for now since we only have one environment.

aws_region               = "eu-west-1"
ecs_task_definition_name = "vite-app-runner"
ecs_container_name       = "vite-app"
ecs_cluster_name         = "vite-app-cluster"
ecs_service_name         = "vite-app-service"

We will also need to create an outputs.tf file to output some of the resources that we will be using later.

output "ecr_repo_url" {
  value = module.vite_app_repository.repository_url
}

output "github_actions_user_access_key_id" {
  value = aws_iam_access_key.github_actions_user_access_key.id
}

output "github_actions_user_access_secret_key" {
  value     = aws_iam_access_key.github_actions_user_access_key.secret
  sensitive = true
}

Now that, we've setup the AWS infrastructure in place, let's run terraform init to initialise our Terraform project. Then, we can run terraform plan -var-file=main.tfvars to see what resources will be created. If everything looks good, we can run terraform apply -var-file=main.tfvars to create the resources.

Apply complete! Resources: 26 added, 0 changed, 0 destroyed.

Outputs:

ecr_repo_url = "********.dkr.ecr.eu-west-1.amazonaws.com/vite-app-repository"
github_actions_user_access_key_id = "**********"
github_actions_user_access_secret_key = <sensitive>

Now, we have our AWS infrastructure in place. We can now push our docker image to our ECR repository and run our ECS service.

Pushing Docker Image to ECR

Now, let's push our docker image to our ECR repository. First, we need to login to our ECR repository using aws ecr get-login-password --region your-region | docker login --username AWS --password-stdin your-account-id.dkr.ecr.your-region.amazonaws.com. Then, we can tag our docker image using docker tag vite-app:latest your-account-id.dkr.ecr.your-region.amazonaws.com/vite-app-repository:latest. Then, we can push our docker image using docker push your-account-id.dkr.ecr.your-region.amazonaws.com/vite-app-repository:latest. If everything is working fine, you should be able to see the docker image in your ECR repository. (You can also see the push command if you go to your ECR repository and click on View push commands)

If everything works as expected, you should be able to see a task running in your ECS cluster. If you go into the networking, you will see this:

If you open that Public IP in your browser, and go to port 8000, you should be able to see the web application running.

Github actions

We wouldn't want to do the entire process manually every time we make a change to our web application. So, let's create a github action that will do that for us. We will be creating a workflow that will detect changes that are pushed to the main branch, build the docker image, push it to our private ECR repository, and update the ECS service to use the latest docker image. We will be creating a deploy.yml file in our .github/workflows directory.

name: Push image to Amazon ECR and deploy to ECS

on:
  push:
    branches:
      - main
      - master

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4.1.1

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4.0.1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2.0.1
        id: login-ecr

      - name: Set outputs
        id: vars
        run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Build, tag and Push image to Amazon ECR
        id: build-and-tag-docker-image
        working-directory: ./app
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: ${{ secrets.AWS_ECR_REPOSITORY }}
          IMAGE_TAG: git-${{ steps.vars.outputs.sha_short }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "IMAGE_URI=${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }}" >> $GITHUB_OUTPUT

      - name: Download task definition
        run: |
          aws ecs describe-task-definition \
          --task-definition ${{ secrets.AWS_ECS_TASK_DEFINITION_NAME}} \
          --query taskDefinition \
          --output json > taskDefinition.json

      - name: Fill in the new image ID in the Amazon ECS task definition
        id: update-task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1.1.3
        with:
          task-definition: taskDefinition.json
          container-name: ${{ secrets.AWS_ECS_CONTAINER_NAME }}
          image: ${{ steps.build-and-tag-docker-image.outputs.IMAGE_URI }}

      - name: Deploy Amazon ECS task definition
        id: deploy-ecs
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1.4.11
        with:
          task-definition: ${{ steps.update-task-def.outputs.task-definition }}
          service: ${{secrets.AWS_ECS_SERVICE_NAME}}
          cluster: ${{secrets.AWS_ECS_CLUSTER_NAME}}
          wait-for-service-stability: true

Let's go through each step.

With the checkout action, we are checking out the code from the repository.
We need some sort of a programmatic access to our AWS account (which is why we created a github-actions-user IAM user in our iam.tf file earlier here), so we are configuring our AWS credentials using the aws-actions/configure-aws-credentials action.
We then need to login to our ECR repository using the aws-actions/amazon-ecr-login action.
We then need to build our docker image, tag it, and push it to our ECR repository. We are also using the git rev-parse --short HEAD command to get the short SHA of the commit that triggered the workflow. We will be using this short SHA as our docker image tag. We are also using the aws-actions/amazon-ecs-render-task-definition action to update our ECS task definition with the new docker image.
We then need to download our ECS task definition using the aws ecs describe-task-definition command. What this step will do is simply call the aws ecs describe-task-definition command and save the output to a file called taskDefinition.json.
We then need to update our ECS task definition with the new docker image. We are using the aws-actions/amazon-ecs-render-task-definition action to do that. We are also using the aws-actions/amazon-ecs-deploy-task-definition action to deploy our ECS task definition.

After waiting for a few minutes, you should be able to see the new docker image in your ECR repository. You should also be able to see the new docker image being used in your ECS task definition.

You will notice a few environment variables that are being used in this workflow. We will be storing these environment variables in our Github repository secrets. You can access your repository secrets by going to Settings > Secrets > New repository secret. We will be storing the following secrets:

Most of the secrets are from our terraform/main.tfvars file which are being used to pass as some sort of variables to our github action. We will also need to store our AWS credentials in our Github repository secrets.

Great, now that everything's in place. Let's test out our pipeline. We'll remove the smiley face from our App.tsx inside the vite application, and push it. As soon as we pushed it, we'll see it triggers the github actions.

After waiting for a while, let's go into our ECS console and go onto our newly running task, and click on the Public IP and go onto port 8000. You should be able to see the web application running without the smiley face.

Conclusion

And that's it! We've successfully created a CI/CD pipeline using Github Actions to deploy a simple web application to ECS. The application is a simple vite application that will be dockerized and pushed to ECR. The pipeline will be triggered on every push to the main branch. The pipeline will build the docker image, push it to ECR, and update the ECS service with the new image.

Let me know if you have any questions or suggestions. You can also find the codes on Github here. Don't forget to star the repo and share this article if you find it useful 😄

IAM Permissions Boundaries

chester htoo — Sat, 12 Aug 2023 21:35:44 +0000

I used to be a picky eater when I was young. Veggies like tomatoes, peas, and onions were always left untouched on my plate. My mom's solution was straightforward: eat your veggies or lose out on TV time. It was a cycle of negotiations and scoldings. One day, a guest was over for dinner, and the same old routine unfolded. However, this time, something interesting happened. The guest said, 'If you don't like it, you can leave it on the plate.' That sounded like music to my ears, but there was a catch. I knew that once the guest left, I'd still have to face my mom's disapproval, even though I had been given 'permission' not to eat my veggies.

This childhood memory reminds me of something quite technical - IAM Permissions Boundaries. In the realm of computer systems, it's like my mom setting the rules for my veggie intake, while the guest offered a contrasting 'permission'. But there's more to it than just veggies, and it all ties into how we manage access and actions within digital spaces.

Imagine you're playing a story mode game, something like God of War, which by the way was my favourite game growing up, on PlayStation, you go through the story mode to collect power-ups and souls to upgrade your weapons which can later be used in the game. Think of these as "permissions", something that gives you permissions to do certain actions, ie. use powers and deal more damages. But sometimes, when you're fighting a boss or doing some quests, you're not allowed to use those powers, even though you have unlocked it. If you're a keen story mode games player, I'm sure that you have experienced these sorts of scenarios.

Now let's get back to the real world of AWS. Consider a user or a role in AWS having multiple permissions to carry out actions, eg. being able to create a database or reading a secret text from the vault (Parameter Store), but bounded by a Permissions Boundary of only able to carry out actions that are database related. So in this case, even if the user has the ability to read the text from the vault, they can't do that, since they're bounded by a fence known as IAM Permissions Boundaries.

Now let's take this approach into AWS environment, our digital playground. Consider a user named John. John loves managing Amazon S3, CloudWatch and Amazon EC2 - her favourite parts of the AWS playground. To make sure he sticks to these areas, you, as the administrator has set up an IAM permissions boundary. This boundary says, "John, you can only play in Amazon S3, CloudWatch and Amazon EC2".

But remember, the boundary doesn't grant John the powers. He needs policies to do that. So, you create a policy that allows John to perform actions, both his policy and the Permissions Boundary need to agree. If they don't, his request is denied, just like I couldn't leave my plate without my mom's whopping.

However, IAM Permissions Boundaries aren't the only rule setters. There are other types of policies too, like resource-based policies and session policies. And sometimes, there might be conflicts between them. It's like having multiple referees in a football field, each enforcing their own rules of, in-game referee and referees from the sidelines.

So, to sum it all up, IAM Permissions Boundaries are like the rules that determine how far you can go on the AWS playground. They ensure that even if you have many different powers, you can only use the ones that are approved by all relevant policies.

In our next blog post, we will dive deeper into the security pillar of the AWS well-architected framework.

Deploy Next JS Application to Amazon CloudFront with S3

chester htoo — Tue, 18 Jul 2023 22:32:45 +0000

Picture this: You and your friend had launched your SaaS application and the entire globe's rushing to your platform. The application's written in the bleeding-edge technology, Next JS and hosted on AWS Amplify, in Europe region. After a couple days, you saw your inbox's flooded with angry emails from your customers saying the website takes a long time to load up. As a young CTO, challenges rise and you're now scratching your head not know how to improve the performance.

Behold: Edge locations

You then stumbled upon something called "Edge locations". What are they? Edge locations refer to global network of data centres strategically placed around the world and are designed to bring content closer to your end-users geographically. They reduce latency and improves the overall performance of content delivery.

Each edge location serves as a caching endpoint for content delivery networks (CDNs). In our case, our website may be hosted in Europe region, but if someone from Japan wants to access the website, the initial load takes a bit of time but after that, it caches the web page or a file on the closest edge location to the user. So next time when a user from that particular region wants to access your website, the network doesn't have to travel all the way across the world to Europe but can just get the content from that closet edge location, ready to be served and boost your sales.

So how are we going to achieve this? Simple. We will create an Amazon S3 bucket, enable static website hosting, sync our Next JS static files into that bucket and let the CloudFront do its thing.

Let's dive right into it.

Next JS Setup

First we'll create a working Next JS app with a few pages, so we'll create a new directory and use next-app template for it.

yarn create next-app nextjs-s3-cloudfront

Select the options that you want to use in creating Next JS application, I'll leave everything as default and use App Router rather than Pages Router.

Wait for a couple of minutes and you got your tiny little working Next JS application. So we'll go ahead and make some changes in the app/page.tsx.

This will now be what our app/page.tsx would look like.

import Image from "next/image";

export default function Home() {
  return (
    <main className="flex min-h-screen flex-col items-center justify-between p-24">
      <div className="lg:flex"></div>

      <div className="flex place-items-center before:absolute before:h-[300px] before:w-[480px] before:-translate-x-1/2 before:rounded-full before:bg-gradient-radial before:from-white before:to-transparent before:blur-2xl before:content-[''] after:absolute after:-z-20 after:h-[180px] after:w-[240px] after:translate-x-1/3 after:bg-gradient-conic after:from-sky-200 after:via-blue-200 after:blur-2xl after:content-[''] before:dark:bg-gradient-to-br before:dark:from-transparent before:dark:to-blue-700 before:dark:opacity-10 after:dark:from-sky-900 after:dark:via-[#0141ff] after:dark:opacity-40 before:lg:h-[360px] z-[-1]">
        <p className="text-4xl">Live long and prosper!</p>
      </div>

      <div className="mb-32 grid text-center lg:mb-0 lg:grid-cols-4 lg:text-left"></div>
    </main>
  );
}

And then let's head into next.config.js file to configure Next JS build setting to be output type of export. This is now what our next.config.js should look like.

/** @type {import('next').NextConfig} */
const nextConfig = {
  output: "export",
};

module.exports = nextConfig;

Let's build our Next JS application.

yarn run build

You'll notice a new folder appear which is out and if you open it, you will see a bunch of HTML files and _next static files. This will come in handy when we transfer them into S3 later.

AWS Resources Setup

We will then, create a new terraform file right inside the application to avoid having to create a mono-repo or another directory. In an actual working environment, what would be ideal is to create a separate folder and have the Terraform resources there.

mkdir terraform && cd terraform

And as usual, we will need 4 main files

providers.tf - To configure Terraform prodivers
main.tf - To provision resources
variables.tf - To use variables inside terraform files
output.tf - To get the URL of the CloudFront distribution (or any other properties that we want to check)

touch providers.tf main.tf variables.tf output.tf

We will need to grab our AWS provider Terraform registry into the providers.tf file along with your AWS Access Key and Secret Key (Read more on how to generate these keys here). Since I'm using AWS IAM Identity centre with SSO login, I won't be adding the code for Access key and Secret Key, but I'll leave the config as it is. You will need to create a main.tfvars to supply these values.

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.8.0"
    }
  }
}

provider "aws" {
  region     = var.aws_region
  access_key = var.access_key
  secret_key = var.secret_key
}

# variables.tf
variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "eu-west-1"
}

variable "secret_key" {
  type        = string
  description = "AWS Secret Key"
}


variable "access_key" {
  type        = string
  description = "AWS Access Key"
}

Now we will be using Terraform AWS S3 module and CloudFront module to provision our resources. The architecture here is to create a S3 bucket with static website hosting option, and then have our Next JS static files there and then use CloudFront to actually serve the content on the edge! We will be making use of custom Access Control Lists (ACLs) so that users cannot directly access the S3 static website URL rather than the CloudFront URL. Here's what our main.tf file should look like now.

module "s3_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "3.14.0"
  bucket  = "my-crazy-good-nextjs-bucket"
}

module "cloudfront" {
  source              = "terraform-aws-modules/cloudfront/aws"
  version             = "3.2.1"
  is_ipv6_enabled     = true
  enabled             = true
  price_class         = "PriceClass_All"
  retain_on_delete    = false
  wait_for_deployment = false

  create_origin_access_identity = true
  origin_access_identities = {
    "oai-nextjs" = "cloudfront s3 oai for nextjs website"
  }

  origin = {
    s3 = {
      domain_name = module.s3_bucket.s3_bucket_bucket_regional_domain_name
      s3_origin_config = {
        origin_access_identity = "oai-nextjs" # key from origin_access_identities map
      }
    }
  }

  default_cache_behavior = {
    target_origin_id       = "s3" # key from origin map
    allowed_methods        = ["GET", "HEAD", "OPTIONS"]
    cached_methods         = ["GET", "HEAD", "OPTIONS"]
    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
    compress               = true
  }

  custom_error_response = [
    {
      error_code         = 403
      response_code      = 403
      response_page_path = "/index.html"
    }
  ]

  default_root_object = "index.html"
}

data "aws_iam_policy_document" "s3_policy" {
  version = "2012-10-17"

  statement {
    sid       = "1"
    effect    = "Allow"
    actions   = ["s3:GetObject"]
    resources = ["${module.s3_bucket.s3_bucket_arn}/*"]
    principals {
      type        = "AWS"
      identifiers = module.cloudfront.cloudfront_origin_access_identity_iam_arns
    }
  }
}

resource "aws_s3_bucket_policy" "s3_policy" {
  bucket = module.s3_bucket.s3_bucket_id
  policy = data.aws_iam_policy_document.s3_policy.json
}

Let's go through them.

We have 2 modules, each using terraform module specified above. We give a name of my-crazy-good-nextjs-bucket for our S3 bucket. For CloudFront module, we set the OAI to be enabled so that users can only access the S3 website content from our CloudGront URL (Read more about Origin Access Identity, OAIs here). For the cache behaviours, we cache all the GET request to our website on an edge location and only allow HTTPS.

There's another data block which is the IAM policy document for our S3 bucket. The reason why we need to attach this policy is that, it is not a good idea to have our bucket publicly accessible from the internet and we only want to allow access from the ARN of the CloudFront OAI.

We also want to see some details of the resources that we provision after running terraform apply so let's create a outputs.tf file to grab some values from the provisioning.

# outputs.tf
output "s3" {
  description = "S3 module outputs"

  value = {
    bucket_id  = module.s3_bucket.s3_bucket_id
  }
}


output "cloudfront" {
  description = "Cloudfront module outputs"

  value = {
    distribution_id = module.cloudfront.cloudfront_distribution_id
    domain          = module.cloudfront.cloudfront_distribution_domain_name
  }
}

What we're doing in this file is pretty much letting terraform know that, we want these certain values to be shown in the CLI, after the resources have been provisioned.

That's pretty much it! Let's run terraform init and terraform plan. It will show us a bunch of resources that terraform will create. Normally this plan should be reviewed with other team members to finalise the changes but since it's only a small business, let's go ahead and run terraform apply.

After waiting for a couple of seconds, the resources will be provisioned and it will show us these output values.

Apply complete! Resources: 13 added, 0 changed, 0 destroyed.

Outputs:

cloudfront = {
  "arn" = "arn:aws:cloudfront::389144622841:distribution/E3QS5X2RJNINOF"
  "distribution_id" = "E3QS5X2RJNINOF"
  "domain" = "d11r27a15bgorv.cloudfront.net"
}

s3 = {
  "bucket_id" = "my-crazy-good-nextjs-bucket"
}

We will need 2 things which are the S3 bucket name and the CloudFront distribution ID. You'll also see a CloudFront domain from the outputs, but if you visit it now, you will see nothing but an Error page. This is because we have only created the resources but not moved the data from our Next JS app to our S3 bucket.

So for that, let's go back to our Next JS app by running cd ... We will be using AWS cli to copy our Next JS out/ for static HTML pages to our s3 by running

aws s3 sync out/ s3://my-crazy-good-nextjs-bucket

It will now copy all the content inside the Next JS export directory out to our newly created S3 bucket. Now that we got our required static pages, we will revalidate our CloudFront distribution with the new contents by running

aws cloudfront create-invalidation --distribution-id E3QS5X2RJNINOF --paths "/*"

Note: Paste the distribution ID from the outputs in the args of --distribution-id

Now if we visit our CloudFront domain again, we'll see our blazing fast Next JS website served from the edge.

That's it for this blog! All the codes can be found here on my Github!

Improvements

Of course, there are ways that we can improve this deployment further.

We can deploy on our own custom domain name. If we want to have a custom domain name, we need to name the S3 bucket as the same name as our domain's A record (ie. www.google.com), and have our certificate in the ACM.
Setting up CI/CD pipeline to automate static file transfer to S3 and revalidating the CloudFront cache every-time we push to our VCS could also help in our case if we want to automate the process.
We can even integrate this application with a custom CRM that we did on another blog-post and have our customers reach out to us.

That's it for now and I hope to see you in the next one! Ciao!

CRM with Lambda and Terraform

chester htoo — Fri, 14 Jul 2023 01:57:18 +0000

Many of us have visited websites, scrolled around and click about. If the website's interesting, you guys have also send inquiry on more details about the product on the website's little form that's called Contact Us.

So, today let's dive into building a minimal working backend service for contact us form and saving that inquiry into our CRM, Hubspot.

Technologies that we'll use are as follows:

Terraform (IaC - Resource provisioning)
AWS Lambda (Compute Infrastructure on the AWS)
HubSpot API (to save inquiries)

Setup

Let's start by creating a new working directory

mkdir contact-us
cd contact-us

The plan here is to create a minimal lambda function that saves the user data from the client application (WordPress website, wix, custom client website, etc) into our HubSpot CRM. So let's get straight into it.

Plan

We'll start off by provisioning our lambda function from Terraform.

touch main.tf providers.tf outputs.tf variables.tf

providers.tf - This file is for defining which Terraform providers that we want to use.
main.tf - Our resource provisioning logic will sit in this file (if it's a huge application, we would create separate module files)
variables.tf - This file is for defining variables that are needed for our terraform module.
outputs.tf - Any output data that we want after provisioning our resources.

We will need to grab our AWS provider Terraform registry into the providers.tf file along with your AWS Access Key and Secret Key (Read more on how to generate these keys here).

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.8.0"
    }
  }
}

provider "aws" {
  region     = var.aws_region
  access_key = var.access_key
  secret_key = var.secret_key
}

# variables.tf
variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "eu-west-1"
}

variable "secret_key" {
  type        = string
  description = "AWS Secret Key"
}


variable "access_key" {
  type        = string
  description = "AWS Access Key"
}

For our lambda function planning, we will be using terraform lambda module rather than the resource so that if we were to reuse, we can just use that particular module.

module "lambda" {
  source        = "terraform-aws-modules/lambda/aws"
  version       = "5.2.0"
  function_name = "contact-us"
  architectures = ["arm64"]
  runtime       = "nodejs18.x"
  handler       = "index.handler"

  attach_policy_statements = true

  policy_statements = {
    AmazonSSMReadOnlyAccess = {
      sid       = "AmazonSSMReadOnlyAccess"
      effect    = "Allow"
      actions   = ["ssm:Describe*", "ssm:Get*", "ssm:List*"]
      resources = ["*"]
    }
  }

  source_path = [{
    path = "${path.module}/functions/contact-us"
  }]

  create_lambda_function_url = true

  cors = {
    allowed_credentials = false
    allowed_headers     = ["*"]
    allowed_methods     = ["POST", "OPTIONS", ]
    allowed_origins     = ["*"] # We would only want to allow our domain here
    max_age_seconds     = 3000
  }
}

So let's go through the inputs line by line. First we define what source we are using from terraform module and its version. We then define what sort of architecture, programming language and runtime we're using for the lambda function. For permissions, we're using inline policy statements here, in which we're allowing access to SSM Parameter store, since we don't want to store the HubSpot API keys in the lambda function directly, and will be storing in the Parameter store. We want to create a lambda function url that we can invoke directly so we flag true for create_lambda_function_url (this is not the most ideal way, more on it later), followed by CORS config.

Let's run terraform init and get the required providers.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Great stuffs! Now we'll setup HubSpot.

HubSpot

HubSpot is a CRM platform with a lot of integrations and resources for marketing, sales and content management. The product that we want to focus on for this part is their CRM hub contacts. Here is their documentation on how to setup your HubSpot account.

In the hubspot app, we can create our own private app (which is similar to connected app if you've ever used SalesForce or think of it as a client), and then under scopes, choose crm.objects.contacts read/write access. That's it! We then get our own HubSpot Access key.

We will then store this key in our AWS Parameter store, and store it as an encrypted string.

That's it! Now we get into coding the actual lambda function.

Lambda

We'll create a new folder called functions and then a subfolder called contact-us. This is where the lambda function will sit. In there, we will create a package.json file by called yarn init -y and create a blank index.js with the following content.

const handler = async (event, context) => {};

module.exports = { handler };

We'll install 3 libraries, which are @aws-sdk/client-ssm to get our HubSpot access key and @hubspot/api-client to interact with HubSpot API.

yarn add @aws-sdk/client-ssm @hubspot/api-client

and this is what our index.js looks like

const AWS = require("@aws-sdk/client-ssm");
const hubspot = require("@hubspot/api-client");

const handler = async (event, context) => {
  const body = JSON.parse(event.body);

  const ssm = new AWS.SSM({
    region: "eu-west-1",
  });

  const hubspot_key = await ssm.getParameter({
    Name: "HUBSPOT_ACCESS_KEY",
    WithDecryption: true,
  });

  const hubspot_access_key = hubspot_key.Parameter.Value;

  const hubspotClient = new hubspot.Client({
    accessToken: hubspot_access_key,
  });

  await hubspotClient.crm.contacts.basicApi.create({
    properties: {
      email: body.email,
      firstname: body.firstname,
      lastname: body.lastname,
      phone: body.phone,
      message: body.message,
    },
  });

  return {
    statusCode: 200,
    body: JSON.stringify({
      message: "Thanks for contacting us! We will be in touch soon.",
    }),
  };
};

module.exports = { handler };

Now before we do terraform plan to see the changes that it's going to make, first we'll need to create main.tfvars and then give the AWS Access key and Secret Key, but personally, I have an IAM Identity Centre enabled on my personal organisation, so I will be skipping this.

Then we do terraform plan. It lists out a bunch of changes that terraform is planning to make, and if everything looks good, we can go ahead and do terraform apply. It will then apply the changes and now your lambda function will be live in no time!

So with our newly created Lambda function, let's test it out on Postman.

Great stuffs! Now we've successfully deployed our lambda function and if we go check to HubSpot, we'll also see a new contact added there.

So, that's it really. We've successfully built our own little contact us functionality, waiting to be integrated with your client applications!😄

Improvements

Sure, you wouldn't use this lambda function alone when your application grows bigger and bigger. There are definitely ways on how this can be improved.

We wouldn't use the lambda function URL as its own endpoint in big application. Instead, we can create an API Gateway that fronts the lambda function and set them as targets.
If we want the inquiries to be notified to us, we can either add SES service or Slack API to be notified in our own channel.
Setting up CD pipelines for lambda function on deploy is also another thing that can be improved.
In real world application, where there are a bunch of routes in the API gateway, we wouldn't use just a single Terraform file to deploy them. We would have a dedicated file structure on dealing with different terraform states for different functions and resources.

So that's the end of this little lab. I hope you guys enjoy it and I hope to see yous in the next one! Ciao!

Link to Github repo: https://github.com/halchester/contact-us-lambda-hubspot

Why Infrastructure-as-Code is a way to go

chester htoo — Mon, 03 Jul 2023 23:16:02 +0000

Picture this: You and your friend just launched an amazing SaaS web app. The response is overwhelming, with customers flocking to your platform from all corners of the globe. But here's the challenge: the surge in traffic is pushing your app to its limits, and you fear it might crash. As resourceful, albeit inexperienced, engineers, you scramble to provision additional instances to keep up with the demand. The plot thickens when you realise your customers are now spread across the world, requiring instances in various locations. It's like a wild adventure, searching high and low for server space to cater to their needs.

With a sense of urgency, you dash to the dashboard, determined to handle the mounting traffic. Region after region, you tirelessly replicate the same architecture, but it hits you: the repetition is daunting. A flicker of curiosity sparks within you, and you ponder if there's a better way to conquer this challenge.

Behold: Infrastructure as Code tools

In the pre-IaC era, deploying resources meant embarking on an endless clicking spree across multiple regions on the dashboard—a phenomenon aptly dubbed ClickOps. While ClickOps had its charm, human errors were always lurking around the corner.

But fear not! Engineers came to the rescue with the ingenious concept of Infrastructure as Code. This approach treats infrastructure provisioning, configuration, and management as code. Think of it as software engineering meets infrastructure magic, where programming languages (like the mighty Python, but not limited to it) and declarative configuration files or scripting languages take centre stage. These tools automate the creation and management of infrastructure resources such as servers, networks, and storage.

Enter the stages of the Development lifecycle. Day 0 marks the grand planning phase, where the architecture's foundation takes shape and high-level overviews come to life. Day 1 witnesses the successful deployment of apps, with the Board of Directors applauding the team's performance. But the true challenge lies in Day 2 and beyond—the maintenance and patching of the infrastructure resources while ensuring reliability, stability, and top-notch performance.

Here's where the Operations team realizes that manually clicking through tens of thousands of deployed resources is simply impractical. Enter the superhero IaC tools, ready to save the day from Day 0 itself.

Provisioning

Now, let's dive into two resource provisioning tools: Terraform and AWS CDK.

Terraform

Terraform, developed by HashiCorp, is an open-source infrastructure provisioning tool. With its simple and readable syntax, you can define your infrastructure resources in code, creating an execution plan that automates the provisioning and management process. It's versatile, supporting multiple cloud providers like AWS, Azure, and Google Cloud Platform.

AWS CDK

AWS CDK, on the other hand, is Amazon Web Services' Cloud Development Kit. This open-source framework lets you define your cloud infrastructure using familiar programming languages such as TypeScript, Python, or Java. By writing code that represents your resources, you can leverage programming language features to manage complex setups effortlessly. Under the hood, CDK uses AWS CloudFormation to create and manage your defined resources.

Both Terraform and AWS CDK provide powerful options for infrastructure provisioning. The choice between them ultimately depends on factors like your coding preferences and the complexity of your infrastructure setup. No matter which tool you choose, embracing infrastructure as code will unlock automation, scalability, and efficient resource management for your applications.

Configuration Management

In the realm of IaC tools, configuration management tools are also superheroes when it comes to managing your resources. Now, let's explore two of them: Ansible and Puppet, each with its own unique approach.

Ansible

Ansible, my personal favorite, is an open-source automation tool that focuses on simplicity and ease of use. It empowers you to define and manage infrastructure configurations through "playbooks" written in YAML. With Ansible, tasks like package installation, configuration file management, and service deployments can be effortlessly automated across multiple servers.

One of Ansible's standout features is its agentless architecture. By leveraging SSH connections, it eliminates the need for additional software or agents on the managed nodes. Ansible playbooks are designed to be idempotent, ensuring you can safely run them multiple times without unintended side effects.

Puppet

Puppet, on the other hand, provides a robust configuration management solution. Using a declarative language, either Puppet DSL or Ruby code, you describe the desired state of your infrastructure. Puppet's client-server architecture involves a central Puppet master server and Puppet agent nodes on managed servers. This setup allows for centralized management and consistent enforcement of configurations across your infrastructure.

Puppet boasts an extensive ecosystem of pre-built modules and resources, enabling you to manage various aspects of your infrastructure. From users and packages to services and files, Puppet covers a wide range of configuration needs. It also offers reporting and auditing capabilities to track changes and ensure compliance.

Whether you lean towards Ansible's simplicity or Puppet's scalable architecture, both tools are invaluable for automating configuration management. By embracing these tools, you can streamline operations, achieve consistency, and simplify the maintenance of your infrastructure. It's time to bid farewell to manual configurations and welcome the efficiency of configuration management tools like Ansible and Puppet.

What now?

The world of infrastructure management has been revolutionised by Infrastructure as Code (IaC) tools. With the rise of platform engineering practices, engineers realised that manual configurations and repetitive tasks were holding them back. By embracing IaC tools, such as Terraform, AWS CDK, Ansible, and Puppet, teams can automate resource provisioning, simplify configuration management, and ensure the stability and scalability of their infrastructure.

Gone are the days of ClickOps, where clicking through endless dashboards was the norm. With IaC, engineers can define their infrastructure resources in code, leveraging the power of programming languages and declarative configuration files. This shift brings software engineering principles to infrastructure management, enabling teams to treat infrastructure as software.

By adopting IaC tools, teams can tackle the challenges of scaling their applications, managing resources across multiple regions, and maintaining consistency and reliability. Whether it's provisioning instances, automating configuration tasks, or ensuring the desired state of infrastructure, IaC tools provide the means to navigate the dynamic landscape of IT operations.

So, in your quest for seamless infrastructure management, remember to harness the power of Infrastructure as Code. Embrace the automation, scalability, and efficiency it brings to your operations. Say goodbye to manual configurations and repetitive tasks, and welcome a world where infrastructure is as agile and adaptable as the applications it supports.

In my next blog, I'll explain how to use Terraform to provision resources on AWS and also another blog on Kubernetes. So please leave some feedback, or follow for more Cloud content.

I passed Certified Solutions Architect Exam in 4-ish months

chester htoo — Sun, 18 Jun 2023 08:39:15 +0000

A little bit of backstory

I'm a software engineer/student who's been working in the industry for a couple of years now and I've been purely on the software side of things. Before 2021, I have no idea what cloud is. I've used S3 for one of my backend services to store photos but I have no idea how the whole cloud or AWS thingy works. These were all buzzwords for me.

Spoiler alert: It's just someone else's computer.

There are plenty of cloud vendors out there, from small startups to big providers from companies like Amazon, Microsoft and Alibaba, and each platform has their own version of certification that determines the credibility of an engineer using that particular platform and foundational knowledge of the cloud.

But why is the industry standard of cloud certifications sort of lean towards AWS and why I personally took one of AWS' associate level certification?

Amongst all cloud providers like Azure, GCP and AWS, Amazon was one of the earliest players in the cloud computing field, and they're still dominating the current cloud market at a whopping 32% this year.

Personal Motives

Personally, I strongly believes in real world problem solving skills are better than traditional education of degrees and certifications, but what makes me go ahead and break my own beliefs? Because AWS certifications are aimed towards solving real world problems rather than just knowing theories of how each services work.

It does not ask you things that you can normally go and look up in the documentation (well, not entirely true, keep reading, I'll get into it in a bit). It asks you how you will handle scenario x with y requirements when you have to focus on z. Let's say a customer comes up to you and ask how you are going to build a frontend application that involves GraphQL and React, you provide a solution of how that application should look like with simple and resilient architecture at a minimum cost.

Different certifications

AWS offers certifications at 4 levels, Practitioner level, Associate level, Professional level and Speciality level. Some of the exams focus on building solutions for customer specific problems, some on DevOps like how we would leverage AWS to build a better performant applications and some on specific fields such as Databases, SAP and Machine Learning. I took the Certified Cloud Practitioner exam and Solutions Architect associate exam. I am not going to dive into all these exams but here is a link if you want to know more.

What is the Solutions Architect Exam

The Solutions Architect exam has two levels, associate level (the easier one that I took) and the professional level. I personally took this exam to get a deeper understanding of how AWS services work in conjunction with one another and how I can provide better solutions for the customers. This exam focuses on more technical side of cloud services offered by AWS and a bit harder than the practitioner exam.

To be completely honest, it took me 6 months to prepare for this exam rather than 4 months (sorry, wasn't a clickbait xD) because I spent the first 2 months procrastinating. But do keep in mind that it took me 4 months to pass this exam because I am a full-time student and working part-time as a Software Engineer who has no idea how AWS works and trying to maintain a good GPA to keep my asian parents happy. This timeline can vary to different people on how much time they can commit to studying, past experiences and level of expertise using Amazon Web Services.

How to prepare for the exam

There are 3 things to keep in mind when you are preparing for any sort of certification exams (or any exams in general).

Domain Knowledge of the field
Hands-on (practical) labs and
Practice Exams

Usually I would recommend using online learning platforms like Udemy, youtube and google, but whizlabs.com offers a great platform for people who are taking certification exams. Of course there are other platforms like O'Reilly and Udemy, but back then whizlabs were having a sale so I just got that.

I used whizlabs.com to study for the domain specific knowledge. They have great video lectures on how different services work, explain really well and touch on almost all domain knowledge that the exam can ask for. It even explains on how you would use it in the real world application and how you can pair it with some other services to create a better product with great performance.

Having just the domain knowledge is not definitely enough to pass the exam, that's for sure. That's when you need to get your hands dirty. There are plenty of free platforms that you can use to practice different architectures like hosting a website on S3 and serving it though CloudFront or building a serverless API that the public can access, with AWS workshop or using AWS' free tier plan. I used whizlabs' demo environment to practice my understanding of how different architectures work since it is also included in my purchase.

Finally I cannot stress this enough but practice exams. Do your practice exams! By do your practice exams, don't just go through questions and click "Reveal Answer" and reading the explanation. I would really recommend to take the practice exam under strict exam condition, like not leaving up to take a leak or use your phone every 30 minutes or so (yes, I know. been there, done that). And after each exam questions, always review what and how it went wrong and revise that particular topic back. There are practice exams that you can purchase from Udemy. I personally recommend practice exams by Tutorial dojo.

Tips and Tricks

Now that you've prepared for the exam with sleepless nights and pulled every hair out of your head trying to understand what different services are, now is the time to take the exam. Here are some tips and tricks (I am not going to give you generic tips like eat well before exam, drink water and blah blah.)

You probably will not understand all the technologies that are used in a question like RabbitMQ and Apache Kafka, but when you see something that you have no idea what that is, my advice is to take an educated guess. There is definitely no way that you can use RDS with a key-value database, nor a way to use a structured data in a Dynamo Database.

Some of the Multiple choice questions are really really ambiguous but they always have one single word that shifts the possibility of the right answer towards one option. What I would recommend is to look for keywords like graph, relational, disaster recovery, etc.

You will see questions that has phrases like "most cost efficient way" or "most secure". These questions test your scope of focus and priority. You wouldn't choose EFS over EBS if your main focus is cost, like you wouldn't choose Security Groups over NACL for network level security. Look for what the questions' focuses are.

One another personal trick is to keep yourself entertained during the entire 2 hours. I know that the exam can be stressful and all that but for someone who has really really short attention spam, sometimes it's hard for me to focus on one thing for a long time. Normally what I do is, after every 10-15 questions answered, I always close my eyes and let my mind go wild for 5-10 seconds, and then focus back. This not only helps me with my attention deficiency but also helps with my eyes.

Now that you're all prepared for the exam, you can chill and ace the exam. Good luck folks! May the Jeff be with you!

Reflections

Looking back, I probably could've done the exam much faster than 4 months if I had spent more time on the labs and then have some blocked out time during the day to study. But I'm happy that I passed the exam as well as the Certified Cloud Practitioner exam. Now I'm studying for the professional level Solutions Architect exam and once I passed, I will create more blogs like this. In the meantime, since this is my first ever blog post, feel free to let me know if there are any mistakes or more tips!