Forem: Daniel Slapelis

Building a Simple CI/CD Pipeline with GitLab, Terraform, and Amazon Web Services

Daniel Slapelis — Wed, 12 May 2021 13:17:29 +0000

How to build a CI/CD pipeline using GitLab for your business's website.

Prerequisites

A GitLab account
An AWS account
Terraform is installed
A KeyBase account
A domain managed in Route53
An ACM certificate for your domain.

Set up the infrastructure

We'll be using Terraform to build out the infrastructure. For the website, all we'll need is an S3 bucket and a CloudFront deployment.

Create a file named main.tf and paste this into. You can change the bucket name to whatever you want, just make sure you set this correctly later on in another file (you'll see).

variable "bucket_name" {
  default = "website.example.com"
}

variable "cnames" {
  type    = list(string)
  default = ["example.com", "www.example.com"]
}

variable "certificate_arn" {
  default = "arn:::acm::example23132423"
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "bucket" {
  bucket = "${var.bucket_name}"
  acl    = "private"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "AddPerm",
          "Effect": "Allow",
          "Principal": "*",
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::${var.bucket_name}/*"
      }
  ]
}
EOF

  website {
    index_document = "index.html"
    error_document = "index.html"
  }

  tags = {
    billing = "${var.billing_tag}"
  }
}

locals {
  s3_origin_id = "S3-${var.bucket_name}"
}

resource "aws_cloudfront_distribution" "s3_distribution" {
  origin {
    domain_name = "${aws_s3_bucket.bucket.bucket_regional_domain_name}"
    origin_id = "${local.s3_origin_id}"
  }

  wait_for_deployment = false

  enabled = true
  is_ipv6_enabled = true
  default_root_object = "index.html"

  aliases = "${var.cnames}"

  default_cache_behavior {
    allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods = ["GET", "HEAD"]
    target_origin_id = "${local.s3_origin_id}"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl = 0
    default_ttl = 3600
    max_ttl = 86400
  }


  price_class = "PriceClass_100"

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  custom_error_response {
    error_code = 403
    error_caching_min_ttl = 0
    response_code = 200
    response_page_path = "/index.html"
  }

  viewer_certificate {
    acm_certificate_arn = "${var.certificate_arn}"
    ssl_support_method = "sni-only"
    minimum_protocol_version = "TLSv1.2_2018"
  }
}

Now create the bucket

terraform init
terraform apply -auto-approve

Awesome! All of our infrastructure is set up in AWS and now we need to set up our GitLab runner!

Configure our runner

In GitLab, create a repository for your project, or use an existing repository. We just need to do a few things before our app is ready to deploy.

First, let's set up our runner. We're going to use a shared runner from GitLab. They're free to use up for up to 2,000 minutes of deployments per month -- and they're enabled by default.

We need to give it an AWS account to use to deploy to S3. Create another terraform config with this content:

variable "keybase_user" {
  description = "A keybase username to encrypt the secret key output."
  default     = "dannextlinklabs"
}

provider "aws" {
  region = "us-east-1"
}


resource "aws_iam_access_key" "gitlab_ci" {
  user    = "${aws_iam_user.gitlab_ci.name}"
  pgp_key = "keybase:${var.keybase_user}"
}

resource "aws_iam_user_policy" "gitlab_ci" {
  name = "gitlab-ci-policy"
  user = "${aws_iam_user.gitlab_ci.name}"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::website.example.com/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "cloudfront:*",
            "Resource": "*"
        }
    ]
}
EOF
}

resource "aws_iam_user" "gitlab_ci" {
  name = "gitlab-ci"
}

output "access_key" {
  value = "${aws_iam_access_key.gitlab_ci.id}"
}

output "secret_access_key" {
  value = "${aws_iam_access_key.gitlab_ci.encrypted_secret}"
}

Make sure you set the keybase user to your own keybase user. Okay whatever, run the config.

terraform init
terraform apply -auto-approve

The terraform config returns an access key and a secret key for this user. We need to decrypt the secret key with the command (this is why you needed to use your own keybase user).

terraform output encrypted_secret | base64 --decode | keybase pgp decrypt

Now that we have the access key and the secret key for our GitLab user, we need to supply those variables to our runner by adding them to the variables section in the CI/CD settings.

We set three variables:

AWS_ACCESS_KEY_ID - to the key we just got
AWS_SECRET_ACCESS_KEY - to the key we just got
AWS_DEFAULT_REGION - us-east-1

Run a deployment

GitLab CI/CD is based around a file called .gitlab-ci.yml. Our file needs to look like this:

stages:
  - deploy-s3
  - deploy-cf

variables:
  AWS_BUCKET: website.example.com

deploy_s3:
  image: python:3.6
  stage: deploy-s3
  tags:
    - docker
    - gce
  before_script:
    - pip install awscli -q
  script:
    - aws s3 sync . s3://$AWS_BUCKET/ --delete --acl public-read
  only:
    - master

deploy_cf:
  image: python:3.6
  stage: deploy-cf
  tags:
    - docker
    - gce
  before_script:
    - pip install awscli -q
  script:
    - export distId=$(aws cloudfront list-distributions --output=text --query 'DistributionList.Items[*].[Id, DefaultCacheBehavior.TargetOriginId'] | grep "S3-$AWS_BUCKET" | cut -f1) 
    - while read -r dist; do aws cloudfront create-invalidation --distribution-id $dist --paths "/*"; done <<< "$distId"
  only:
    - master

This gitlab-ci file sets up two stages: deploy-s3, and deploy-cf. The first stage uploads our application to our S3 bucket, and the second invalidates the CloudFront zone for that bucket to present the new changes to our website!

This simple configuration is all you need to have a complete CI/CD pipeline for your business' website.

This post first appeared on our blog where we write about devops and devops consulting services.

Serverless contact form using AWS Lambda, API Gateway, and SES

Daniel Slapelis — Fri, 07 May 2021 14:50:54 +0000

How to set up an email contact form using AWS Lambda, API Gateway, and SES.

Maybe you went through my last article on how to build a CI/CD pipeline for your website using GitLab, or maybe you just have a website or application in an S3 bucket that you want to add a contact form for. Either way, you've come to the right article -- today I'll show you how to set up this reliable contact form using Lambda, API Gateway, and SES (and, boy, is it easy).

Dan, what's the point?

When you have a traditional website on a server that uses things like Apache and PHP, you can just use PHP's mail() function if you setup an email service like Postfix. If you're serving your website from a bucket like I do, you can't exactly use Postfix (nor do I really want to because then I have to worry about the availability of another server). That's when this combination of services comes in handy!

Prerequisites

There aren't too many prerequisites for this (especially if you went through my last tutorial).

An AWS account
Terraform is installed
Simple Email Service is configured in AWS for a domain or email account

So for number 3, we need that set up because we'll need to use this service to send emails when someone fills out our contact form. I won't go through it because it's super easy, but if you need to do it, here's the AWS documentation that you can check out.

The infrastructure

I went through how to do this by hand using this guide, but I always try to recreate my infrastructure with Terraform so that if I ever need it again, I can quickly use a module I have written to spin up the same thing...so luckily for you, you don't need to go through that guide! You'll just use the module I wrote.

Go ahead and create your main.tf file in your Terraform project directory. Let's make it look like this:

provider "aws" {
  region = "us-east-1"
}

module "las" {
    source = "git::https://gitlab.com/nextlink/lambda-api-sendmail.git?ref=v1.0"
    role_name = "las_role"
    function_name = "lambda-function-name"
    billing_tag = "las"
    api_gateway_name = "las"
    api_gateway_description = "API gateway for las"
}

This is all the code we need to write to build our infrastructure. Can you believe it? The module is doing a lot of work behind the scenes to make our lives easier. Feel free to change those variables to whatever you need or want them to be.

Besides this Terraform config, we just need to do a couple more things. Our Lambda function is defined by a file called exports.js. I have a copy of it here. You should copy the source code and create a new file in your working directory called exports.js. You need to do just a little bit of work, though. Lines 4 and 5 in that file look like this:

var RECEIVER = "YOUR_EMAIL@example.com"
var SENDER = "no-reply@example.com"

You'll need to change the RECEIVER variable to be the email you want to receive contact forms at, and SENDER to be the email that you want to send emails from. This is why we set up SES earlier -- so that AWS can send emails from our domain to us.

After that's all done, we have one more task to do. We need to zip up the exports file using this command:

zip exports.js.zip exports.js

After that's done, all that's left to do is to run the config.

terraform apply

Awesome! Our infrastructure is all set up!

Using the infrastructure

I won't get too into the HTML and JS side of this because how you do this will vary depending on what framework your website/app uses, but I want to show you what endpoint you need to call in order to send a message.

The Invoke URL in that image is what we will POST our completed form to. The only stipulation with our form (due to the exports.js file we used in the Terraform config) is that we post JSON data to that url with these fields:

{
  "name": "",
  "email": "",
  "message": ""
}

If those fields aren't what you want, you can change them back in the exports.js file, zip it, and rerun the Terraform config.

And that's that

With that short Terraform configuration, and a little HTML/JavaScript magic, we have an endpoint that is always available (and is really inexpensive to use) so that our users can reach out and contact us! This method is a great alternative to third-party paid solutions, and leaves a lot of room for customization. I hope this example can be of use to you in your future creations.

This post first appeared on our blog where we write about devops and devops consulting services.

Building a Kubernetes CI/CD Pipeline with GitLab and Helm

Daniel Slapelis — Fri, 07 May 2021 14:10:46 +0000

Everyone loves GitLab CI and Kubernetes.

GitLab CI (Continuous Integration) is a popular tool for building and testing software developers write for applications. GitLab CI helps developers build code faster, more confidently, and detect errors quickly.

Kubernetes, popularly shortened to K8s, is a portable, extensible, open-source platform for managing containerization workloads and services. K8s is used by companies of all sizes everyday to automate deployment, scaling, and managing applications in containers.

The purpose of this post is to show how you can bolt on the Continuous Delivery (CD) piece of the puzzle to build a CI/CD pipeline so you can deploy your applications to Kubernetes. But before we get too far, we're going to need to talk about Helm, which is an important part of the puzzle.

What the Helm?

Helm calls itself "the package manager for Kubernetes". That's a pretty accurate description. Helm is a versatile, sturdy tool DevOps engineers can use to define configuration files in, and perform variable substitution to create consistent deployments to our clusters, and have different variables for different environments.

It's certainly the right solution to the problem we're covering here.

How do we do it?

First off, a few prerequisites. You’re going to have to have this all hammered out before you started with the project. There’s links to helpful documentation below if you need help.

You already have an Amazon EKS cluster.
You already know how to use GitLab CI.
You have a GitLab CI runner configured in your Kubernetes cluster.
You have the AWS Load Balancer Controller running in your cluster.

With those boxes checked, we can get started. You'll want to create a new repository in GitLab first for us to use in this example. Once you've done that we can get started with creating our files.

File tree

Basically, at the end our folder/file structure is going to look like this:

<dir>
├── chart/
|   ├── Chart.yaml
|   ├── values.yaml
|   └── templates/
|      ├── deployment.yaml
|      ├── service.yaml
|      ├── ingress.yaml
|      └── configmap.yaml
└── gitlab-ci.yml

values.yaml

applicationName: my-first-app
certArn: your-certificate-arn
domain: your domain name
subnets: your subnets
securityGroups: your security groups

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.applicationName }}
  namespace: {{ .Values.applicationName  }}
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: {{ .Values.applicationName }}
  template:
    metadata:
      labels:
        app: {{ .Values.applicationName  }}
    spec:
      containers:
        - name: {{ .Values.applicationName }}
          imagePullPolicy: Always
          image: nginx:1.19.4
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /usr/share/nginx/html/index.html
              name: nginx-conf
              subPath: index.html
      volumes:
        - name: nginx-conf
          configMap:
            name: {{ .Values.applicationName  }}-configmap

This is the configuration file that defines our deployment. You can see there are a few lines with {{ some text }}. This is how we use a variable we define in our values file within our chart.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Values.applicationName }}-configmap
  namespace: {{ .Values.applicationName }}
data:
  index.html: |
    <html>
    <head>
      <h1>My first Helm deployment!</h1>
    </head>
    <body>
      <p>Thanks for checking out my first Helm deployment.</p>
    </body>
    </html>

This config map just defines a simple index page that we'll display for our app.

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.applicationName }}
  namespace: {{ .Values.applicationName }}
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: {{ .Values.applicationName }}
    - port: 80
      targetPort: 80
      protocol: TCP
      name: {{ .Values.applicationName }}
  type: NodePort
  selector:
    app: {{ .Values.applicationName }}

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ .Values.applicationName }}
  namespace: {{ .Values.applicationName }}
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/subnets: {{ .Values.subnets }}
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/security-groups: {{ .Values.securityGroups }}
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn:  {{ .Values.certArn }}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
  rules:
    - host: {{ .Values.applicationName }}.{{ .Values.domain }}
      http:
        paths:
        - path: /*
          backend:
            serviceName: ssl-redirect
            servicePort: use-annotation
        - path: /*
          backend:
            serviceName: {{ .Values.applicationName }}
            servicePort: 80

.gitlab-ci.yml

stages:
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375/
  DOCKER_DRIVER: overlay2
  APP_NAME: my-first-app

deploy:
  stage: deploy
  image: alpine/helm:3.2.1
  script:
    - helm upgrade ${APP_NAME} ./charts --install --values=./charts/values.yaml --namespace ${APP_NAME}
  rules:
    - if: $CI_COMMIT_BRANCH == 'master'
      when: always

Okay we have all the files. Now what?

Well, after you have all the files defined and your infrastructure follows our prerequisites, there's not much left to do.

If you commit these files, GitLab will interpet your .gitlab-ci.yml file and initiate a pipeline. Our pipeline only has one stage and one job (deploy). It'll spin up a container in the cluster for the deployment using the helm:3.2.1 image and run our script command. This does all of the heavy lifting for us with creating all of the files required in our namespace and starting our application.

If you configure in Route53 a DNS record like my-first-app.my-domain.com with an A record to the load balancer that the ingress controller created, you'll see the index page we defined in the configmap!

This post first appeared on our blog where we write about devops and devops consulting services.