Forem: ChigozieCO

Dockerized Deployment of a Full Stack Application with Reverse Proxy, Monitoring & Observability

ChigozieCO — Sat, 04 Jan 2025 22:55:45 +0000

The goal here is to provide a robust and scalable application infrastructure while showcasing best practices in containerization, monitoring, and cloud deployment. By the end of this project, users will have a fully functional web application deployed on a cloud platform with proper domain configuration, detailed logging, and real-time performance monitoring.

Project Overview

This project demonstrates the deployment of a full-stack application with a FastAPI backend and a React frontend using Docker. It integrates a reverse proxy for routing, and implements comprehensive monitoring and observability using Prometheus, Grafana, Loki, Promtail, and cAdvisor. We will eventually host our application from a cloud platform.

Prerequisite

Docker and Docker Compose: Installed and configured on your system.
Basic Knowledge of Docker: Understanding containerization concepts and Docker CLI commands.
Git: Installed to clone the project repository.
Code Editor: Like VS Code or any preferred IDE.
System Requirements: At least 4GB RAM and a stable internet connection for pulling images and dependencies.
AWS Account and AWS CLI: Required for deploying the application to the cloud.
Observability Tools Knowledge: Familiarity with Prometheus, Grafana, Loki, cAdvisor, and Promtail for monitoring and observability setup.

Clone Application Repo

The application we are to deploy can be found here. To begin the project we will crone the repo into an empty directory using the below command:

git clone https://github.com/The-DevOps-Dojo/cv-challenge01.git .

The . at the end of the command is to ensure that the repo is cloned directory into the directory where we are, without an additional folder.

Let's begin.

Containerization

This project leverages Docker to containerize the frontend, backend, database, a database management tool, a reverse proxy and monitoring and observability (Prometheus, Grafana, cAdvisor, Promtail and Loki).

Using docker compose, the services are orchestrated to run seamlessly in isolated environments, enabling consistent and efficient development, testing, and deployment.

Now we will go ahead and write the Dockerfile for each of the services starting from the backend, frontend, database Adminer; the and database management tool.

Dockerize Backend

We will dockerize the backend now by writing the Dockerfile we will use to create our image. Navigate in the backend directory and create a new file Dockerfile. You can find all the code used in this project here.

Using a multi-stage Docker build I was able to reduce my image size by 70.99%, initially starting from a size of 262Mi to a final size of 76Mi as can be seen in the screenshots below:

Version 1

Multi-stage Version 2 Build

My Dockerfile can be found here

Add the below code into the Dockerfile you created:

# Stage 1: Builder stage, Base image for the backend
FROM python:3.10-slim AS builder

# Install system dependencies
RUN apt-get update && apt-get install -y \
    curl \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

# Add poetry to PATH
ENV PATH="/root/.local/bin:$PATH"

# Set working directory and copy only dependency files first
WORKDIR /backend
COPY pyproject.toml poetry.lock* ./

# Install dependencies with Poetry
RUN poetry config virtualenvs.create false && poetry install

# Stage 2: Slim final image
FROM python:3.10-slim

# Install system dependencies
RUN apt update && apt install -y \
    curl \
    postgresql-client \
    && rm -rf /var/lib/apt/lists/*

# Install Poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

# Add poetry to PATH
ENV PATH="/root/.local/bin:$PATH"

# Set working directory
WORKDIR /backend

# Copy virtual environment from builder stage
COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy application files
COPY . .

RUN poetry config virtualenvs.create false

# Expose application port
EXPOSE 8000

# Set the default command to run the application
CMD ["poetry", "run", "uvicorn", "app.main:app", "--reload", "--host", "0.0.0.0", "--port", "8000"]

✨ I opted to use the slim variant of the python image as it is lightweight while still being developer-friendly and avoids common build issues.

It has better compatibility than the Alpine variant, few dependency issues, moderately sized and has better ease of debugging than Alpine does.

✨ I installed curl as I would be needing it for the installation of Poetry, then installed Poetry and added it to PATH.

✨ We used the ENV command to set the $PATH variable globally, making sure Poetry is available in all subsequent layers and for the runtime environment.

✨ Like earlier mentioned, I used a multi-stage build, in the first stage, I installed poetry and in the 2nd stage I copied environment with poetry installed to the final image, however seeing as we will still need to use poetry to set up the database with the necessary tables we will install poetry in the second stage.

This has minimal impact on the size of the image so we are in the clear.

Build Backend Image

We don't necessarily have to build the image but if you want to build it for testing purposes, navigate into the backend directory (where you have your backend Dockerfile) and use the command:

docker build -t devopsdojo/backend:v1 .

⚠️ NOTE:

Ensure docker is running before running the above command otherwise you will get an error message.

To check the size of your image you can use the below command:

docker image inspect devopsdojo/backend:v1 --format='{{.Size}}' | numfmt --to=iec-i

As an aside (in preparation of the next phase of this project) you can push your image to your Docker Hub repo, we will use the image from this Docker Hub repo when we want to automate this project.

Dockerize Frontend

Just as we did with the Backend, we will write a Dockerfile that we will use in building our frontend image. Navigate into the frontend directory and create a new file Dockerfile.

Even though I opted to use the slim variant of the node image and also use a multi stage build as with my backend image I was not able to substantially reduce the size of my frontend image, I did manage to reduce it from over 1gb to 850mb though but I would have loved to further trim it down.

This dockerfile is pretty straightforward, install npm, expose the port (merely for documentation) and start the container.

Find my Frontend Dockerfile here.

Enter the below code into your Dockerfile:

# Base image for the frontend
FROM node:22-slim AS base

# Set working directory
WORKDIR /frontend

# Install debugging tools
RUN apt update && apt install -y bash net-tools && rm -rf /var/lib/apt/lists/*

# Copy package.json and package-lock.json separately for better caching
COPY package*.json ./

# Install dependencies and clear cache
RUN npm install && npm cache clean --force

# Base image for stage 2 to reduce size of image
FROM node:22-slim

# Set working directory
WORKDIR /frontend

# Copy dependencies and source code
COPY --from=base /frontend/node_modules ./node_modules

# Expose application port
EXPOSE 5173

# Copy the rest of the application files
COPY . .

# Command to start the application
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0", "--port", "5173"]

As an aside (in preparation of the next phase of this project) you can push your image to your Docker Hub repo, we will use the image from this Docker Hub repo when we want to automate this project.

Configure Application Docker Compose file

Docker Compose is a tool for defining and running multi-container Docker applications. It simplifies managing complex environments by using a single YAML file to configure services, networks, and volumes. It's necessary for this project to streamline the setup of interconnected services like the frontend, backend, database, database admin UI and observability, ensuring they run consistently with minimal effort.

Navigate to the root directory of your project and create a new file compose.yaml, here we will define the services we want to run, create a docker network that these services will use to communicate with each other and configure volumes for the services that need them.

The first compose file we will create is our application compose file. In this compose file we will define the backend, frontend, database, reverse proxy (Traefik) and Adminer services.

First Draft of Compose File to test locally

The below is the first draft of my docker compose file to enable me test my containers locally. Here you will see that the ports are mapped and it is missing Traefik, the reverse proxy service. This is because I wanted to test my images first before I go too far in the project.

You can choose to do it like I did or skip ahead to the next configuration.

Once Traefik is configured, there will no need to map the ports, like we did in this version of the docker compose file below, because Traefik will be the point of entry of all traffic to our application.

If you want to test the images before moving ahead, add the below to your compose.yaml file:

services:
  frontend:
    env_file:
      - ./frontend/.env
    build: ./frontend
    ports:
      - "80:5173"
    networks:
      - devopsdojo
  backend:
    env_file:
      - ./backend/.env
    environment:
      - PYTHONPATH=/backend
    build: ./backend
    ports:
      - "8000:8000"
    networks:
      - devopsdojo
    depends_on:
      - db
    command: >
      sh -c "
      until pg_isready -h db -U app; do
        echo 'Waiting for database...';
        sleep 2;
      done;
      poetry run bash ./prestart.sh && poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000"
  db:
    image: postgres:13-alpine
    restart: always
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=changethis123
      - POSTGRES_DB=app
    # ports:
    #   - "5432:5432"
    networks:
      - devopsdojo
    volumes: 
      - db:/var/lib/postgresql/data
  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
    networks:
      - devopsdojo

networks:
  devopsdojo:
    driver: bridge
    name: devopsdojo

volumes:
  db:
    driver: local

⚠️ NOTE

In your backend/.env file update your POSTGRES_SERVER variable from app to db and if you made any other changes either to the port or the password, in your compose file, update it accordingly in the backend/.env file so that both values match.

Explanation

The above setup simplifies development by isolating services while allowing them to communicate seamlessly within the same Docker network. Here’s what each service does:

frontend:
- Builds the React frontend from the ./frontend directory.
- Uses environment variables from ./frontend/.env.
- Exposes port 5173 internally, mapped to port 80 on the host.
- Connects to the shared devopsdojo network.

backend:
- Builds the FastAPI backend from the ./backend directory.
- Uses environment variables from ./backend/.env and sets an additional PYTHONPATH for module resolution.
- Exposes port 8000 for the application.
- Depends on the db service and waits for the database to be ready before starting.
- Runs a prestart script to populate the database and then serves the API with Uvicorn.

db:
- Runs a lightweight PostgreSQL 13-alpine database.
- Configured with user, password, and database names.
- Stores persistent data in a Docker volume (db).
- Connected to the shared devopsdojo network.

Spin up Application

To build and run the containers use the below command:

docker compose up

⚠️ TIP

You can use the -d flag to run the containers in a detached mode but I like to see the logs which is why I opted not to use the -d flag.

When your containers are run you can open up localhost on your browser to see the frontend (login with the super user username and password found in the backend .env file) and localhost:8000 and localhost:8000/docs to see the backend. To view Adminer navigate to localhost:8080 and login with the correct credentials. You should see the below on your browser if you have correctly configured your application.

Application Frontend

Application's Swagger UI

Adminer Dashboard

To bring down your application run the docker compose down command, you can use the -v flag to also remove the volume(s).

Cloud Deployment

At this stage in our project we will be deploying our application to the cloud from where we will continue and finish our configuration and serve our application.

We will use an EC2 instance for our deployment to keep everything simple and self managed, another reason to use EC2 is to easily run docker-compose without adapting configurations to ECS-specific constructs.

We will go ahead and create an EC2 instance through the management console; if you don't know how to do it follow the instructions below:

Steps to Create an EC2 Instance

Log in to the AWS Management Console.
Navigate to the EC2 Dashboard under the "Compute" section.
Click Launch Instance and provide a name for your instance.
Select an AMI (Amazon Machine Image), such as Amazon Linux 2 or Ubuntu. I'd be using the Amazon Linux.
Choose an instance type (e.g., t2.micro for free tier) however we will use a t2.medium as cAdvisor requires more space than is available on the t2.micro.
Configure a key pair for SSH access (or create a new one).
Edit network settings to allow required inbound traffic, check the boxes for SSH, HTTPS and HTTP. We will be using the default VPCs and subnets, if you don't want to you can create a new VPC.
Add storage, leave the default settings as it.
Click Launch Instance and wait for it to initialize. Your key pair should be downloaded, note the download location
After our instance has been launched we will SSH into it using the command below:

ssh -i "<path/to/your/keypair.pem>" ec2-user@<your public dns>

Copy project File into Instance

Once connected to the EC2 instance we need to copy our project directory into the instance as we will be working from the instance going forward.

We will use the scp utility to copy our folder from our local machine to our instance using the below command.

Open a new terminal (or close the connection to the instance) and enter the command:

 scp -i "<path/to/your/keypair.pem>" -r <path/to/your/project/directory> ec2-user@<your public dns>:/home/ec2-user/

⚠️ REMINDER

Ensure you have docker and docker compose installed in your instance to continue with the project.

If you need help installing Docker on an Amazon Linux EC2 instance check out this post

Configure Reverse Proxy - Traefik

Traefik is a modern reverse proxy and load balancer designed for containerized environments. It automatically discovers services in your Docker setup and routes external traffic to the appropriate service based on configuration rules. As a reverse proxy, it sits between clients and backend services, managing requests, enhancing security, and improving performance.

In this project, Traefik is the main gateway to our entire application, it handles routing for the frontend, backend, Adminer and the observability, simplifying domain management, enabling HTTPS with ease, and streamlining application deployment in the cloud. It's basically the traffic controller that decides how requests get routed to different parts of our website.

Let's configure Traefik now.

Traefik Configuration

The Traefik configuration file defines how Traefik operates as a reverse proxy and load balancer. It includes settings for:

EntryPoints: Specify the ports (e.g., 80 for HTTP, 443 for HTTPS) where Traefik listens for incoming requests.
CertificatesResolvers: Enable automatic SSL/TLS certificate generation and renewal using Let's Encrypt.
Providers: Define where Traefik gets routing information (e.g., Docker labels, Kubernetes, or static configuration files).
Routing Rules: Configure how incoming requests are routed to specific services based on domains, subdomains, or paths.
Metrics and Observability: Optionally expose performance metrics (e.g., Prometheus) to monitor Traefik’s behavior.

To begin, create a new file traefik.yml in the root of your project and add the below to the file:

entryPoints:
  # Entry point for HTTP traffic on port 80
  web:
    address: ":80"
  # Entry point for HTTPS traffic on port 443
  websecure:
    address: ":443"

# Configuration for obtaining SSL certificates via Let's Encrypt
certificatesResolvers:
  myresolver:
    acme:
      # Email address for Let's Encrypt notifications and account management
      email: <your email address>
      # Storage file for SSL certificates
      storage: acme.json
      # Use HTTP challenge for domain verification
      httpChallenge:
        entryPoint: web

http:
  # Configure routers for handling requests
  routers:
    # Global HTTP to HTTPS redirection
    redirect-to-https:
      # Match all HTTP requests with any hostname
      rule: "HostRegexp(`{host:.+}`)" 
      entryPoints:
        - web 
      # Middleware to handle the redirection
      middlewares:
        - redirect-to-https 
      # No actual service, just a placeholder for redirection
      service: noop@internal

    # Router to redirect www to non-www
    redirect-www:
      rule: "Host(`<your domain>`)"
      entryPoints:
        - web
        - websecure
      # Middleware to handle the redirection
      middlewares:
        - redirect-www
      # No actual service, just a placeholder for redirection
      service: noop@internal

  middlewares:
    # Middleware to redirect HTTP to HTTPS
    redirect-to-https:
      redirectScheme:
        scheme: https
        # Sends a permanent redirect (HTTP 301)
        permanent: true

    # Middleware to redirect 'www' subdomain to the root domain
    redirect-www:
      redirectRegex:
        # Matches URLs starting with 'www.'
        regex: "^https?://www\\.(.*)"
        # Replaces 'www.' with the root domain
        replacement: "https://$1"
        # Sends a permanent redirect (HTTP 301)
        permanent: true

# Enable Prometheus metrics for monitoring
metrics:
  prometheus:
    # Add labels for entry point and service
    addEntryPointsLabels: true
    addServicesLabels: true

# Use Docker as the provider for Traefik configurations
providers:
  docker:
    # Only explicitly exposed containers will be served by Traefik
    exposedByDefault: false

Explanation

This traefik configuration serves as the smart traffic manager for our website. It does a few key things:

Sets up two main entry points: Port 80 (regular web traffic) and Port 443 (secure, encrypted traffic).
Automatically gets free SSL certificates from Let's Encrypt, keeping our site secure without manual certificate hunting. It uses your email specified in the configuration for certificate management.
Automatically sends all HTTP traffic to HTTPS (no unsecured connections), redirects www.yourdomain.com to yourdomain.com and ensures visitors always reach the right version of your site.
Adds tracking labels for Prometheus (helps you watch how your site is performing).
Only serves containers you explicitly tell it to expose.

Docker Compose Configuration for Traefik

We need to update our compose file with the reverse-proxy service and add the necessary Traefik labels to our other services. Your compose.yaml should now look like this:

services:
  frontend:
    env_file:
      - ./frontend/.env
    build: ./frontend
    container_name: frontend
    # ports:
    #   - "80:5173"
    networks:
      - devopsdojo
    labels:
      # Enable Traefik for this service and specify the secure entrypoint (HTTPS)
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`<yourdomain>`)" # substitute with your domain name
      - "traefik.http.routers.frontend.entrypoints=websecure"

      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.frontend.tls=true"
      - "traefik.http.routers.frontend.tls.certresolver=myresolver"

  backend:
    env_file:
      - ./backend/.env
    environment:
      - PYTHONPATH=/backend
    build: ./backend
    container_name: backend
    # ports:
    #   - "8000:8000"
    networks:
      - devopsdojo
    depends_on:
      - db
    command: >
      sh -c "
      until pg_isready -h db -U app; do
        echo 'Waiting for database...';
        sleep 2;
      done;
      poetry run bash ./prestart.sh && poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000"
    labels:
      # Enable Traefik for this service and specify the secure entrypoint (HTTPS)
      - "traefik.enable=true"
      # # Middleware for CORS
      - "traefik.http.middlewares.backend-cors.headers.accessControlAllowOriginList=https://<yourdomain>" # substitute with your domain name
      # Route /api to the backend root
      - "traefik.http.routers.backend-api.rule=Host(`<yourdomain>`) && PathPrefix(`/api`)" # substitute with your domain name
      - "traefik.http.middlewares.api-strip-prefix.stripPrefix.prefixes=/api"
      - "traefik.http.routers.backend-api.middlewares=api-strip-prefix,backend-cors"
      - "traefik.http.routers.backend-api.entrypoints=websecure"

      # Route /docs to /docs (Swagger UI)
      - "traefik.http.routers.backend-docs.rule=Host(`<yourdomain>`) && PathPrefix(`/docs`)" # substitute with your domain name
      - "traefik.http.routers.backend-docs.middlewares=backend-cors"
      - "traefik.http.routers.backend-docs.entrypoints=websecure"

      # Route /api/v1/openapi.json to the OpenAPI spec
      - "traefik.http.routers.backend-openapi.rule=Host(`<yourdomain>`) && Path(`/api/v1/openapi.json`)" # substitute with your domain name
      - "traefik.http.routers.backend-openapi.middlewares=backend-cors"
      - "traefik.http.routers.backend-openapi.entrypoints=websecure"

      # Enable TLS for these routers & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.backend-api.tls=true"
      - "traefik.http.routers.backend-api.tls.certresolver=myresolver"
      - "traefik.http.routers.backend-docs.tls=true"
      - "traefik.http.routers.backend-docs.tls.certresolver=myresolver"
      - "traefik.http.routers.backend-openapi.tls=true"
      - "traefik.http.routers.backend-openapi.tls.certresolver=myresolver"

  db:
    image: postgres:13-alpine
    container_name: db
    restart: always
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=changethis123
      - POSTGRES_DB=app
    # ports:
    #   - "5432:5432"
    networks:
      - devopsdojo
    volumes: 
      - db:/var/lib/postgresql/data

  adminer:
    image: adminer
    container_name: adminer
    restart: always
    # ports:
    #   - 8080:8080
    networks:
      - devopsdojo
    labels:
      # Enable Traefik for this service and specify the secure entrypoint (HTTPS)
      - "traefik.enable=true"
      - "traefik.http.routers.adminer.rule=Host(`db.<yourdomain>`)" # substitute with your domain name
      - "traefik.http.routers.adminer.entrypoints=websecure"
      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.adminer.tls=true"
      - "traefik.http.routers.adminer.tls.certresolver=myresolver"

  reverse-proxy:
    image: traefik:v3.2
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    command:
      - --api
    networks:
      - devopsdojo
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/etc/traefik/traefik.yml
      - ./acme.json:/acme.json
    labels:
      # Enable Traefik for this service
      - "traefik.enable=true"

      # Dashboard route
      - "traefik.http.routers.dashboard-api.rule=Host(`<yourdomain>`) && PathPrefix(`/dashboard`) || (PathPrefix(`/debug`) || PathPrefix(`/api/http`) || PathPrefix(`/api/tcp`) || PathPrefix(`/api/udp`) || PathPrefix(`/api/entrypoints`) || PathPrefix(`/api/overview`) || PathPrefix(`/api/rawdata`) || PathPrefix(`/api/version`))" # substitute with your domain name
      - "traefik.http.routers.dashboard-api.service=api@internal"
      - "traefik.http.middlewares.dashboard-auth.basicauth.users=admin:<hashed password>" # Substitute with your username:hashed-password you generated with htpasswd
      - "traefik.http.routers.dashboard-api.middlewares=dashboard-auth,redirect-dashboard"
      - "traefik.http.routers.dashboard-api.entrypoints=websecure"

      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.dashboard-api.tls=true"
      - "traefik.http.routers.dashboard-api.tls.certresolver=myresolver"

      # Redirect /dashboard to /dashboard/
      - "traefik.http.middlewares.redirect-dashboard.redirectregex.regex=^https?://(.*)/dashboard$$"
      - "traefik.http.middlewares.redirect-dashboard.redirectregex.replacement=https://$1/dashboard/"
      - "traefik.http.middlewares.redirect-dashboard.redirectregex.permanent=true"

networks:
  devopsdojo:
    driver: bridge
    name: devopsdojo

volumes:
  db:
    driver: local

Updates Made

We added the necessary labels to all our services to allow traefik route traffic to those services, the only service that is exempted is the db service because we will only access the database via Adminer, our database administration service.
We removed the port mappings which we had added to our services to enable us test our application locally. This is necessary because all the traffic to our application will be routed through Traefik.
We added the reverse-proxy service; the configuration:
- Opens ports 80 (HTTP), 443 (HTTPS), and 8080 (Traefik dashboard).
- Allows incoming web traffic and Traefik management.
- Enables the Traefik dashboard at /dashboard. Our dashboard uses authentication so that unauthorized persons do not have access to our dashboard. (I will show you how to hash your password soon)
- Ensures the dashboard is only accessible via HTTPS.
- The last update we made was to redirect traffic from /dashboard to /dashboard/ as traefik expects the trailing backslash to ensure proper routing and resource loading for certain applications, like the dashboard. Without the trailing slash, API calls or static resource paths may fail, leading to incomplete or broken functionality. This redirect guarantees consistency and resolves such issues.

Generate Password Hash

To generate the hashed password, you can use an htpasswd generator online or use the htpasswd command but before using the command ensure you install it using the below command:

# Install apache2-utils if not already installed
sudo apt-get install apache2-utils

# Or if using an Amazon Linux server like me use the below command 

sudo yum install httpd-tools

⚠️ NOTE

When used in a docker compose file, all dollar signs in the hash need to be doubled for escaping. This is the reason for the sed command used in the command below.

Generate that password hash using the command below:

echo $(htpasswd -nbB admin <yourpassword>) | sed -e s/\\$/\\$\\$/g

The output will give you the username:hashed-password format to use in the configuration. Copy the output and substitute it into the dashboard-auth middleware label in the reverse-proxy service in your compose file.

Prepare the `acme.json` File

If you noticed, the reverse-proxy service has an acme.json volume where the SSL certificate will be saved. This file needs to already exist in our project directory (same directory where your compose file is) and be writeable so that traefik can save our SSL certificate there so we need to create that file.

Use the command below:

touch acme.json
chmod 600 ./acme.json

We will not commit this file to version control but we will encrypt the file and commit the encrypted version so that whenever we are working on the project we will decrypt it and use it.

Doing this allows seamless reuse of the certificates across projects, preventing unnecessary certificate renewals and avoiding Let's Encrypt rate limits.

Add the acme.json file to .gitignore.

Once we are done for the day we will encrypt the acme.json file with sops and push that encrypted version to version control.

Encrypt and Decrypt Acme File

⚠️ ⚠️ NOTE: Skip this step, only implement it when you already have your SSL certificate and want to commit your project to version control.

Install sops:

sops is simple, integrates with version control, and supports multiple encryption backends (e.g., AWS KMS, GCP KMS, Azure Key Vault, or a GPG key).

# For Ubuntu/Debian
SOPS_LATEST_VERSION=$(curl -s "https://api.github.com/repos/getsops/sops/releases/latest" | grep -Po '"tag_name": "v\K[0-9.]+')
curl -Lo sops.deb "https://github.com/getsops/sops/releases/download/v${SOPS_LATEST_VERSION}/sops_${SOPS_LATEST_VERSION}_amd64.deb"
sudo apt --fix-broken install ./sops.deb
rm -rf sops.deb

# For macOS (Homebrew)
brew install sops

Encrypt with Age Key:

First will install age before we are able to generate a key with age:

# On macOS
brew install age

# On Ubuntu/Debian
sudo apt install age

# On Windows with Chocolatey
choco install age.portable

Then we will generate an Age key pair which we will use for our encryption:

# Create a directory for the key
mkdir -p ~/.config/sops/age

# Generate the key
age-keygen -o ~/.config/sops/age/keys.txt

Important: Keep your Keypair file secure, we will have to save it as an env var to use for CI/CD. Do not commit the private key to version control.

Now we can encrypt the file using the public key from our generate keypair:

# Extract public key from keys.txt
export AGE_PUBLIC_KEY="age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# Encrypt your file
sops --encrypt \
  --age $AGE_PUBLIC_KEY \
  --output acme.json.enc \
  acme.json

To Decrypt Files Locally:

# SOPS will automatically look for the key in ~/.config/sops/age/keys.txt
sops --decrypt acme.json.enc > acme.json

Create DNS (A) Record for Domains

DNS records are like the postal address for your website, telling the internet exactly where to find your digital home. When you create these records, you're essentially setting up a precise navigation system that directs internet traffic to your specific server.

These records are crucial because they enable services like Let's Encrypt to verify your domain ownership, allow automatic SSL certificate generation, and ensure that when someone types your domain name, they're routed to the correct IP address.

Think of it as creating a map that guides visitors directly to your online doorstep, making sure they arrive safely, securely, and exactly where you want them to be.

Depending on your domain hosting service, the process to create these records may differ; however you need to create an A record for your domain, db.<yourdomain> and www.<your domain>.

⚠️ NOTE

Ensure you create these records, if they are not created your application will not be served on your domain.

Use the public IP address of your EC2 instance created earlier for the A record.

Update your Environments

In both your backend and frontend directories you need to update the .env files to include your domain to avoid CORS issue and so that your application is accessible from your domain.

`backend/.env`

Update the DOMAIN and BACKEND_CORS_ORIGINS variables in your backend/.env to include your domain, it should look like this now:

DOMAIN=<yourdomain> # This has no leading http or https eg example.com

BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,https://localhost,https://localhost:5173,http://<yourdomain>,https://<yourdomain>"

`frontend/.env`

Update VITE_API_URL with your domain name, it should now look like this:

VITE_API_URL=https://<yourdomain>/api

Now we are set to rebuild our image and test our traefik configuration.

Build Application

After you have made all these adjustments, you are set to build the containers and run your application, do this by simply running the Docker compose up command as seen below, we will use the -d flag to run it in a detached mode, if you would like to see the logs you can omit the -d flag.

docker compose up -d

You should now be able to access your application from your domain as seen in the images below.

Application Frontend

Application Backend Root

Application's Swagger UI

Adminer Dashboard

Traefik Dashboard Asking for Authentication

If you do not supply an credentials or you enter a wrong username or password you won't be granted access as seen in the image below.

Unauthorized Response From Traefik Dashboard

Traefik Dashboard

If you successfully authenticate with the correct credentials you should see a dashboard that resembles the one below:

Monitoring and Observability

We're almost at the end of this project, we need to configure our monitoring and observability stack. Effective monitoring and observability are critical for maintaining the health and performance of modern applications. In this project, we implement a robust monitoring stack using Prometheus, Grafana, Loki, Promtail, and cAdvisor to ensure real-time visibility into system metrics, logs, and container performance.

Prometheus: Collects and stores metrics, enabling detailed insights into application and infrastructure performance.
Grafana: Visualizes metrics and logs through customizable dashboards.
Loki: Provides log aggregation and querying capabilities.
Promtail: Streams logs from application containers to Loki.
cAdvisor: Monitors resource usage and performance of running containers.

Together, these tools create an integrated solution for proactive monitoring, streamlined troubleshooting, and maintaining operational excellence in containerized environments.

Configure Prometheus

Due to its robust querying language, effective storage, and simplicity in integrating with several metrics sources, Prometheus is a popular open-source monitoring and alerting solution.

✨

Docker Compose Configuration for Prometheus

We will separate our application stack from our monitoring stack and so we need to create a new compose file, in your project root create a new file compose.monitoring.yaml.

touch compose.monitoring.yaml

Add the below in the new file:

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    command: 
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=/prometheus'
    networks:
      - devopsdojo
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom_data:/prometheus
    labels:
      # Enable Traefik for this service, configure router and specify entrypoint
      - "traefik.enable=true"
      - "traefik.http.routers.prometheus.rule=Host(`<yourdomain>`) && PathPrefix(`/prometheus`)"
      - "traefik.http.routers.prometheus.entrypoints=websecure"

      # Tell Traefik to use the port 9090 to connect to prometheus
      - "traefik.http.services.prometheus.loadbalancer.server.url=http://prometheus:9090/"

      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.prometheus.tls=true"
      - "traefik.http.routers.prometheus.tls.certresolver=myresolver"

volumes:
  prom_data:

networks:
  devopsdojo:
    driver: bridge
    name: devopsdojo

✨ This Docker Compose setup deploys Prometheus in a Docker container, along with a specified configuration file for custom monitoring settings.

✨ This Prometheus service is configured to run in a Docker container with the prom/prometheus image. It uses a custom configuration file prometheus.yml, which we will create and populate next, and persists data in a named volume prom_data. It connects to the devopsdojo network and integrates with Traefik for secure access.

✨ Traefik routes traffic to Prometheus via websecure and TLS, using the domain you specify with the path /prometheus. The service listens on port 9090, but since we will access the prometheus dashboard via a subpath routed through Traefik we need to explicitly make the prometheus service aware of this subpath by explicitly appending the subpath to the target server URL in it's loadBalancer configuration as you can see that we did above.

Next we create the prometheus configuration file that instructs prometheus on what to do.

Prometheus Configuration File

The Prometheus configuration file is a YAML-based document that outlines how Prometheus should scrape, collect, and process metrics from various targets.

It defines parameters such as scrape intervals, targets to scrape, and rules for alerting, providing the blueprint for effective monitoring setups. The configuration file is the core of the Prometheus setup and is crucial for accurate and efficient monitoring. It's human-readable and easy to edit.

In your project root create a new file prometheus.yml and add the below blocks of code to the file:

# Global defaults, applies to all scrape jobs unless explicitly overridden
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

# Define the specify endpoints prometheus should scrape data from
scrape_configs:
  # Config to scrape data from the prometheus service itself
  - job_name: 'prometheus'
    honor_timestamps: true
    metrics_path: prometheus/metrics
    scheme: http
    static_configs:
      - targets: ['prometheus:9090']

  # Config to scrape data from the traefik service
  - job_name: 'traefik'
    metrics_path: /metrics
    static_configs:
      - targets: ['traefik:8080']

This Prometheus configuration does the following:

Global Settings: These settings apply to all scrape jobs unless overridden. They define default behavior for how Prometheus scrapes metrics:
- scrape_interval: Frequency of scraping. Sets the default time between each scrape (15 seconds).
- scrape_timeout: Maximum time Prometheus waits for a scrape to complete. Specifies how long Prometheus should wait for a scrape to complete (10 seconds).
- evaluation_interval: Defines how often Prometheus evaluates alerting and recording rules (15 seconds).

✨

Scrape Configurations: Defines how Prometheus scrapes data from specific services:
- Prometheus Job: Scrapes Prometheus's own metrics at prometheus:9090/prometheus/metrics.
- honor_timestamps: Ensures that scraped data honors the timestamps from the source.
- Traefik Job: Scrapes metrics from the Traefik service at traefik:8080/metrics.

Both jobs define their targets and metrics paths for scraping, ensuring that Prometheus collects data from the specified services.

We will still come back to this configuration when we setup cAdvisor so we can add a job to scrape it's data.

Configure cAdvisor

Google created cAdvisor (Container Advisor), a tool that allows for real-time tracking of performance metrics and resource utilization for containers in use. It gathers, compiles, analyses, and exports data about containers that are currently running so that Prometheus can use it for monitoring.

We will create the cadvisor service in docker compose now by adding the below block of code into the compose.monitoring.yaml file:

compose.monitoring.yaml

  cadvisor:
    image: gcr.io/cadvisor/cadvisor
    container_name: cadvisor
    privileged: true
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - devopsdojo

Now we need to tell Prometheus to scrape data from cAdvisor, to do that add the below block of code to the prometheus.yml file

prometheus.yml

  # Config to scrape data from the cAdvisor service
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Configure Loki

Loki is a log aggregation system designed by Grafana Labs for efficiently collecting, storing, and querying logs. Unlike traditional logging systems, Loki is optimized for cost-efficiency and simplicity by indexing only metadata, not the content of logs. It's often paired with Promtail, which collects logs from various sources (e.g., Docker containers, system logs) and pushes them to Loki. Together with Grafana, Loki provides a powerful, scalable solution for centralized log management and visualization.

Docker Compose Configuration for Loki

In your compose.monitoring.yaml file add the following after the cadvisor service:

`compose.monitoring.yaml`

  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    command: 
      - '--config.file=/etc/loki/loki-config.yaml'
    networks:
      - devopsdojo
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
      - loki-data:/loki
    labels:
      # Enable Traefik for this service, configure router and specify entrypoint
      - "traefik.enable=true"
      - "traefik.http.routers.loki.rule=Host(`<yourdomain>`) && PathPrefix(`/loki`)"
      - "traefik.http.routers.loki.entrypoints=websecure"

      # Tell Traefik to use the port 3100 to connect to loki
      - "traefik.http.services.loki.loadbalancer.server.url=http://loki:3100/"

      # Add middleware to strip the prefix
      - "traefik.http.middlewares.loki-strip.stripprefix.prefixes=/loki"
      - "traefik.http.routers.loki.middlewares=loki-strip"

      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.loki.tls=true"
      - "traefik.http.routers.loki.tls.certresolver=myresolver"

Add the Loki volume to the Volume top level element, your volume should look like this now:

volumes:
  prom_data:
  loki-data:

Loki Configuration File

If you noticed, in docker compose configuration above we specified a config file but we do not have any config file yet and so the next step is to create that file.

We need to download the necessary configuration files for Loki, to do this run the below command:

wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

We will leave the config file as is as the configuration is sufficient for what we need to do.

Configure Promtail

Promtail is an agent designed to collect and forward logs to Loki. It works seamlessly with various log sources, including system logs, application logs, and Docker container logs.

Promtail reads log files, adds metadata such as labels (e.g., container name, job, or hostname), and sends the enriched logs to Loki for aggregation and querying. It integrates natively with Kubernetes, leveraging pod labels and annotations to simplify log collection in containerized environments.

Promtail is lightweight, easy to configure, and essential for building a robust logging pipeline with Loki.

Docker Compose Configuration for Promtail

Add the below to your compose.monitoring.yaml file

`compose.monitoring.yaml`

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    command: 
      - '--config.file=/etc/promtail/config.yml'
    networks:
      - devopsdojo
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/log:/var/log:ro

Promtail Configuration File

Like we did with the Loki configuration, we need to download the configuration file for Promtail, do that by running the below command:

wget https://raw.githubusercontent.com/grafana/loki/v3.0.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

We will update the promtail configuration file we just downloaded, we need to update our client url and add another job to the configuration.

Update your client: - url to look like the below:

clients:
  - url: https://<your domain>/loki/loki/api/v1/push

After the system job at the end of the file, add the docker job:

  - job_name: docker
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*.log

Configure Grafana

Next we will configure our Grafana service so that we can visualize our data via Grafana's dashboard. Grafana enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they’re stored. Grafana data source plugins enable you to query data sources including time series databases like Prometheus and CloudWatch, logging tools like Loki and Elasticsearch and a lot more.

Grafana OSS provides you with tools to display that data on live dashboards with insightful graphs and visualizations.

Docker Compose Configuration for Grafana

For our Docker compose configuration of our Grafana service, add the below to your compose.monitoring.yaml file

grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    environment:
      - "GF_SERVER_DOMAIN=<yourdomain>"
      - "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana"
      - "GF_SERVER_SERVE_FROM_SUB_PATH=true"
    networks:
      - devopsdojo
    volumes:
      - grafana-storage:/var/lib/grafana
    labels:
      # Enable Traefik for this service, configure router and specify entrypoint
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`<yourdomain>`) && PathPrefix(`/grafana`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"

      # Tell Traefik to use the port 3000 to connect to grafana
      - "traefik.http.services.grafana.loadbalancer.server.url=http://grafana:3000"

      # Enable TLS for this router & use the 'myresolver' certificates resolver for obtaining SSL certificates
      - "traefik.http.routers.grafana.tls=true"
      - "traefik.http.routers.grafana.tls.certresolver=myresolver"

The volume part of the compose.monitoring.yaml file will now look like this:

volumes:
  prom_data:
  grafana-storage:

Spin up Services

Now that we have written all our configurations we can go ahead and start up all our monitoring services.

Since we want to run both our compose files at the same time we will add an include element to our first (application) compose file.

Open the compose.yaml file and add the below block of code to the very top of the file.

`compose.yaml`

include:
  - compose.monitoring.yaml

Now if you have any of the earlier containers running already, you should stop them and start everything all over again. Use the command below:

docker compose down && docker compose up -d

All our containers should be up and running and available now so we can go ahead and create our grafana dashboards now.

If you head to your prometheus web UI (https:///prometheus) and click on target health under the status dropdown you should see the different scrape targets we configured in our prometheus configuration.

Prometheus Web UI

Create Dashboards

To begin creating our Grafana dashboards we need to login to our Grafana web UI. Navigate to https:///grafana, the login page will popup and you need to enter the default username and password which are both admin.

Grafana Web UI Login

Once you are logged in you will be asked to change your password, do that and let's continue.

Add Data sources

Upon logging in you will see a welcome page that looks like the one below, click on the Data Sources box (illustrated by the arrow) to add your first data source.

Grafana welcome page

Prometheus Data source

✨ Click on Prometheus from the options on the page that opens up, it should be right on top.

Prometheus data source

✨ On the next page, scroll down and enter https://<yourdomain>/prometheus as your prometheus server url under the Connection category.

Prometheus server url

✨ Scroll to the bottom of the page and click on Save and test, you should get a confirmation that the prometheus API has been successfully queried, as you can see below

Successfully queried Prometheus API

Loki Data Source

✨ To add Loki as a data source, click on the hamburger button on the top left part of the page and under connections, click on data sources as shown below

Data sources menu

✨ On the next page, click on the add new data source button on the top right corner of the page.

✨ Scroll down a bit and select Loki as the data source. Just as we did with prometheus, in the next page enter your loki url in the connection category, finally scroll down and save and test.

Loki data source addition

Import cAdvisor Dashboard

Now that we have added our Data sources we can proceed with creating our dashboard. We won't be creating these dashboards from scratch though, we will simply import dashboards that have been created by community members that fit what we are trying to do.

You can find these dashboards at https://grafana.com/grafana/dashboards/ but I've done the heavy lifting for you and found the dashboard we will use, however feel free to explore the link and see if any other dashboards fit your need better.

✨ Navigate to Grafana web Ui homepage and click on the Dashboards box

Create first Dashboard

✨ On the resultant new page click on import dashboard and in the next page enter the code 19792 and click on load

Import dashboard - cAdvisor

✨ On the new page, scroll to the bottom, in the Prometheus category, click and select your prometheus data source from the drop down and click on import

Add cAdvisor Dashboard

✨ You should be able to see your beautiful dashboard now, it will look like what I have below

cAdvisor Dashboard

Import Loki Dashboard

✨ To add a new dashboard, click on the hamburger button on the top left and click on dashboards from the options.

✨ When the next page loads, click on the arrow on the New button for the dropdown and select import from the options.

✨ From here you know the drill, enter the code 13186, load the dashboard, add your loki data source and import the dashboard.

✨ You should see the dashboard now, tinker around with it to see more information.

✨ You could also explore your Loki data source and run custom queries as you see fit.

Import Traefik Dashboard

The process to import the traefik dashboard is the same as we went through for the Loki dashboard.

✨ Enter the code 4475 and load the dashboard, add your prometheus data source and import the dashboard.

✨ Now you should have a dashboard that looks like the one below.

Conclusion

Congratulations! After all the effort and time invested, we’ve successfully Dockerized our full-stack application with a FastAPI backend and a React frontend, set up a reverse proxy, and deployed a robust monitoring and observability stack using Prometheus, Grafana, Loki, Promtail, and cAdvisor. Along the way, we built fully functional dashboards to keep an eye on system performance and logs.

This journey most definitely demanded patience and determination, but it also provided the opportunity to sharpen essential skills in containerization, deployment, and observability. From orchestrating containers to visualizing metrics and logs, these are valuable tools in any developer's toolkit.

Take a moment to appreciate how far you’ve come, whether it’s mastering Docker, setting up monitoring systems, or troubleshooting with confidence, this project is a testament to your growth and perseverance. Here's to many more successful deployments ahead!

If you found this post helpful in anyway, please let me know in the comments and share with your friends. If you have a better way of implementing any of the steps I took here, please do let me know, I love to learn new (~better~) ways of doing things.

Follow for more DevOps content broken down into simple understandable structure.

ChigozieCOFollow

Cloud and DevOps Engineer who's a little bit obsessed with the whole cloud computing scene, always ready to dive into the nitty-gritty of cloud infrastructure, DevOps and Security.

If this post has helped you, consider supporting me:

Terraform Remote Backend: How to Manage Terraform State File for Easier Collaboration across Teams

ChigozieCO — Wed, 06 Nov 2024 11:05:41 +0000

Have you ever wondered how Terraform manages to keep track of the resources you create with terraform and apply your changes to just those specific resources whenever you try to create and update resources?

If you have used terraform you would have noticed that every time you ran terraform plan, terraform apply or terraform destroy Terraform was able to find the resources it created previously and update them accordingly. Howwww? How did Terraform know which resources it was supposed to manage? Your AWS account could have several different infrastructures deployed through a variety of ways (eg some manually, some via Terraform, some via the CLI, some via AWS SAM etc), so how does Terraform know the particular infrastructure it created and hence is responsible for?

If you find this post useful, consider supporting me:

In two words “Terraform State”

📌 What is Terraform State

Terraform maintains track of everything it creates in your cloud environments so that it will know what it created and can go back and make modifications for you if you need to update or remove something later. This is what is meant when they say that Terraform is a stateful application.

Terraform records information about what infrastructure it created in a Terraform state file. By default, when you run Terraform init in the folder "/hello/world", it will create the file /hello/world/terraform.tfstate. This file includes a customized JSON format that records a mapping between the Terraform resources in your configuration files and their actual representations.

Every time you run Terraform, it can fetch the latest status of your resources from your cloud account and compare that to what’s in your Terraform configurations to determine what changes need to be applied. In other words, the output of the plan command is a diff between the code on your computer and the infrastructure deployed in the real world, as discovered via IDs in the state file.

⚠️⚠️ Note

The state file format is a private API that is meant only for internal use within Terraform. You should never edit the Terraform state files by hand or write code that reads them directly. If for some reason you need to manipulate the state file—which should be a relatively rare occurrence—use the terraform import or terraform state commands.

📌 Benefits of the Terraform State File

✨ Idempotence
Whenever a Terraform configuration is applied, Terraform checks if there is an actual change made. Only the resources that are changed will be updated.
✨ Deducing dependencies
Terraform maintains a list of dependencies in the state file so that it can properly deal with dependencies that no longer exist in the current configuration.
✨ Performance
Terraform can be told to skip the refresh even when a configuration change is made. Only a particular resource can be refreshed without triggering a full refresh of the state, hence improving performance.
✨ Collaboration
State keeps track of the version of an applied configuration, and it's stored in a remote, shared location. So collaboration is easily done without overwriting.
✨ Auditing
Invalid access can be identified by enabling logging.
✨ Safer storage
Storing state on the remote server helps prevent sensitive information.

📌 Limitations Teams Face with Terraform State Files

The above is all great and nice if you are working on your project alone, if you’re using Terraform for a personal project, you can get away with storing state in a single terraform.tfstate file that lives locally on your computer. This will work just fine without any form of conflicts occurring. However, when working on a large project along side your team members, using Terraform, storing the terraform.tfstate file will present the following problems:

✨ Shared storage for state files

To be able to use Terraform to update your infrastructure, each of your team members needs access to the same Terraform state files. That means you need to store those files in a shared location. If you're thinking of using version control such as Git, think again. Although you should definitely store your Terraform code in version control, storing Terraform state in version control is a bad idea as it would lead to a new set of problems like manual error, locking and secrets exposure.

Manual error refers to the ease which which one can forget to pull down the latest changes from version control before running Terraform or to push your latest changes to version control after running Terraform. It’s just a matter of time before someone on your team runs Terraform with out-of-date state files and as a result, accidentally rolls back or duplicates previous deployments. The problem of secrets exposure presents itself due to the fact that all data in Terraform state files is stored in plain text. This is a problem because certain Terraform resources need to store sensitive data.

✨ Locking state files

As soon as data is shared, you run into a new problem: locking. Without locking, if two team members are running Terraform at the same time, you can run into race conditions as multiple Terraform processes make concurrent updates to the state files, leading to conflicts, data loss, and state file corruption. Locking using version control such as Git is not possible as most version control systems do not provide any form of locking that would prevent two team members from running terraform apply on the same state file at the same time.

✨ Isolating state files

When making changes to your infrastructure, it’s a best practice to isolate different environments. For example, when making a change in a testing or staging environment, you want to be sure that there is no way you can accidentally break production. But how can you isolate your changes if all of your infrastructure is defined in the same Terraform state file?

📌 Using Terraform's Remote Backend.

The ideal method to handle shared storage for state files is to use Terraform's built-in support for remote backends rather than version control. How Terraform loads and saves state is determined by a backend. The local backend, which saves the state file on your local drive, is the default backend you use when you work on a project alone. You can keep the state file in a distant, shared store using remote backends. There are several remote backends available, including:

✨ HashiCorp's Terraform Cloud and
✨ Terraform Enterprise,
✨ Amazon S3,
✨ Azure Storage,
✨ Google Cloud Storage and others.

Remote backends solve the manual error, locking and secrets exposure errors we mentioned that could arise with the use of version control.

There is no danger of manual human mistake since, after configuring a remote backend, as each time you run plan or apply Terraform will automatically load the state file from that backend and will automatically put the state file in that backend after each apply.

In regards to the error posed as a result of locking you should note that locking is supported natively by the majority of remote backends. Terraform will automatically get a lock in order to execute terraform apply; if someone else is currently running apply, they will already have the lock, and you will need to wait. Apply can be performed with the -locktimeout= <TIME> argument to tell Terraform to wait for a lock to be released up to TIME (for example, -lock-timeout=10m will wait for 10 minutes).

Most of the remote backends natively support encryption in transit and encryption at rest of the state file. Although the best solution would be for Terraform to natively support encrypting secrets within the state file, these remote backends reduce most of the security concerns, given that at least the state file isn’t stored in plain text on disk anywhere.

We would be using Amazon S3, AWS' managed file store. This is the best bet as a remote backend when working within an AWS environment for several reasons.

⚠️ NOTE

The easiest way, and might I say most effective way, and also quickest way to do this will be to manually create your S3 bucket and DynamoDB table via the management console or via the AWS CLI and then configure your backend in your terraform script. However that method introduces a level of manual configuration that could introduce manual error.

Even though the method I am about to show you makes the whole process automated and repeatable, it's limitation is that the creation of our resources must then be in a two step process:

First we write Terraform code to create the S3 bucket and DynamoDB table and deploy that code with a local backend.

Then we go back to the Terraform code, add a remote backend configuration to it to use the newly created S3 bucket and DynamoDB table, and run terraform init to copy your local state to S3.

If you ever want to delete the S3 bucket and DynamoDB table, you’d have to do this two-step process in reverse:

First go to the Terraform code, remove the backend configuration, and rerun terraform init to copy the Terraform state back to your local disk.

Then run terraform destroy to delete the S3 bucket and DynamoDB table.

✨ I personally always prefer to create my bucket and table manually.
Also you can use the same bucket and table for several different configurations but be careful when you do this so that you don't delete these resources and lose the state files of so many of your environments.

Follow along to see how you can use terraform to create the bucket and table where you want to store your Terraform state.

📌 Creating an S3 Remote Backend

To enable remote state storage with Amazon S3, the first step is to create an S3 bucket.

✨ Create a main.tf file in a new folder.

Use any code editor of your choice, eg vscode, Specify AWS as the provider at the top of the file:

provider "aws" {
    region = "us-east-1"
}

✨ Next, create an S3 bucket by using the aws_s3_bucket resource:

resource "aws_s3_bucket" "tf-state" {
  bucket = "tf-code-state"

  #The below line of code will prevent accidental deletion of this bucket

  lifecycle {
    prevent_destroy = true
  }
}

⚠️⚠️ Note that S3 bucket names must be globally unique among all AWS customers. Therefore, you will need to change the bucket parameter from "tf-code-state" (which I already created) to your own name. Make sure to remember this name and take note of what AWS region you’re using because you’ll need both pieces of information again a little later on.

📌 Adding Security Elements to our Remote Backend

Let’s now add several extra layers of protection to this S3 bucket.

✨ First, use the aws_s3_bucket_versioning resource to enable
versioning on the S3 bucket so that every update to a file in the bucket actually creates a new version of that file. This allows you to see older versions of the file and revert to those older versions at any time, which can be a useful fallback mechanism if something goes wrong:

# Next set of lines of code will enable versioning so that we can see previous versions of our state file

resource "aws_s3_bucket_versioning" "enabled" {
  bucket = aws_s3_bucket.tf-state.id
  versioning_configuration {
    status = true
  }
}

✨ Next we will use the aws_s3_bucket_server_side_encryption_configuration
resource to turn server-side encryption on by default for all data written to this S3 bucket. This ensures that your state files, and any secrets they might contain, are always encrypted on disk when stored in S3:

# Now we enable server side encryption by default

resource "aws_s3_bucket_server_side_encryption_configuration" "dafault" {
  bucket = aws_s3_bucket.tf-state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

✨ The very next thing we have to do is to block all public access to the s3 bucket using the aws_s3_bucket_public_access_block resource. S3 buckets are private by default, but as they are often used to serve static content—e.g., images, fonts, CSS, JS, HTML—it is possible, even easy, to make the buckets public. Since your Terraform state files may contain sensitive data and secrets, it’s worth
adding this extra layer of protection to ensure no one on your team can ever accidentally make this S3 bucket public:

# Explicitly block all public access to the s3 bucket

resource "aws_s3_bucket_public_access_block" "public_access" {
  bucket = aws_s3_bucket.tf-state.id

  block_public_acls = true
  block_public_policy = true
  ignore_public_acls = true
  restrict_public_buckets = true
}

📌 Create a DynamoDB Table for Locking

✨ The next thing we have to do is to create a DynamoDB table that we would use for locking. DynamoDB is Amazon’s distributed key–value store. It supports strongly consistent
reads and conditional writes, which are all the ingredients you need for a distributed lock system. Moreover, it’s completely managed, so you don’t have any infrastructure to run yourself, and it’s inexpensive, with most Terraform usage easily fitting into the free tier.

To use DynamoDB for locking with Terraform, you must create a
DynamoDB table that has a primary key called LockID (with this exact spelling and capitalization). You can create such a table using the aws_dynamodb_table resource:

# Create dynamodb table for terraform state locking
resource "aws_dynamodb_table" "tf_state_lock" {
  name           = tf-state-lock
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

📌 Create Resources

Now we will create the table and the bucket.

We have configured our remote backend yet (remember the two step process we spoke about earlier?) so our statefile will be stored locally for now.

Run the commands below to initialize terraform, show the plan and create the resources:

terraform init
terraform plan
terraform apply

⚠️

Note that when you run terraform apply without the -auto-approve flag you will to prompted to either apply or decline the creation with a yes or no.

Type yes to continue with the process.

📌 Configure the remote backend and move the state file

Now our bucket and DynamoDB table are both created, but we still have our statefile save locally, therefore we need to move it to these resources we just created.

To do this, first we will update our configuration, we need to add a backend configuration to our Terraform code. This configuration will be in the terraform block as shown below.

At the top of your main.tf add the below block of code:

terraform {
  backend "s3" {
    # Replace this with your bucket name!
    bucket = "terraform-up-and-running-state"
    key = "global/s3/<your s3 bucket name>"
    region = "us-east-2"

    # Replace this with your DynamoDB table name!
    dynamodb_table = "<your dynamodb table name>"
    encrypt = true
  }
}

Now you simply have to rerun the terraform init command again.

This will instruct Terraform to store your state file in this S3 bucket. When you run this command Terraform will automatically detect that you already have a state file locally and prompt you to copy it to the new S3 backend.

After running the command, your Terraform state will be stored in the S3 bucket. You can confirm that your state file is in your bucket by heading to the AWS Management Console, clicking on S3 bucket and clicking into your bucket.

Cleanup

To destroy your infrastructure and cleanup your environment you need to take out the backend configuration you just added, run terraform init again and then run terraform destroy.

This is so that the state files will be copied back into your local environment and you can delete the S3 buckets. This is only necessary because your configuration is the same configuration you used in creating the S3 bucket and Dynamodb table. Supposing you didn't create the backend S3 bucket and dynamodb table with this configuration you wouldn't need to remove the backend configuration from your setup.

Hope you found this helpful, if you did please share with your community and connect with me on LinkedIn and follow me on GitHub.

Follow for more DevOps content broken down into simple understandable structure.

ChigozieCOFollow

Cloud and DevOps Engineer who's a little bit obsessed with the whole cloud computing scene, always ready to dive into the nitty-gritty of cloud infrastructure, DevOps and Security.

Complete Guide to Automate the Deployment of the Sock Shop Application on Kubernetes with IaC, CI/CD, and Monitoring

ChigozieCO — Tue, 22 Oct 2024 12:13:38 +0000

This project is about deploying a microservices-based application using automated tools to ensure quick, reliable, and secure deployment on Kubernetes. By focusing on Infrastructure as Code, you'll create a reproducible and maintainable deployment process that leverages modern DevOps practices and tools.

For a detailed breakdown of what this project is trying to achieve check out the requirements here

Prerequisites

An AWS Account
AWS CLI installed and configured
Terraform installed
Kubectl installed
Helm
A custom domain

Set Up AWS Hosted Zone and Custom Domain

The first thing to do to begin this project is to create a hosted zone and configure our custom domain.

I have already purchased a custom domain projectchigozie.me and so I created an AWS hosted zone to host this domain. I didn't use terraform to create this hosted zone because this step still required manually configuration to add the nameservers to the domain.

Steps to create an AWS hosted zone

Navigate the the aws management console
Click on services in the top left corner, click on the networking and content delivery category and choose the Route53 subcategory.
Click on create hosted zone
When it opens up, enter your custom domain name.
Leave the rest as default, you can add a tag if you want to.
Click create hosted zone

Once the hosted zone is created, we then retrieve the namespaces from the created hosted zone and use it to replace those already in our custom domain.

The specific steps to take to do this will vary depending on your domain name registrar but it's pretty much very easy across board.

Provision AWS EKS Cluster with Terraform

For automation and to speed up the process, we will write a terraform script to deploy an EKS cluster and configure the necessary VPC, subnets, security groups and IAM roles.

We won't be reinventing the wheel here as there are a lot of terraform modules out there that do just exactly what we are trying to do. I will be using the official terraform/aws vpc and eks modules.

My terraform script for the EKS cluster provisioning can be found in the terraform directory

Create a `.gitignore` File

Before we begin it is usually best to create a .gitignore where we will specify the files that will not be pushed to version control.

For the sake of keeping this post short, find my git ignore file here, copy its contents and add to your own .gitignore file.

Setup Remote Backend

We will make use of remote backend for this project, using an S3 bucket to save our statefiles abd DynamoDb to save our lock file.

Create an S3 bucket and name it anything you like, remember that your bucket name must be unique so find a unique name to use, you can enable versioning in your bucket to ensure older statefiles are not deleted incase you need to go go back to a previous configuration.

In that bucket create 2 folders, I named mine eks and k8s, for the two different terraform scripts.

Next create a DynamoDb table with any name of your choosing, ensure the partition key is named LockID written exactly like this and leave all other settings as default. We will use them for our terraform configurations.

Add Providers

terraform/main.tf

First create a new directory called terraform this is where all our Terraform scripts will be.

Create a main.tf file in the terraform directory and add the code below to the file.

# Copyright (c) HashiCorp, Inc.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }

  backend "s3" {
    bucket         = "sockshop-statefiles" # Replace with your bucket name
    key            = "eks/terraform.tfstate" # Replace with your first folder name
    region         = "us-east-1"
    dynamodb_table = "sock-shop-lockfile" # Replace with your DynamoDb table name
  }

}

provider "aws" {
  region = var.region
  shared_credentials_files = ["~/.aws/credentials"]
}

Retrieve Data from AWS

terraform/data.tf

We will retrieve the availability zones data from AWS so we can use them in creating our resources and so create a data.tf file in the terraform directory and add the below lines of code to the file:

# Filter out local zones, which are not currently supported 
# with managed node groups
data "aws_availability_zones" "available" {
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

Create VPC and Other Networking Resources

terraform/vpc.tf

Also in the terraform directory create a new file vpc.tf

You can find the official terraform/aws vpc module here, the code below is mostly the same with a little modification.

# Create vpc using the terraform aws vpc module
module "vpc" {
  source                  = "terraform-aws-modules/vpc/aws"

  name                    = var.vpcname
  cidr                    = "10.0.0.0/16"

  azs                     = slice(data.aws_availability_zones.available.names, 0, 3)
  private_subnets         = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets          = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway      = true
  single_nat_gateway      = true
  enable_dns_hostnames    = true

 public_subnet_tags = {
  "kubernetes.io/role/internal-elb"             = 1 # If you want to deploy load balancers to a subnet, the subnet must have 
 }

 private_subnet_tags = {
  "kubernetes.io/role/internal-elb"             = 1 # If you want to deploy load balancers to a subnet, the subnet must have 
 }
}

Define VPC Variables

terraform/variables.tf

Create a variables.tf file in the terraform directory, we will define our variables in this file. If you noticed, we have already referenced some variables already in the code above and your code should already be throwing errors. To rectify them, create the file and add the below code to it:

variable "region" {
  description = "The region where the VPC will be located"
  type        = string
  default     = "us-east-1"
}

variable "vpcname" {
  description = "Vpc name"
  type        = string
}

Create EKS Cluster

We will also be using the official terraform EKS module to create our EKS cluster. You can find the EKS Module here.

Create a new file in the terraform directory, name it eks.tf, this is where we will store our terraform script to create an eks cluster.

The reason we are breaking down our code into several files is for readability and maintainability, it makes the code easier to read and maintain when all scripts that fall into the same group are found in the same place.

terraform/eks.tf

Add the below code to the eks.tf file

module "eks" {
  source                                   = "terraform-aws-modules/eks/aws"
  version                                  = "~> 20.0"

  cluster_name                             = var.cluster_name
  cluster_version                          = "1.30"

  cluster_endpoint_public_access           = true

  vpc_id                                   = module.vpc.vpc_id
  subnet_ids                               = module.vpc.private_subnets

  # EKS Managed Node Group(s)
  eks_managed_node_group_defaults = {
    # Starting on 1.30, AL2023 is the default AMI type for EKS managed node groups
    ami_type                               = "AL2023_x86_64_STANDARD"
  }

  eks_managed_node_groups = {
    one = {
      name                                 = "node-group-1"
      instance_types                       = ["t2.medium"]

      min_size                             = 1
      max_size                             = 3
      desired_size                         = 2
    }

    two = {
      name                                 = "node-group-2"
      instance_types                       = ["t2.medium"]

      min_size                             = 1
      max_size                             = 2
      desired_size                         = 1
    }
  }

  # Cluster access entry
  # To add the current caller identity as an administrator
  enable_cluster_creator_admin_permissions = true

}

Looking at the above code, you will notice that we referenced a variable and so we need to declare that variable in our variables.tf file, and so we will do that next.

terraform/variables.tf

Add this to your variables.tf file

variable "cluster_name" {
  description = "Name of EKS cluster"
  type        = string
}

Declare Outputs

We will need some details from our configuration after the provisioning of our resources is over to enable us deploy our app and so we will tell terraform to give us those details when it is done.

We do this by defining our outputs in an output.tf file. Create another file in the terraform directory, name it outputs.tf

terraform/outputs.tf

Add the below code to your outputs.tf file

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

output "cluster_oidc_issuer_url" {
  description = "The URL on the EKS cluster for the OpenID Connect identity provider"
  value       = module.eks.cluster_oidc_issuer_url
}

output "aws_account_id" {
  description = "Account Id of your AWS account"
  sensitive = true
  value = data.aws_caller_identity.current.account_id
}

output "cluster_certificate_authority_data" {
  description = "Base64 encoded certificate data required to communicate with the cluster"
  sensitive = true
  value = module.eks.cluster_certificate_authority_data
}

Add Values for your Variables

Next we need to set values for all the variables we defined in our variables.tf file. We will do this in a *.tfvars file, terraform will recognise any file with this extension as holding the values for your variables and look here for those variables values.

This file usually holds secrets (although non of our variables are secrets of such) and should therefore never be committed to version control in other to avoid exposing your secrets.

If you copied the .gitignore file in the link at the beginning of this post then your .tfvars file will be ignored by version control.

Create a new file terraform.tfvars in the terraform directory and add the code below, substituting it with your details.

terraform/terraform.tfvars

vpcname = "<your vpc name>"
cluster_name = "your eks cluster name"

Test Configuration

Now we can test our configuration to see what will be created and ensure we have no errors in our script.

On your terminal, navigate to the terraform directory (where you saved all your terraform scripts) and run the following commands on the terminal:

terraform init

terraform plan

After running those commands you will see that, just like the image above, terraform will create 63 resources for us when we run the terraform apply command.

We won't apply the configuration just yet, we still have to create some extra roles.

Create Policy and Role for Route53 to Assume in the ClusterIssuer Process

While writing the configuration to spin up my VPC and other networking resources as well as my EKS I also added configuration to configure IAM roles and policies for Route 53 with cert-manager.

I created an IAM role with a trust policy that specifies the Open ID Connect (OIDC) provider and conditions for when the role can be assumed based on the service account and namespace.

The ClusterIssuer will need these credentials for the certificate issuing process and as a safe way to handle my secrets I will use IAM roles associated with Kubernetes service accounts to manage access to AWS services securely. This is why it is necessary to create this policy and role for Route53 and I did it using terraform.

You can find the script to create the role here

Create Roles and Policy for Route53

Down the line we will create a certificate for our domain with LetsEncrypt. This process will need us to create a ClusterIssuer for Let’s Encrypt that uses DNS-01 validation and since we are using AWS Route 53 as our DNS provider we will need to pass Route53 credentials for domain verification.

The ClusterIssuer will need these credentials for the certificate issuing process and as a safe way to handle our secrets we will use IAM roles associated with Kubernetes service accounts to manage access to AWS services securely. This is why it is necessary to create the next set of policy and role for Route53.

We will add the configuration to our terraform script.

terraform/route53-role-policy.tf

First we need to create the policy document for the policy that grants the necessary permissions for managing Route 53 DNS records.

Create a new file route53-role-policy.tf in the terraform directory and add the below code block to it.

# Retrieves the current AWS account ID to dynamically reference it in the policy document
data "aws_caller_identity" "current" {}

# Policy document for the Route53CertManagerPolicy
data "aws_iam_policy_document" "route53_policy" {
  version = "2012-10-17"
    statement {
      effect = "Allow"
      actions = ["route53:GetChange"]
      resources = ["arn:aws:route53:::change/*"]
    }
    statement {
      effect = "Allow"
      actions = ["route53:ListHostedZones"]
      resources = ["*"]
    }
    statement {
      effect = "Allow"
      actions = [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ]
      resources = ["arn:aws:route53:::hostedzone/*"]
    }
    statement {
      effect = "Allow"
      actions = [ 
        "route53:ListHostedZonesByName",
        "sts:AssumeRoleWithWebIdentity"
      ]
      resources = ["*"]
    }
}

terraform/route53-role-policy.tf

Now we create the policy, add the next block of code to the terraform/route53-role-policy.tf file

# Create an IAM policy for Route53 that grants the necessary permissions for managing Route 53 DNS records based on the above policy document
resource "aws_iam_policy" "route53_policy" {
  name = "Route53CertManagerPolicy"
  policy = data.aws_iam_policy_document.route53_policy.json
}

We will create an IAM role for Service accounts (IRSA) for the Kubernetes service account we will create soon.

IRSA enables assigning IAM roles to Kubernetes service accounts. This mechanism allows pods to use AWS resources securely, without needing to manage long-lived AWS credentials.

Before we create the role let's define our trust relationship policy document with a data block like we did with the policy document before.

terraform/route53-role-policy.tf

A trust relationship establishes a trust relationship between your Kubernetes cluster and AWS, allowing Kubernetes service accounts to assume IAM roles.

Add the below code to your terraform/route53-role-policy.tf file:

# Strip the "https://" prefix from the OIDC issuer URL
locals {
  cluster_oidc_issuer = replace(module.eks.cluster_oidc_issuer_url, "https://", "")
}

# Trust relationship policy document for the Route53CertManagerRole we will create
data "aws_iam_policy_document" "oidc_assume_role" {
  version = "2012-10-17"
  statement {
    effect = "Allow"
    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.cluster_oidc_issuer}"]
    }
    actions = [
      "sts:AssumeRoleWithWebIdentity",
    ]

    condition {
      test     = "StringEquals"
      variable = "${local.cluster_oidc_issuer}:sub"
      values   = [
        "system:serviceaccount:${var.namespace}:${var.service_account_name}"
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.cluster_oidc_issuer}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

Now we need to declare the variables we just called to our variables.tf file

terraform/variables.tf

variable "namespace" {
  description = "The Kubernetes namespace for the service account"
  type        = string
}

variable "service_account_name" {
  description = "The name of the Kubernetes service account"
  type        = string
}

Now we can create the IRSA, do this by adding the following code to the terraform/route53-role-policy.tf file

terraform/route53-role-policy.tf

# Create IAM Role for service account
resource "aws_iam_role" "Route53CertManagerRole" {
  name               = "Route53CertManagerRole"
  assume_role_policy = data.aws_iam_policy_document.oidc_assume_role.json
}

Lastly, we need to attach the policy we created above to this newly created role

terraform/route53-role-policy.tf

Add this to your terraform/route53-role-policy.tf file:

# Attach the Route53CertManagerPolicy to the Route53CertManagerRole
resource "aws_iam_role_policy_attachment" "Route53CertManager" {
  role = aws_iam_role.Route53CertManagerRole.name
  policy_arn = aws_iam_policy.route53cmpolicy.arn
}

Update Your secrets `terraform.tfvars` File

We added some new variable so we need to add the values for these variables to our .tfvars file.

Add the following to your file and substitute the values with your correct values

terraform/terraform.tfvars

namespace = "<the name of the namespace you will create for your service account>"
service_account_name = "<your service account name>"

⚠️ Note

Take note of the name you specify here as it must match the name you use for your service account when you create it.

I will advice you use cert-manager for both values (namespace and service_account_name), that's what I will be using.

Create EKS Resources

I provisioned my EKS cluster at this point to ensure there was no error up to this point.

Now we can go ahead and create our VPC and EKS cluster with terraform.

Run the below command:

terraform apply --auto-approve

The screenshots below show a successful deployment of the EKS cluster.

Terraform CLI Showing the Successful Deployment of the Resources

AWS Console Showing the EKS Cluster

AWS Console Showing the VPC Deployed Along with the EKS Cluster

Configure HTTPS Using Let’s Encrypt

Before we deploy our application let's go ahead and configure HTTPS using Let's Encrypt. We will be using terraform as well, using the Kubernetes provider and the kubectl provider. We will write a new terraform configuration for this, is this best so that the EKS cluster configuration is separate in other to breakdown the process.

You can find the terraform scripts for this deployment in the K8s-terraform directory here

Now let's write our terraform script together.

Add Providers

Create a new directory k8s-terraform, and create a new file main.tf. We will add the AWS, kubernetes, kubectl and helm providers as we will need them for the next set of configurations. We will also add our remote backend. We would then need to also add the configurations for those providers.

Open your main.tf file and add the following to it:

k8s-terraform/main.tf

# Add required providers
terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }

    kubernetes = {
      source = "hashicorp/kubernetes"
    }

    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "~> 1.14"
    }

    helm = {
      source  = "hashicorp/helm"
    }
  }

  backend "s3" {
    bucket         = "sockshop-statefiles" # Replace with your bucket name
    key            = "k8s/terraform.tfstate" # Replace with your second folder name
    region         = "us-east-1"
    dynamodb_table = "sock-shop-lockfile" # Replace with your DynamoDb table name
  }
}

Now we can go ahead to add and configure our providers.

Configure Provider

Update your main.tf file, add the below to your file

k8s-terraform/main.tf

# Configure the AWS Provider
provider "aws" {
  region = var.region
  shared_credentials_files = ["~/.aws/credentials"]
}

# Configure the kubernetes Provider, the exec will use the aws cli to retrieve the token 
provider "kubernetes" {
  host = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
  }
}

# Configure the kubernetes Provider, the exec will use the aws cli to retrieve the token 
provider "kubectl" {
  host = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
  }
}

# Configure the helm Provider using the kubernetes configuration
provider "helm" {
  kubernetes {
    host = var.cluster_endpoint
    cluster_ca_certificate = base64decode(var.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", var.cluster_name]
    }
  }
}

We need to retrieve some values from AWS so create another file data.tf in the k8s-terraform directory and add the below code to it:

k8s-terraform/data.tf

# Retrieves the current AWS account ID to dynamically reference it in the policy document
data "aws_caller_identity" "current" {}

# Retrieve details of the current AWS region to dynamically reference it in the configuration
data "aws_region" "current" {}

# Retrieve the eks cluster endpoint from AWS
data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

Declare Variables

We have referenced some variables in our configuration and so we need to declare them for terraform. To declare the variables we used in the main.tf file, create a new file variables.tf and add the code:

k8s-terraform/variables.tf

variable "region" {
  description = "The region where the VPC will be located"
  type        = string
  default     = "us-east-1"
}

variable "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  type        = string
}

variable "cluster_certificate_authority_data" {
  description = "Base64 encoded certificate data required to communicate with the cluster"
  type        = string
}

Let's begin writing the actual configuration

Create Kubernetes Service Account for the Cert Manager to use

Earlier, while writing our EKS cluster configuration, we added a configuration to create an IAM role for service account (IRSA). The first thing we will do here was to create the namespace for cert-manager and also create a service account and annotate it with the IAM role.

Before we create the service account we will create the namespace for cert-manager as our service account will exit in the cert-manager namespace.

To create a namespace for cert-manager use the kubernetes-namespace resource. Create a new file cert-manager.tf in the k8s-terraform directory and add this code in:

k8s-terraform/cert-manager.tf

Add this code to the file

# Create a cert-manager namespace that our service account will use
resource "kubernetes_namespace" "cert-manager" {
  metadata {
    name = "cert-manager"
  }
}

Now to the actual creation of the service account, add this code to your configuration

k8s-terraform/cert-manager.tf

# Create the service account for cert-manager
resource "kubernetes_service_account" "cert_manager" {
  metadata {
    name      = "cert-manager"
    namespace = "cert-manager"
    annotations = {
      "eks.amazonaws.com/role-arn" = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/Route53CertManagerRole"
    }
  }
}

Configure Ingress Controller

Before creating the cert-manager resource we will configure our ingress controller, it's crucial to ensure that your Ingress controller is deployed and running before you create Ingress resources that it will manage.

If you will like to go ahead, you can find the complete configuration of the ingress controller here. It will be deployed using helm, using the helm_release resource in terraform.

Let's get to writing the configuration together:

First we will create an ingress-nginx namespace where the ingress controller will reside, we are creating this manually cos helm has a behaviour of not deleting the namespaces it creates.

Create a new file ingress.tf in the k8s-terraform directory and add the code, this is the code to create both the namespace and deploy the ingress controller using helm.

k8s-terraform/ingress.tf

# Create the ingress-nginx namespace
resource "kubernetes_namespace" "ingress-nginx" {
  metadata {
    name = "ingress-nginx"
  }
}

# Use helm to create an nginx ingress controller
resource "helm_release" "ingress-nginx" {
  name       = "nginx-ingress"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = "ingress-nginx"
  create_namespace = false
  cleanup_on_fail = true
  force_update = true
  timeout = 6000

  set {
    name  = "controller.service.name"
    value = "ingress-nginx-controller"
  }

  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }

  set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-connection-idle-timeout"
    value = "3600"
  }

  set {
    name  = "controller.publishService.enabled"
    value = "true"
  }

  set {
    name  = "controller.config.cleanup"
    value = "true"
  }

  set {
    name  = "controller.extraArgs.default-ssl-certificate"
    value = "sock-shop/${var.domain}-tls"
  }

  depends_on = [ kubernetes_namespace.ingress-nginx, kubernetes_namespace.sock-shop  ]
}

Declare Variables

Declare the domain name variable in the variables.tf file

k8s-terraform/variables.tf

variable "domain" {
  description = "The domain name to access your application from and use in the creation of your SSL certificate"
  type = string
}

Configure Cert-Manager

After configuring the ingress controller, the next thing to do is to configure the cert-manager, I did this also using helm. Find the configuration here

Now we will deploy cert-manager using a helm chart with the helm_release resource in terraform.

Add this code to the k8s-terraform/cert-manager.tf file

k8s-terraform/cert-manager.tf

# Use Helm to deploy the cert-manager
resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  namespace  = "cert-manager"
  create_namespace = false
  cleanup_on_fail = true
  force_update = true

  set {
    name  = "installCRDs"
    value = "true"
  }

  set {
    name  = "serviceAccount.create"
    value = "false"
  }

  set {
    name  = "serviceAccount.name"
    value = kubernetes_service_account.cert_manager.metadata[0].name
  }

  set {
    name  = "securityContext.fsGroup"
    value = "1001"
  }

  set {
    name  = "controller.config.cleanup"
    value = "true"
  }

  set {
    name = "helm.sh/resource-policy"
    value = "delete"
  }

  depends_on = [ kubernetes_service_account.cert_manager, kubernetes_namespace.cert-manager, helm_release.ingress-nginx ]
}

RBAC (Role-based access control )

In order to allow cert-manager issue a token using your ServiceAccount you must deploy some RBAC (Role-based access control ) to the cluster. Find the complete code here

Create a new file role-roleBinding.tf in the k8s-terraform directory and add the following to the file:

k8s-terraform/role-roleBinding.tf

# Deploy some RBAC to the cluster
resource "kubectl_manifest" "cert-manager-tokenrequest" {
  yaml_body = <<YAML
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cert-manager-tokenrequest
  namespace: cert-manager
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["cert-manager"]
    verbs: ["create", "update", "delete", "get", "list", "watch"]
YAML
}

# Deploy a RoleBinding to the cluster
resource "kubectl_manifest" "cmtokenrequest" {
  yaml_body = <<YAML
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-manager-cert-manager-tokenrequest
  namespace: cert-manager
subjects:
  - kind: ServiceAccount
    name: cert-manager
    namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cert-manager-tokenrequest
YAML
}

Configure ClusterIssuer

Next we will configure the ClusterIssuer manifest file with the kubectl_manifest resource from the kubectl provider so that terraform can adequately use it in the certificate issuing process. We will use the DNS01 resolver instead of HTTP01

I had wanted to use the kubernetes_manifest resource from the kubernetes provider even though I knew it would require two stages of the terraform apply command as the cluster has to be accessible at plan time and thus cannot be created in the same apply operation, another limitation of the kubernetes_manifest resource is that it doesn't support having multiple resources in one manifest file, to circumvent this you could either break down your manifest files into their own individual files (but where's the fun in that) or use a for_each function to loop through the single file.

However from research I was able to discover that the kubectl's kubectl_manifest resource handles manifest files better and allows for a single stage run of the terraform apply command.

Find the ClusterIssuer configuration file here.

To configure our ClusterIssuer create a new file cluster-issuer.tf in the k8s-terraform directory and add the below code to it.

k8s-terraform/cluster-issuer.tf

# Create the Cluster Issuer for the production environment
resource "kubectl_manifest" "cert_manager_cluster_issuer" {
  yaml_body = <<YAML
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ${var.email}
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        route53:
          region: ${data.aws_region.current.name}
          hostedZoneID: ${data.aws_route53_zone.selected.zone_id}
        auth:
            kubernetes:
              serviceAccountRef: 
                name: cert-manager
                namespace: cert-manager
YAML
  depends_on = [helm_release.cert_manager]
}

Declare Variables

Update your variable.tf file by adding the following to it:

k8s-terraform/variables.tf

variable "email" {
  description = "The email address to use in the creation of your SSL certificate"
  type = string
}

Create Certificate

To create the certificate we will use the kubectl_manifest resource to define our manifest file for the certificate creation. You can find my certificate manifest file here

Create a new file certificate.tf in the k8s-terraform directory and add the below code to it.

First we will create a sock-shop namespace as this is the namespace where we want our certificate to be saved. Our certificate will be in this namespace as this is the namespace where our application will be in, the application and the certificate have to be in the same namespace so that the SSl certificate can apply correctly to our site.

The creation of the certificate will depend on the sock-shop namespace and so the depends_on argument is added to the configuration.

k8s-terraform/certificate.tf

# Create a sock-shop namespace that our service account will use
resource "kubernetes_namespace" "sock-shop" {
  metadata {
    name = "sock-shop"
  }
}

# Resource to create the certificate
resource "kubectl_manifest" "cert_manager_certificate" {
  yaml_body = <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${var.domain}-cert
  namespace: sock-shop  
spec:
  secretName: ${var.domain}-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  commonName: ${var.domain}
  dnsNames:
    - ${var.domain}
    - "*.${var.domain}"
YAML
depends_on = [ kubernetes_namespace.sock-shop, kubectl_manifest.cert_manager_cluster_issuer ]
}

Declare Variables

We have referenced some more variables in our configuration so we will declare them for terraform as usual. Update your variables.tf file accordingly

terraform/variables.tf

variable "domain" {
  description = "The domain name to access your application from and use in the creation of your SSL certificate"
  type = string
}

Configure Ingress

Now that we have configured Cert Manager, Cluster Issuer and Certificate we need to setup our Ingress Controller and Ingress resource that will allow us access to our application, we will also be doing this using our terraform configuration.

I decided to use the nginx-ingress helm chart with the aid of the helm-release resource from the helm provider.

Find my ingress configuration here

Create a new file ingress.tf in the k8s-terraform directory and add the code below to it.

Our Ingress will be located in the ingress-nginx namespace and so we will create the namespace first.

After creating the namespace we will create the ingress controller that controls the ingress resources and creates a LoadBalancer. The creation of the ingress controller will depend on the ingress-nginx and the sock-shop namespace and so the depends_on argument is added to the configuration.

k8s-terraform/ingress.tf

# Create the ingress-nginx namespace
resource "kubernetes_namespace" "ingress-nginx" {
  metadata {
    name = "ingress-nginx"
  }
}

# Use helm to create an nginx ingress controller
resource "helm_release" "ingress-nginx" {
  name       = "nginx-ingress"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = "ingress-nginx"
  create_namespace = false
  cleanup_on_fail = true
  force_update = true
  timeout = 6000

  set {
    name  = "controller.service.name"
    value = "ingress-nginx-controller"
  }

  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }

  set {
    name  = "controller.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-connection-idle-timeout"
    value = "3600"
  }

  set {
    name  = "controller.publishService.enabled"
    value = "true"
  }

  set {
    name  = "controller.config.cleanup"
    value = "true"
  }

  set {
    name  = "controller.extraArgs.default-ssl-certificate"
    value = "sock-shop/${var.domain}-tls"
  }

  depends_on = [ kubernetes_namespace.ingress-nginx, kubernetes_namespace.sock-shop  ]
}

Now we can then create the actual ingress, this will be located in the sock-shop namespace as our resources will be located there and for effective flow of traffic the ingress needs to be in the same namespace as the application. Add the below code to the k8s-terraform/ingress.tf file.

k8s-terraform/ingress.tf

# Create an Ingress resource using the kubectl_manifest resource
resource "kubectl_manifest" "ingress" {
  depends_on = [ kubectl_manifest.cert_manager_cluster_issuer, kubectl_manifest.cert_manager_certificate ]
  yaml_body = <<YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  namespace: sock-shop
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: dns01
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - ${var.domain}
    - "www.${var.domain}"
  secretName: "${var.domain}-tls"
  rules:
  - host: ${var.domain}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: front-end
            port:
              number: 80
  - host: "www.${var.domain}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: front-end
            port:
              number: 80
YAML
}

Connect Domain to LoadBalancer

The Ingress-controller will create a LoadBalancer that give us an external IP to use in accessing our resources and we will point our domain to it by creating an A record.

I used this LoadBalancer to create two A records, one for my main domain name and the other for a wildcard (*) for my subdomains, this will enable us access the sock shop application from our domain.

Remember the hosted zone we created at the beginning of this project? We need information about it to be available to terraform so we need to retrieve the data from the hosted zone using the data block.

We will also retrieve the name of the LoadBalancer created by our ingress controller. When I was using the exact name created by the ingress controller, I wss getting an error that the LoadBalancer's name was too long so I retrieved the name and split it to make it shorter. It is all shown in the code block below.

Add the following to you k8s-terraform/data.tf file

k8s-terraform/data.tf

# Retrieve the Route53 hosted zone for the domain
data "aws_route53_zone" "selected" {
  name         = var.domain
}

# Retrieve the ingress load balancer hostname
data "kubernetes_service" "ingress-nginx-controller" {
  metadata {
    name      = "nginx-ingress-ingress-nginx-controller"
    namespace = "ingress-nginx"
  }

  # Ensure this data source fetches after the service is created
  depends_on = [
    helm_release.ingress-nginx
  ]
}

# Extract the load balancer name from the hostname
locals {
  ingress_list = data.kubernetes_service.ingress-nginx-controller.status[0].load_balancer[0].ingress
  lb_hostname  = element(local.ingress_list, 0).hostname
  lb_name     = join("", slice(split("-", local.lb_hostname), 0, 1))
}

# Data source to fetch the AWS ELB details using the extracted load balancer name
data "aws_elb" "ingress_nginx_lb" {
  name = local.lb_name
}

As you can see in the above block of code, the original LoadBalancer name was split using the locals block and the new name is set with a data block.

Now we can go ahead and create the A records. Add the following to your k8s-terraform/ingress.tf file.

k8s-terraform/ingress.tf

# Route 53 record creation so that our ingress controller can point to our domain name
resource "aws_route53_record" "ingress_load_balancer" {
  zone_id = data.aws_route53_zone.selected.zone_id  # Replace with your Route 53 Hosted Zone ID
  name    = var.domain # Replace with the DNS name you want
  type    = "A"

  # Use the LoadBalancer's external IP or DNS name
  alias {
    name                   = data.aws_elb.ingress_nginx_lb.dns_name
    zone_id                = data.aws_elb.ingress_nginx_lb.zone_id  # zone ID for the alias
    evaluate_target_health = true
  }
}

# Route 53 record creation so that our ingress controller can point to subdomains of our domain name
resource "aws_route53_record" "ingress_subdomain_load_balancer" {
  zone_id = data.aws_route53_zone.selected.zone_id
  name    = "*.${var.domain}"
  type    = "A"
  alias {
    name                   = data.aws_elb.ingress_nginx_lb.dns_name
    zone_id                = data.aws_elb.ingress_nginx_lb.zone_id
    evaluate_target_health = true
  }
}

Define Outputs

In your k8s-terraform/output.tf add the following:

k8s-terraform/output.tf

output "ingress_load_balancer_dns" {
  description = "The dns name of the ingress controller's LoadBalancer"
  value = data.kubernetes_service.ingress-nginx-controller.status[0].load_balancer[0].ingress[0].hostname
}

output "ingress_load_balancer_zone_id" {
  value = data.aws_elb.ingress_nginx_lb.zone_id
}

Monitoring, Logging and Alerting

To setup prometheus, grafana, alertmanager and Kibana for monitoring, logging and alerting retrieve the respective manifest files from my Github repo. We will then create two additional ingresses that will exist in the monitoring namespace and the kube-logging namespace so that we can access these dashboards from your subdomain. We will also copy the SSL secret covering the entire domain to the monitoring namespace and the kube-logging namespace.

Download the alerting manifest files from my github here

Download the logging manifest files from my github here

Download the monitoring manifest files from my github here

Add these three directories to your project directory.

Now we will create two new ingresses.

Create a Prometheus and Grafana Ingress

To create an ingress that will point to the prometheus, grafana and alertmanager subdomains, open the k8s-terraform/ingress.tf file add the below code:

k8s-terraform/ingress.tf

# Create an Ingress resource using the kubectl_manifest resource for our monitoring resources
resource "kubectl_manifest" "ingress_monitoring" {
  depends_on = [ kubectl_manifest.cert_manager_cluster_issuer, null_resource.update_secret ]
  yaml_body = <<YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: dns01
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - "prometheus.${var.domain}"
    - "grafana.${var.domain}"
    - "alertmanager.${var.domain}"
  secretName: "${var.domain}-tls"
  rules:
  - host: "prometheus.${var.domain}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus
            port:
              number: 9090
  - host: "grafana.${var.domain}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 80
  - host: "alertmanager.${var.domain}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: alertmanager
            port:
              number: 9093
YAML
}

Add Secret to the Monitoring and Kube-logging Namespace

If you remember, the SSL certificate we created is in the sock-shop namespace as well as the secret for that certificate but we need the secret to also exist in the monitoring namespace if we will use that SSL for our monitoring sub-domain.

We can create separate certificates and include them in these namespaces but I do not want to go that route. What we will do is copy the secret from the sock-shop namespace to the monitoring and kube-system namespace.

We need to fetch the existing secret, using the null_resource with the local-exec provisioner. It runs aws eks update-kubeconfig to configure kubectl then fetches the Secret from the sock-shop namespace, modifies it and saves it in the monitoring and kube-logging namespace.

The kube-logging namespace is where our elasticsearch, Fluentd and Kibana resources will reside so will will create that first before sending any secret there.

To create the kube-logging namespace, add the below code to k8s-terraform/ingress.tf file:

k8s-terraform/ingress.tf

# Create kube-logging namespace
resource "kubernetes_namespace" "kube-logging" {
  metadata {
    name = "kube-logging"
  }
}

Add the following code to your k8s-terraform/data.tf file:

k8s-terraform/data.tf

# Retrieve the certificate secret and copy to the monitoring namespace
resource "null_resource" "update_secret" {
  provisioner "local-exec" {
    command = <<EOT
    aws eks update-kubeconfig --region us-east-1 --name sock-shop-eks
    kubectl get secret projectchigozie.me-tls -n sock-shop -o yaml | sed 's/namespace: sock-shop/namespace: monitoring/' | kubectl apply -f -
    EOT
  }
  # Ensure this data source fetches after the service is created
  depends_on = [kubectl_manifest.cert_manager_certificate]
}

To create the secret in the kube-logging namespace, we will just repeat the same code changing only the namespace in the metadata.

Add this code to your k8s-terraform/certificate.tf file

k8s-terraform/certificate.tf

# Retrieve the certificate secret and copy to the kube-logging namespace
resource "null_resource" "update_secret_kibana" {
  provisioner "local-exec" {
    command = <<EOT
    aws eks update-kubeconfig --region us-east-1 --name sock-shop-eks
    kubectl get secret projectchigozie.me-tls -n sock-shop -o yaml | sed 's/namespace: sock-shop/namespace: kube-logging/' | kubectl apply -f -
    EOT
  }
  # Ensure this data source fetches after the service is created
  depends_on = [kubectl_manifest.cert_manager_certificate]
}

The records for the subdomains are covered by the wildcard (*) record we added so we do not need to add any more A records.

Create Ingress for Kibana

To create an ingress that will point to the kibana subdomains open the k8s-terraform/ingress.tf file add the below code:

k8s-terraform/ingress.tf

# Create an Ingress resource using the kubectl_manifest resource for our kibana resources
resource "kubectl_manifest" "ingress_kibana" {
  depends_on = [ kubectl_manifest.cert_manager_cluster_issuer, null_resource.update_secret_kibana ]
  yaml_body = <<YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  namespace: kube-logging
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: dns01
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - "kibana.${var.domain}"
  secretName: "${var.domain}-tls"
  rules:
  - host: "kibana.${var.domain}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kibana
            port:
              number: 5601
YAML
}

⚠️ NOTE

To use Alertmanager and have it send slack alerts you need to configure your slack webhook, I will make a separate post on how to configure that so that you can get the messages in your slack.

For now you can skip it by not applying the alerting configurations.

Set Environment Variables (Variable Value)

If you noticed, so far we haven't defined any of our variables. We have declared them in our variables.tf file, we have referenced them in a lot of our configurations but we haven't added any value for any them.

Typically we do this using a .tfvars file or by setting them as environment variables. We will be using the latter but with a twist. Most of the variables we used here are the same variables we used in our EKS configuration and so we won't be repeating ourselves.

When we defined our EKS configuration we defined some outputs and all those outputs are exactly the same with our variables, so we will write a script to take the outputs of the EKS configuration and set them as env vars for this configuration to use.

Our setup is in such a way that we will build the EKS cluster first, then using the outputs from that deployment we will set the environment variables for our next terraform build to create our ingress resources and SSL certificate.

We will also export our terraform output values as environment variables to use with kubectl and other configurations. This will aid to make the whole process more automated reducing the manual configurations.

I wrote different scripts to do this, find the scripts here, the first script will create the terraform variables that we will use to run our next deployment, find it here

This script will extract the key and value from your terraform output and then generate a new script file env.sh containing the export commands for the Terraform inputs of the next terraform deployment, it also takes into consideration outputs that are marked as sensitive and handles them appropriately and so sets both regular and sensitive Terraform outputs as environment variables.

To write the scripts, create a new directory scripts in your root directory. In the scripts directory, create a file 1-setTFvars.sh we will save our script to export our environment variables here.

scripts/1-setTFvars.sh

Add the following block of code to your file:

#!/usr/bin/env bash

# This script sets the Terraform variables from the outputs of the Terraform configuration. Run this script immediately after the first terraform apply that creates the EKS cluster.

# Set the path where the script is being run from
RUN_FROM=$(pwd)

# Define the directory containing your Terraform configuration files by walking backwards to the project root path so that this script works from whatever directory it is run.
ABS_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
DIR_PATH=$(dirname "$ABS_PATH")
PROJECT_PATH=$(dirname "$DIR_PATH")
TF_PATH="$PROJECT_PATH/terraform"

# Change to the Terraform directory
cd "$TF_PATH" || { echo "Failed to change directory to $TF_PATH"; return 1; }

# Check if jq is installed
if ! command -v jq &> /dev/null; then
  echo "jq is not installed. Please install jq and try again."
  return 1
fi

# Get the Terraform outputs and check if they are non-empty
TF_OUTPUT=$(terraform output -json)

if [ -z "$TF_OUTPUT" ] || [ "$TF_OUTPUT" == "{}" ]; then
  echo "No terraform outputs found or outputs are empty. Please run 'terraform apply' first."
  return 1
fi

# Generate TFenv.sh file with export commands
{
  echo "$TF_OUTPUT" | jq -r 'to_entries[] | "export TF_VAR_" + .key + "=" + (.value.value | tostring)'
  echo 'echo "Environment variables have been exported and are available for Terraform."'
} > "$RUN_FROM/TFenv.sh"

echo "Terraform variables export script has been created as TFenv.sh."

# Make it executable
chmod u+x "$RUN_FROM/TFenv.sh"

# Source the TFenv.sh script to export the variables into the current shell
echo "Sourcing TFenv.sh..."
source "$RUN_FROM/TFenv.sh"
cd $RUN_FROM

The script generates a new script file TFenv.sh containing the export commands and then run the script to use for our next Terraform build.

⚠️ NOTE

This script takes about 15 seconds to run, let it run its course.

Make Script Executable

You don't necessarily have to make the script executable as we will run it with the source command but if you want to, save the script as 1-setTFvars.sh and make it executable using the command below:

chmod u+x 1-setTFvars.sh

Run script

Now we run the script to set our environment variables. Take note of the directory from which you are running the script and adjust the command to reflect the correct path to the script.

Use the command below:

source scripts/1-setTFvars.sh

The new script generated from this script is not committed to version control as it contains some sensitive values.

Create Resources

We've broken down our infrastructure into a two stage build where we will create our EKS cluster first before we configure certificate and ingress, so before you apply this script the first terraform configuration script should have already been deployed.

Now you are ready to create your resources. Ensure you deploy your EKS cluster using the first Terraform script then make sure you run your first script to set your environment variables. (if you are following this post sequentially you wont have done this step just before getting to this section so skip ahead)

source scripts/1-setTFvars.sh

Then apply your terraform script, you should be in the k8s-terraform directory before running the below commands.

terraform init
terraform plan
terraform apply

Now that we have our infrastructure up and running we need to connect kubectl to our cluster so that we can deploy our application and manage our resources.

Connect Kubectl to EKS Cluster

Once my EKS Cluster is fully provisioned on AWS and I have deployed my ingress and certificate resources in the cluster, the next thing to do is to connect kubectl to the cluster so that I can use kubectl right from my local machine to define, create, update and delete my Kubernetes resources as necessary.

The command to do this is shown below:

aws eks update-kubeconfig --region <region-code> --name <cluster name>

However since this is an imperative command I decided to create a script out of it for easier automation and reproduction of the process. Find the script here

Before we create the script to connect the kubectl though, will need to set our environment variables for the script to use, this will be the same as the first script with a little modification. Create a new file 2-setEnvVars.sh in the scripts directory and add the below code to the file:

scripts/2-setEnvVars.sh

#!/usr/bin/env bash

# This script sets environment variables from Terraform outputs for use in connect our kubectl to our cluster. Run this after applying the k8s-terraform configuration.

# Set the path where the script is being run from
RUN_FROM=$(pwd)

# Define the directory containing your Terraform configuration files by walking backwards to the project root path so that this script works from whatever directory it is run.
ABS_PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
DIR_PATH=$(dirname "$ABS_PATH")
PROJECT_PATH=$(dirname "$DIR_PATH")
TF_PATH="$PROJECT_PATH/terraform"

# Change to the Terraform directory
cd "$TF_PATH" || { echo "Failed to change directory to $TF_PATH"; return 1; }

# Check if jq is installed
if ! command -v jq &> /dev/null; then
  echo "jq is not installed. Please install jq and try again."
  return 1
fi

# Get the Terraform outputs and check if they are non-empty
TF_OUTPUT=$(terraform output -json)

if [ -z "$TF_OUTPUT" ] || [ "$TF_OUTPUT" == "{}" ]; then
  echo "No terraform outputs found or outputs are empty. Please run 'terraform apply' first."
  return 1
fi

# Generate env.sh file with export commands
echo "$TF_OUTPUT" | jq -r 'to_entries[] | "export " + .key + "=" + (.value.value | tostring)' > "$RUN_FROM/env.sh"
echo "Environment variables export script has been created as env.sh."
echo "Run the 3-connect-kubectl.sh script now to connect Kubectl to your EKS cluster"
cd $RUN_FROM

You need to run this script before you run the script to connect your kubectl to your cluster. Make your script executable and run it:

chmod u+x 2-setEnvVars.sh
source 2-setEnvVars.sh

Script to connect Kubectl to Cluster

To create the script to connect your kubectl to your cluster create a new file 3-connect-kubectl.sh in the scripts directory and add the following code block to the file:

scripts/3-connect-kubectl.sh

#!/usr/bin/env bash

# Source environment variables from a file
if [ -f ./env.sh ]; then
  source ./env.sh
  echo "Environment variables loaded from env.sh file and successfully set."
else
  echo "Environment variables file not found."
  exit 1
fi

# Check that AWS CLI is installed, if it isn't stop the script
if ! command -v aws &> /dev/null; then
  echo "AWS CLI is not installed. Please install AWS CLI and try again."
  exit 1
fi

# Check that the kubectl is configured, if it isn't stop the script
if ! command -v kubectl &> /dev/null; then
  echo "kubectl is not installed. Please install kubectl and try again."
  exit 1
fi

# Connect kubectl to your EKS cluster
echo "Connecting kubectl to your EKS cluster......"
aws eks update-kubeconfig --region $region --name $cluster_name

Make the script Executable and Run it

Save the script and make it executable with the second command, then run it with the second command

chmod u+x 3-connect-kubectl.sh
./3-connect-kubectl.sh

Take note of the directory from which you are running the script and adjust the command to reflect the correct path to the script.

After running the script you should see that kubectl is connecting with your EKS cluster and you will get a success response if it was successful as seen in the screenshot below.

When the script is successfully run, our kubectl is connected to our cluster.

Deploy Application

Previously I had wanted to deploy the application using terraform but it seems like an overkill seeing as we using a CI/CD pipeline to automate the whole flow, I eventually resolved to use kubectl to deploy the application to the cluster.

Retrieve the complete-demo.yaml application file from the project repo which is a combination of all the manifests for all the microservices required for our application to be up and running.

Download the complete-demo.yaml file from my repo and add it to your project directory. For me I created a new directory called app and saved the application file there.

Take note of where your saved it cause you will deploy it from that location.

To deploy our app, run the command below:

kubectl apply -f app/complete-demo.yaml
kubectl apply -f monitoring/
kubectl apply -f logging/
kubectl apply -f alerting/

With the above commands you have now deployed your application as well as your monitoring, logging and alerting.

Verify Deployment

Head on over to your domain to see that the sock-shop application is showing up, this is what it should look like:

From the images above you can see that my frontend is secure and the certificate is issued my let's encrypt. If you followed along correctly, yours should be too.

Check your prometheus and grafana subdomains as well for your monitoring and logging dashboards

Prometheus Dashboard

Grafana Dashboard

Commands to use in exploring your cluster

We can also verify our setup from our EKS cluster using some kubectl commands:

🌟 To view the Resources in Cert-manager namespace:

kubectl get all -n cert-manager

Resources in Cert-manager namespace

🌟 To view your sock-shop ingress:

kubectl get ingress -n sock-shop

Sock-shop ingress

🌟 To see all our application pods:

kubectl get pods -n sock-shop

Sock-shop pods

🌟 To see all our application services:

kubectl get svc -n sock-shop

Sock-shop-services

This has been an extremely long post and so we will end here, be on the look out for another post where we setup a CI/CD pipeline with Jenkins for this project.

I enjoy breaking down tech topics into simpler concepts and teaching the world (or at least the DEV Community) how to harness the power of the cloud.

If you'd love to connect on all things cloud engineering or DevOps, then I'd love to as well, drop me a connection request on LinkedIn or follow me GitHub

I'm open to opportunities as well so don't be a stranger, drop a comment, drop a connection, send that DM and let's get cooking. Byeeee until later

Follow for more DevOps content broken down into simple understandable structure.

ChigozieCOFollow

Cloud and DevOps Engineer who's a little bit obsessed with the whole cloud computing scene, always ready to dive into the nitty-gritty of cloud infrastructure, DevOps and Security.

If this post has helped you, consider supporting me:

Automate Build, Test & Deploy Processes with GitHub Actions

ChigozieCO — Mon, 09 Sep 2024 16:22:42 +0000

This project is a quick sample project that demonstrates the automation of the build, test, and deployment process of an application to a staging environment on push to the main branch.

To adequately demonstrate the CI/CD pipeline, we will create a simple Python project with minimal code and then integrate it into GitHub Actions.

Create a Simple Python Project

Like I earlier stated, we will create a simple project that we will use in our pipeline. I chose to do this in python for no particular reason, you can use any other programming language you choose, the main of this project is to demonstrate the pipeline.

Create Project Folder

So go ahead and create the project folder and navigate into that folder:

mkdir automated-testing
cd automated-testing

Write Application File

Now we will write the simple python application. Create a new file app.py in the project folder.

touch app.py

Add the below code block to the file:

def hello():
  return "Hello, World!"

if __name__ == "__main__":
  print(hello())

This is a very simple python "Hello world" function that serves as an example of basic functionality that can be tested in the CI pipeline.

def hello() defines a function named hello which takes no arguments. When this function is called, it returns the string "Hello, World!".

if __name__ == "__main__" is a standard Python construct used to ensure that certain code only runs when the file is executed directly (not when imported as a module). It acts as an entry point for the script.

When app.py is run directly (e.g., by running python app.py), the script will call the hello() function and print the result, which is "Hello, World!".

Create a Requirements File.

A typical project would have dependencies and in a python project they are usually defined in the requirements.txt file.

Create a new file requirements.txt

touch requirements.txt

Add this to the file:

pytest

Create Unit Test

Now we will add a basic test file, test_app.py, to test the function in app.py. Add the below to the file:

from app import hello

def test_hello():
  assert hello() == "Hello, World!"

Now we are ready to create our pipeline.

Set Up GitHub Actions for CI/CD

To configure GitHub actions we need to create a .github/workflows folder inside our repo, this is how we notify GitHub of the CI/CD pipeline in our repo.

Create a new file:

mkdir -p .github/workflows

You can have multiple pipelines in one repo, so create a file proj.yml in the .github/workflows folder. This is where we will define the steps for building, testing, and deploying our Python project.

Add the below code to your file:

name: Build, Test and Deploy

# Trigger the workflow on pushes to the main branch
on:
  push:
    branches:
      - main

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
    # Checkout the code from the repository
    - name: Checkout repo
      uses: actions/checkout@v4

    # Set up Python environment
    - name: Setup Python
      uses: actions/setup-python@v5
      with:
        python-version: '3.x'

    # Install dependencies
    - name: Install Dependecies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

    # Build (this project deosn't require a build but we will simulate a build by creating a file)
    - name: Build Project
      run: |
        mkdir -p build
        # Simulate build output by creating a file
        touch build/output_file.txt

    # Run tests using pytest
    - name: Run tests
      run: 
        pytest

    # Upload the build output as an artifact (we created a file in the build step to simulate an artifact)
    - name: Upload build artifact
      uses: actions/upload-artifact@v4
      with:
        name: build-artifact
        path: build/

  deploy:
    runs-on: ubuntu-latest
    needs: build-and-test
    if: success()

    steps:
    # Download the artifact from the build stage
    - name: Download build artifact
      uses: actions/download-artifact@v4
      with:
        name: build-artifact
        path: build/

    - name: Simulate Deployment
      run: |
        echo "Deploying to staging..."
        ls build/

Breakdown of the CI/CD Pipeline Steps

Trigger on Push to main: The pipeline is triggered whenever there is a push to the main branch.

Checkout Code: This step uses GitHub’s checkout action to pull our code from the repository.

Set Up Python: The pipeline sets up a Python environment on the CI runner (GitHub's virtual machine), ensuring that the correct Python version is used.

Install Dependencies: It installs the required dependencies for our Python project (pytest in this case). This dependency was just added as an example of when a project has dependencies as this particular sample python application does not require any.

Build: This stage was also just added for demonstration purposes, supposing this was a JavaScript/Node.js project this is where we would run the npm run build command and this will create an artifact we can upload and use in the deploy stage. Since this is a python project and it doesn't really require a build, we will create a file and folder in this stage. The file will serve as our artifact for the deploy stage.

Run Tests: It runs the tests using pytest to validate the code.

Upload Build Artifact: After running the tests, the build-and-test stage creates and saves a build artifact (in this case, a simulated output_file.txt in the build folder from the build step). The action upload-artifact is used to store this artifact. You can replace this with whatever actual build output your project creates.

Deploy Stage: Our application will only be deployed if the test was successful which is why I have added the conditionals needs and if. Using “needs” we can require that the deploy job won’t even run unless the test job is successful. The download-artifact action retrieves the build artifact and the last step "Simulate Deployment" simulates deployment by printing a message and lists the artifact. If this was a live project we would have the actual deployment commands to deploy to a real staging environment here. You can replace the echo and ls commands with actual deployment commands (e.g., deploying to a cloud platform). This approach ensures that the output from the build-and-test stage is properly passed to the deploy stage, simulating how a real deployment would work with build artifacts.

Push to GitHub

If you haven't already, you need to initialize a git repository using the commands below:

git init
git add .
git commit -m "Create project as well as CI/CD pipeline"

Now we push to GitHub. Create a GitHub repository and push your code using the below commands:

git remote add origin <your-repo-url>
git push -u origin main

Verify the Pipeline

After pushing the code, you can visit the Actions tab in your GitHub repository. You should see the pipeline triggered, running the steps defined in your proj.yml file.

If everything is set up correctly, the pipeline will build, test, and simulate deployment. You can changes things around in your project and make new pushes to see the the pipeline works, create errors intentional so you can see how the pipeline works when the tests fail.

On a successful run this is how your Actions tab should look.

And that's it, this setup provides a working example of a CI/CD pipeline for a very basic Python project. If you found this helpful, please share with your connection and if you have any questions, do not hesitate to drop the question in the comments.

Deploy a Static Website with Route53, CloudFront and AWS Certificate using a Terraform Script

ChigozieCO — Wed, 10 Jul 2024 13:51:05 +0000

Automation is queen in this side of our world, the more of your work you can automate, the better.

The use of Terraform is very necessary for cloud engineers in order to automate deployments of your infrastructure. Terraform is an infrastructure as code tool that lets you define infrastructure resources in human-readable configuration files that you can version, reuse, and share. You can then use a consistent workflow to safely and efficiently provision and manage your infrastructure throughout its lifecycle.

Terraform Provider and Initialize Terraform

You can find the complete terraform configuration code for the infrastructure we will be building today here.

To begin I will create a main.tf file and a provider.tf file for my root module. The first order of business is to configure the AWS provider for Terraform and initialize to get it ready to deploy AWS resources.

A provider is used by Terraform to interface with the API of whatever infrastructure you're trying to build. Since we are trying to build AWS infrastructure we will be using the AWS infrastructure for our configuration. If you were building in GCP or Azure you will be using a provider for those cloud services.

In the provider.tf file I will add the terraform block as well as the provider block, the terraform code block will allow terraform use the the AWS API build our infrastructure while the provider block will configure the AWS provider with the necessary credentials.

In the provider.tf file add the following lines of code:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  shared_credentials_files = ["~/.aws/credentials"]
}

The great thing about terraform is that you do not have to have any of these codes memorized as you just start out with terraform, you can always refer to the Terraform documentation for help.

In the terraform block I specified AWS as the required provider with source as hashicorp/aws but I omitted the version argument as I would like terraform to download the latest version whenever it is initialized.

The provider block provides the information needed to access AWS specifically. In the provider block I specified the region argument, that is the only credential I will be hardcoding in my configuration. As I have already setup my AWS credentials using AWS configure with the AWS CLI I added the shared_credentials_file argument (if you have multiple profiles ensure you include the profile argument and supply the profile name) so Terraform will use that information to pick up these credentials and make use of them to build our infrastructure.

For a guide on how to configure your AWS credentials in the AWS CLI, check out this post of mine where I take you through the process.

Now I am ready to run a terraform init to initialize the project so that terraform can download the provider required for this project and connect to AWS.

Ensure you are in your project directory and run the below command in your terminal:

terraform init

Terraform Modules

Terraform modules are a powerful feature that allows you to organize and reuse infrastructure configurations. Modules encapsulate groups of resources that are used together and can be referenced and instantiated multiple times within your Terraform configuration.

Modules are a great way to follow the Don't Repeat Yourself (DRY) principle of software development which states that code should be written once and not repeated. Modules encapsulate a set of Terraform config files created to serve a specific purpose.

Modules are used to create reusable components inside your infrastructure. There are primarily two types of modules depending on how they are written (root and child modules), and depending if they are published or not, we identify two different types as well (local and published).

Reusability is the best consideration while writing Terraform code. Repetition of the same configuration will be laborious because HCL is a declarative language and can be very wordy. Therefore, for optimal reusability, we should attempt to use modules as much as possible so be sure to define them at the beginning, as much as possible.

We will be writing our configuration as modules and then run the modules to build the configuration.

It is recommend to place modules in a modules directory when locally developing modules but you can name it whatever you like.

To begin I created the Modules directory, this is where all my modules will reside.

Create S3 Bucket Module

For my S3 bucket module, I created a directory named s3-bucket in the Modules directory. In this directory I create the following files main.tf, variables.tf, output.tf.

Create S3 Bucket

modules/s3-bucket/variables.tf

In the variables.tf I define the bucket name as a variable with the code below:

variable "bucket-name" {
  description = "The name of the S3 bucket"
  type        = string

  validation {
    condition     = (
      length(var.bucket-name) >= 3 && length(var.bucket-name) <= 63 && 
      can(regex("^[a-z0-9][a-z0-9-.]*[a-z0-9]$", var.bucket-name))
    )
    error_message = "The bucket name must be between 3 and 63 characters, start and end with a lowercase letter or number, and can contain only lowercase letters, numbers, hyphens, and dots."
  }
}

The validation simply checks that the bucket name is between 3 and 63 characters, start and end with a lowercase letter or number, and contains only lowercase letters, numbers, hyphens, and dots. This is necessary to prevent any error that AWS might throw as a result of wrong bucket naming convention.

The significance of using a variable file is for simplicity and easy refactoring of the code. In the event that we need to change the value of that variable, we will only need to change it in one place and the change will be picked up anywhere that variable is made reference to using the .var notation.

modules/s3-bucket/main.tf

Now to the creation of the S3 bucket, we will add the aws_s3_bucket resource block to the module's main.tf file as shown below.

# Create S3 Bucket
resource "aws_s3_bucket" "site-bucket" {
  bucket = var.bucket-name
  force_destroy = true
}

The bucket name is supplied by a variable and will be substituted with the value at creation.

modules/s3-bucket/outputs.tf

This is the file in which we will output some of the values we will use in the rest of our configurations.

Go ahead and create a new file called outputs.tf in the s3 bucket module add the below code to the file:

output "bucket_regional_domain_name" {
  description = "This is the bucket domain name including the region name."
  value = aws_s3_bucket.site-bucket.bucket_regional_domain_name
}

Add the S3 Bucket to the Root Module

main.tf

To test that this module works we will create an s3 bucket using this module we have just written. Head to the main.tf file of your root module, outside your Module directory and enter the below piece of code in the file.

module "s3-bucket" {
  source = "./Modules/s3-bucket"
  bucket-name = var.bucket-name
}

variables.tf

Create three new files also in your root module called variables.tf, outputs.tf and terraform.tfvars.

In the variables.tf add the code below

variable "bucket-name" {
  type = string
}

outputs.tf

In the outputs.tf add the following code:

output "bucket-name" {
  value = module.s3-bucket.site-bucket.bucket_regional_domain_name
}

We will make use of this output when creating our cloudfront distribution.

terraform.tfvars

In the terraform.tfvars file, enter the code below:

bucket-name = "<your unique bucket name>

⚠️ NOTE

Your .tfvars file should never be committed to version control, add this file to your .gitignore file. Check out my .gitignore file for files to add to yours.

You can also use this site to generate your gitignore files for this project and future projects.

In your terminal, run the terraform init command again, you must rerun the command when you add a module or change provider. If you fail to run it and run any other terraform command you will get the below error message.

Now you can run terraform plan to see what terraform plans to create in your AWS account.

To create the bucket, run

terraform apply

Whenever you run this command terraform will always ask you if you want to carry out this action, you can either answer yes or no. To avoid this question coming up you can directly include auto approve in the command as shown below:

terraform apply --auto-approve

If you followed along correctly, you would have successfully created an s3 bucket, we will destroy it and continue writing our terraform script as we are not just merely creating a bucket.

Run the command below to destroy the created bucket:

terraform destroy

TF alias for Terraform

Before we continue, I want to set a short alias for terraform as we will need to call terraform a whole lot. Setting terraform alias to be tf will help simplify things for us when we are calling our commands, so we will no longer need to explicitly call out terraform but now we call it tf eg tf apply instead of terraform apply. It helps us shorten our command.

We can do this by setting up an alias in the bash profile.

To open the bash profile for the terminal, I used the below command:

vi ~/.bash_profile

This is where we can set bash configurations, we set our alias by calling alias with the short form as seen:

alias tf="terraform"

To now use the command we set as alias we need to run that bash profile script first so that the change is applied.

source ~/.bash_profile

Now we can use tf instead of terraform

Upload Assets Into S3 Bucket

Before writing the code to upload our website assets into the bucket we should create a directory and save our assets. I will save this in our root module as web-assets and add my website assets in there.

modules/s3-bucket/main.tf

We will use the for_each meta arguments to upload our bucket assets, we are using this approach as we have multiple files to upload. this is useful when you create multiple resources with similar configurations.

It does not make sense to just copy and paste the Terraform resource blocks with minor tweaks in each block. Doing this only affects the readability and unnecessarily lengthens the IaC configuration files. Add the below code to your s3-bucket module main.tf file:

# Upload objects into the s3 Bucket
resource "aws_s3_object" "upload-assets" {
  for_each = fileset("${var.web-assets-path}", "**/*")
  bucket = aws_s3_bucket.site-bucket.bucket
  key = each.value
  source = "${var.web-assets-path}/${each.value}"
  content_type = lookup(var.mime_types, regex("\\.[^.]+$", each.value), "application/octet-stream")
}

The for-each will iterate through the files in the website directory. I used the fileset function to iterates over all files and directories in the specified path, making each file/directory available to the for_each loop in the resource definition.

The path isn't hardcoded, it is defined as a variable in the variable.tf file as you will see below.

The for_each loop over fileset returns file paths, not key-value pairs, this is why we use each.value as our key and not each.key.

We want the website to recognise each file type for it's correct respective MIME type and display it properly on the website which is why we used the lookup function in the content_type argument. lookup(map, key, default) is a function that searches for key in map and returns the associated value if found. If key is not found, it returns default.

The regex function extracts the file extension from each.value, which is the file name obtained from fileset in other to determine a more accurate MIME type.

modules/s3-bucket/variables.tf

Here we will define the variables we called in the piece of code above, add the below code to the file:

# Set the variable for the file path of the files to be uploaded to the bucket
variable "web-assets-path" {
  description = "This is the location of our website files"
  type = string
}

variable "mime_types" {
  description = "Map of file extensions to MIME types"
  type        = map(string)
  default = {
    ".html" = "text/html"
    ".css"  = "text/css"
    ".png"  = "image/png"
    ".jpg"  = "image/jpeg"
    ".jpeg" = "image/jpeg"
    ".pdf"  = "application/pdf"
    "json" = "application/json"
    "js"   = "application/javascript"
    "gif"  = "image/gif"
    # Add more extensions and MIME types as needed
  }
}

Update Root Module `main.tf`, `variable.tf` and `terraform.tfvars` Files

main.tf

we updated our module and so we need to update our root module configuration as well. Your root module's main.tf file should now look like this:

module "s3-bucket" {
  source = "./Modules/s3-bucket"
  bucket-name = var.bucket-name
  web-assets-path = var.web-assets-path
}

variable.tf

Your root module's variable.tf file should now look like this:

variable "bucket-name" {
  type = string
}

variable "web-assets-path" {
  type = string
}

terraform.tfvars

Your root module's terraform.tfvars file should now look like this:

bucket-name = "<your unique bucket name>
web-assets-path = "<the path to your website files (best to supply the absolute path)>

Create Hosted Zone in Route53

⚠️ Note

Even if you are still eligible for the AWS free tier, the Route53 service is never free. This hosted zone will attract a charge of $0.50 per month.

You need a custom domain name for this step, so if you don't already have one, pause, get one and continue along.

This step is going to be completed manually, initially I was going to import the resource into terraform after manually creating it but upon further consideration we have no reason to as I wouldn't want Terraform deleting the hosted zone.

The reason for creating the hosted zone manually is simply because when you create a hosted zone, you are given a new set of name servers which you will need to add to your custom domain configuration. Terraform does not have the infrastructure to complete this step and so your configuration will fail until you manually add your name servers to your custom domain yourself.

Since we already know this, we will manually create the hosted zone, add the name servers to our custom domain and then, using the terraform data resource, retrieve details of the created hosted zone into terraform to avoid any issues that might have arisen.

Create Hosted Zone

Open your AWS management console.
In Services under the Network and Content delivery category choose Route53
Select create hosted zone
It's pretty straightforward from there, enter your domain name in the space for domain name.
Select public hosted zone under type.
You can add a tag and description if you want.
At the bottom of the page, click on create hosted zone.

Once your hosted zone has been created, open it to view details and copy the name servers supplied by AWS.
Copy each name server and replace those already in our domain name with these new ones.

Retrieve the Details of the Hosted Zone Resource into Terraform Configuration

To add details of our hosted zone resource to terraform we will create a new module in the Module directory called route53.

Modules/route53/main.tf

Add the following code to the file:

# Retrieve information about your hosted zone from AWS
data "aws_route53_zone" "created" {
  name = var.domain_name
}

The above code might not look like much but it will retrieve the details of the hosted zone in our AWS account that matches the name we supply and then using it wherever we call that specific data resource in our configuration.

Modules/route53/variables.tf

You know the drill, add the declared variables to keep your code reusable.

# domain name variable
variable "domain_name" {
  description = "This is the name of the hosted zone."
  type = string
}

Create TLS/SSL Certificate and Validate it

This is not a one stage process in terraform, we will need to first create the resource and then validate it with another resource block.

Create Certificate

We will create our certificate before our cloudfront distribution as we will use our SSL certificate in our cloudfront distribution.

As usual, create a certificate directory in the Module directory which will house our certificate module. Create 3 new files in that directory main.tf, variable.tf and output.tf.

Modules/certificate/main.tf

# Create the TLS/SSL certificate
resource "aws_acm_certificate" "cert" {
  domain_name               = var.domain_name
  validation_method         = var.validation_method
  subject_alternative_names = var.subject_alternative_names

  # Ensure that the resource is rebuilt before destruction when running an update
  lifecycle {
    create_before_destroy = true
  }
}

Modules/certificate/variables.tf

Add the necessary variables to your variables file:

variable "domain_name" {
  description = "Domain name for which the certificate should be issued"
  type = string
}

variable "validation_method" {
  description = "Which method to use for validation."
  type = string
  default = "DNS"
}

variable "subject_alternative_names" {
  description = "Set of domains that should be SANs in the issued certificate."
  type = list(string)
  default = []
}

Modules/certificate/outputs.tf

Define the outputs we will need to reference in other modules

output "cert-arn" {
  value = aws_acm_certificate.cert.arn
}

output "domain_validation_options" {
  value = aws_acm_certificate.cert.domain_validation_options
}

Create the ACM Certificate Validation Record

Before we create the resource to validate the certificate, we need to create a DNS record in AWS Route 53, which is used to validate the domain ownership for an AWS ACM certificate. The DNS record details (name, value, type) are obtained from the ACM certificate's domain validation options.

We will create this record in route53 so head on to your Modules/route53/main.tf file. Add the following to your file:

Modules/route53/main.tf

# Create DNS record that will be used for our certificate validation
resource "aws_route53_record" "cert_validation" {
  for_each   = { for dvo in var.domain_validation_options : dvo.domain_name => {
    name     = dvo.resource_record_name
    type     = dvo.resource_record_type
    record   = dvo.resource_record_value
  } }

  name       = each.value.name
  type       = each.value.type
  records    = [each.value.record]
  ttl        = 60
  zone_id  = data.aws_route53_zone.created.zone_id
}

The code above will create a CNAME record in your domain's hosted zone which will be used to validate the certificate which you created. However if you try to apply the code to create te certificate and create the record at the same time, you will get an error message that looks like the own below

This is why we will run the terraform apply command in two stages as you will see eventually.

Modules/route53/variables.tf

Add the following to your route53 module variables file

variable "domain_validation_options" {
  description             = "The domain validation options from the ACM certificate."
  type                    = list(object({
    domain_name           = string
    resource_record_name  = string
    resource_record_type  = string
    resource_record_value = string
  }))
}

Validate the Certificate

The aws_acm_certificate resource does not handle the certificate validation in terraform, we need to use the aws_acm_certificate_validation resource to accomplish that.

As I earlier explained and you saw from the error message, the certificate first needs exist and the value of the domain_validation_options known first before terraform will honour our for_each statement, this is why we need to first create the certificate, then the record and then verify.

For the above reason we won't put the validation step in the certificate module but in the Route53 module, therefore open your Route53 module.

The code for the actual validation is seen below:

Modules/route53/main.tf

# Validate the certificate
resource "aws_acm_certificate_validation" "validate-cert" {
  certificate_arn = var.certificate_arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.fqdn]

  depends_on = [aws_route53_record.cert_validation]
}

The depends_on argument will force terraform to create the aws_route53_record.cert_validation resource first before attempting to validate our certificate.

Modules/route53/variables.tf

Add the required variables to the module's variables.tf file

variable "certificate_arn" {
  type = string
}

Modules/route53/outputs.tf

Add the following to your route53 module outputs.tf file

output "dns_records" {
  value = aws_route53_record.cert_validation
}

We still have to create the CloudFront Distribution Alias Record, this is the record where we will set our cloudfront distribution domain name as an alias for our custom domain name. We will do this after creating our cloudfront distribution.

Create CloudFront Module

Now we can go ahead and create our cloudfront distribution. Create a cloudfront directory in the Modules directory and add create main.tf and variables.tf files in the directory.

Create Origin Access Control - OAC

The first thing we need to do is to create the Origin Access Control we will use in the configuration of our distribution. Do this by adding the code below to your main.tf file:

Modules/cloudfront/main.tf

# Create the access origin control that will be used in creating our cloudfront distribution with s3 origin
resource "aws_cloudfront_origin_access_control" "assign-oac" {
  name                              = var.oac-name
  description                       = "An origin access control with s3 origin domain for cloudfront"
  origin_access_control_origin_type = var.origin_access_control_origin_type
  signing_behavior                  = var.signing_behavior
  signing_protocol                  = var.signing_protocol
}

Modules/cloudfront/variables.tf

Declare the variables:

variable "oac-name" {
  description = "This is the name of the cloudfront origin Access control with s3 bucket origin domain"
  type = string
  default = "s3-bucket-oac"
}

variable "origin_access_control_origin_type" {
  description = "The origin type must be the same as the origin domain"
  type = string
  default = "s3"
}

variable "signing_behavior" {
  description = "Specifies which requests CloudFront signs."
  type = string
  default = "always"
}

variable "signing_protocol" {
  description = "Determines how CloudFront signs (authenticates) requests."
  type = string
  default = "sigv4" # The only valid value
}

Create Distribution

Now we can create our distribution. Add the following in the main.tf file:

Modules/cloudfront/main.tf

# Create CloudFront Distribution
resource "aws_cloudfront_distribution" "cdn" {
  origin {
    domain_name              = var.cdn-domain_name-and-origin_id
    origin_id                = var.cdn-domain_name-and-origin_id
    origin_access_control_id = aws_cloudfront_origin_access_control.assign-oac.id
  }

  default_cache_behavior {
    compress = true
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = [ "GET", "HEAD" ]
    cached_methods         = [ "GET", "HEAD" ]
    target_origin_id       = var.cdn-domain_name-and-origin_id
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400

    forwarded_values {
      query_string = false
      cookies {
        forward    = "all"
      }
    }
  }

  restrictions {
    geo_restriction {
      restriction_type = var.restriction_type
    }
  }

  viewer_certificate {
    acm_certificate_arn            = var.acm_certificate_arn
    ssl_support_method             = "sni-only"
    minimum_protocol_version       = "TLSv1.2_2021"
    cloudfront_default_certificate = false
  }

  enabled             = true
  is_ipv6_enabled     = true
  default_root_object = var.default_root_object
  aliases             = [var.domain_name, "www.${var.domain_name}"] 
}

Modules/cloudfront/variables.tf

Add the required variables

variable "restriction_type" {
  description = "Method that you want to use to restrict distribution of your content by country"
  type = string
  default = "none"
}

variable "default_root_object" {
  description = "Object that you want CloudFront to return when an end user requests the root URL."
  type = string
  default = "index.html"
}

variable "domain_name" {
  description = "your custom Domain name for which the certificate should be issued"
  type = string
}

variable "cdn-domain_name-and-origin_id" {
  type = string
}

variable "acm_certificate_arn" {
  type = string
}

Modules/cloudfront/outputs.tf

output "cloudfront-arn" {
  value = aws_cloudfront_distribution.cdn.arn
}

output "cloudfront_domain_name" {
  value = aws_cloudfront_distribution.cdn.domain_name
}

output "cloudfront_hosted-zone_id" {
  value = aws_cloudfront_distribution.cdn.hosted_zone_id
}

Configure S3 Bucket Permission

Now we need to add the specific bucket permissions, that cloudfront needs to be able to adequately interact with our s3 bucket, to our s3 bucket.

Head back to your s3 bucket module.

Modules/s3-bucket/main.tf

This is the policy that will allow our cloudfront distribution access to our s3 bucket and it's object through it's access origin control.

Add the code below to our s3 bucket module's main.tf file

Modules/s3-bucket/main.tf

# Add the permissions needed by cloudfront's origin access control to access the bucket and it's objects
resource "aws_s3_bucket_policy" "cloudfront-oac-policy" {
  bucket = aws_s3_bucket.site-bucket.bucket
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid = "AllowCloudFrontServicePrincipal",
        Effect = "Allow",
        Principal = {
          Service = "cloudfront.amazonaws.com"
        },
        Action = "s3:GetObject",
        Resource = "${aws_s3_bucket.site-bucket.arn}/*",
        Condition = {
          StringLike = {
            "aws:UserAgent" = "Amazon CloudFront"
          }
        }
      }
    ]
  })
}

Create CloudFront Distribution Alias Record

We will create a new module, specially for this, create a new module called alias, create two files main.tf and variables.tf.

Modules/alias/main.tf

# Retrieve information about your hosted zone from AWS
data "aws_route53_zone" "created" {
  name = var.domain_name
}

# Create an alias that will point to the cloudfront distribution domain name
resource "aws_route53_record" "alias" {
  zone_id                  = data.aws_route53_zone.created.zone_id
  name                     = var.domain_name
  type                     = "A"

  alias {
    name                   = var.cloudfront_domain_name
    zone_id                = var.cloudfront-zone-id
    evaluate_target_health = false
  }
}

Modules/alias/variables.tf

Declare the necessary variables as usual:

variable "domain_name" {
  description = "your custom domain name"
  type = string
}

variable "cloudfront_domain_name" {
  type = string
}

variable "cloudfront-zone-id" {
  type = string
}

Putting it all Together: Move Modules into the Root Module

It's now time to put our modules to use in building our infrastructure. We do this by calling the module in the main.tf file of our root module, this is our main configuration file.

We had previously added our s3-bucket module to our main.tf earlier in the project when we wanted to test out our s3-bucket module, now we will add the rest of our modules to the main.tf.

main.tf

Your final configuration in your main.tf of your root module should look like this:

# Create S3 bucket, upload objects into the bucket and set bucket policy.
module "s3-bucket" {
  source = "./Modules/s3-bucket"
  bucket-name = var.bucket-name
  web-assets-path = var.web-assets-path
}

# Create and validate TLS/SSL certificate
module "certificate" {
  source = "./Modules/certificate"
  domain_name = var.domain_name
  subject_alternative_names  = ["www.${var.domain_name}"]
  }

# Create OAC and cloudfront distribution, 
module "cloudfront" {
  source = "./Modules/cloudfront"
  domain_name = var.domain_name
  cdn-domain_name-and-origin_id = module.s3-bucket.bucket_regional_domain_name
  acm_certificate_arn = module.certificate.cert-arn
  depends_on = [ module.route53 ]
}

# Import the hosted zone from AWS, create dns records for certificate validation, and create A and CNAME records.
module "route53" {
  source = "./Modules/route53"
  domain_name = var.domain_name
  domain_validation_options = module.certificate.domain_validation_options
  certificate_arn = module.certificate.cert-arn
  }

# Create an alias to point the cloudfront cdn to our domain name.
module "alias" {
  source = "./Modules/alias"
  domain_name = var.domain_name
  cloudfront_domain_name = module.cloudfront.cloudfront_domain_name
  cloudfront-zone-id = module.cloudfront.cloudfront_hosted-zone_id
  depends_on = [ module.cloudfront ]
}

variables.tf

Now we will declare the necessary variables, your final variable.tf file should look like this:

variable "bucket-name" {
  type = string
}

variable "web-assets-path" {
  type = string
}

variable "domain_name" {
  type = string
}

terraform.tfvars

Add your secrets to your *.tfvars file like so:

bucket-name = "<your unique bucket name>
web-assets-path = "<the path to your website files (best to supply the absolute path)>
domain_name = "<your custom domain name>"

Now we are all set to deploy our application.

Create the Infrastructure

Install Modules

First run tf init to install all the added modules.

tf init

Validate Configuration

Next you can run the validate command to validate your configuration

tf validate

Create Infrastructure

As I already explained earlier the for_each function will only iterate on values that are already known at the time the apply command is , therefore if we were to apply before creating our certificate terraform will thrown an error.

To avoid this error we will apply in two stages, first with the --target flag and then apply the whole configuration.

First run:

tf apply --target module.certificate

Lastly, create the remaining resources:

tf apply

Confirm Build

You can open your AWS console to see that the resources have been built.

Open your browser and navigate to your custom domain and you will see that your website is showing, here is mine.

Cleanup

Remember to clean up your environment when you are done. Don't leave the resources running in AWS to avoid unnecessary billing.

Use the destroy command:

tf destroy

If this post has helped you, consider supporting me:

Host a Static Website using Amazon S3 and Serve it Through Amazon CloudFront.

ChigozieCO — Mon, 27 May 2024 15:03:17 +0000

AWS S3 buckets can be used to host static web pages but you would need to make it publicly accessible. This opens your bucket and assets to vulnerabilities, how then can you serve your site to the public while minimizing the risks of being attacked by bad actors?

We can do this using AWS Cloudfront. You do this by serving your site via cloudfront. In this walkthrough, we will do just that. At no point will we make our bucket public accessible.

Prereq

You need to have an AWS account, the services used here are covered by the free tier and so you should not incur any costs.
If you are just creating your account ensure you create an IAM user that has administrator privilege (it's not recommended to use the root user for regular day to day activities) as well as an active access key.
You need to have AWS CLI 2 installed and configured for programmatic access to your AWS account.
You also need to have an HTML page (the static website you want to host), don't fret you don't have to start designing one if you don't have one handy you can get html templates here. I will be using a template gotten from there as well.

Amazon S3 Bucket

Amazon S3 (Simple Storage Service) is an object storage built to store and retrieve any amount of data from anywhere. It integrates with many other AWS services, is highly available and durable and also highly cost-effective. Amazon S3 also provides easy management features to organize data for websites, mobile applications, backup and restore, and many other applications.

You can configure a storage bucket to host and serve as a static website, on a static website, individual webpages include static content. They might also contain client-side scripts.

It however cannot be used to host a dynamic website as a dynamic website relies on server-side processing, including server-side scripts, such as PHP, JSP, or ASP.NET. Amazon S3 does not support server-side scripting, but AWS has other resources for hosting dynamic websites.

Configure AWS CLI

I will be using the AWS CLI to programmatically create my bucket and upload the resources into the bucket as well as to enable static web hosting.

⚡ First you need to download the aws cli 2. From the provided link follow the instructions for your device and follow along with the rest of the project.

⚠️ NOTE

If you want to interact with AWS outside of the AWS Management Console and use the cli as I will be in this walkthrough you need to have programmatic access. You need to create an access key for the IAM user you will be using.

⚡ Simply navigate to the IAM user and scroll down to the access key and click on create key

For the use case select the first option being Command Line Interface (CLI) and click on next

You can choose to add a tag or not (tags will make it easier for you to identify your access key) and then go ahead to create the key.

⚡ The next thing you need to do is to setup the cli for use. Use the aws configure command and enter your access key

$ aws configure
AWS Access Key ID [****************W0FM]: <your access key ID>
AWS Secret Access Key [****************okxa]: <your secret access key>
Default region name [us-east-1]: <whatever region you want to be your default>
Default output format [None]: <you can leave empty by clicking enter>

You can run the aws sts get-caller-identity command to verify that you have configured the CLI correctly. The output of this command will display the present user ID, account and ARN.

Verify that this is the correct credentials and continue.

Now you are ready to use the AWS CLI for this walk through.

Create S3 Bucket

We will be creating a bucket with the default settings. We want our bucket to be private and to block all public access.

Because we are serving the site through the content distribution network cloudfront (more on that later) we can afford to block all public access to our bucket as it won't be required.

To create a bucket via the cli, use the below command:

aws s3api create-bucket --bucket <your unique bucket name> --region <your preferred aws region>

⚠️ NOTE

Your bucket name must be globally unique, no two buckets in the whole world can ever have the same exact name, there must be a variation.

Also if you set your default AWS region and you're ok with your bucket residing in that region you do not need to specify the region with the --region flag. Remember that your bucket is region specific so it will only exist in the region you created it.

I won't be using the region tag as I'm ok using my set default region.

aws s3api create-bucket --bucket altschool-sem3-site

If it was successful you should see something that resembles the below:

You can list your available buckets using aws s3 ls command.

Upload Objects into S3

Now we need to upload our static website assets.

A static webpage is one that is sent to a web browser in its identical stored format. It is sometimes referred to as a flat page or a stationary page.

Until the site is redesigned or the site administrator makes changes directly in the code, the page will not be altered by anything either the user or the site administrator does on the page.

Our static website assets are our html, css etc, those files you downloaded from the html templates here.

Time to upload our objects in the S3 bucket, we do that using the code below:

aws s3 cp /local/file/path s3://bucket-name --recursive

aws s3 cp ./2093_flight/ s3://altschool-sem3-site --recursive

The recursive flag will help us upload multiple files at the same time, it will ensure that all the files and subdirectories within the specified directory are uploaded.

To verify that you have successfully uploaded the objects we will list the objects available in the bucket.

Use the command below to do that:

aws s3 ls s3://bucket-name

aws s3 ls s3://altschool-sem3-site

You should see a list of the objects you just uploaded into your bucket. From the screenshot above you can see my uploaded objects.

Amazon CloudFront

Amazon CloudFront is a content delivery network operated by Amazon Web Services. It is a service that helps you distribute your static and dynamic content quickly and reliably with high speed by using a network of proxy servers to cache content, such as web videos or other bulky media, more locally to consumers, to improve access speed for downloading the content.

We will use the management console for this, log in with your IAM user and navigate to cloudfront under the Networking & Content Delivery category.

When you load the cloudfront page click on create distribution

CloudFront configurations

- Origin

⚡ Origin Domain

Origins are where you store the original versions of your web content. CloudFront gets your web content from your origins and serves it to viewers via a worldwide network of edge servers.

The origin domain is the DNS domain name of the Amazon S3 bucket or HTTP server from which you want CloudFront to get objects for this origin.

To that effect select the S3 Bucket we created earlier, from the dropdown, as your origin domain

⚡ Origin path

Leave this blank, as is.

⚡ Name

This will automatically be filled in when you enter your origin domain

⚡ Origin Access

In other to further restrict the access to our Amazon S3 bucket origin to only specific CloudFront distributions we will set our origin access to the AWS recommended setting which is Origin access control settings

Once that is selected, we will create a new Origin access control settings.

Select the Origin access control settings radio button and then click on Create new OAC

The pop up that appears when you click on create new OAC will be populated with the necessary details, leave it as is and click on Create

As soon as this new AOC is created, you will see a warning (same as shown below) telling you that you will need to update your bucket policy with the policy that will be provided after the distribution is created.

- All Other Config

For the Web Application Firewall, select do not enable security protections, You could actually choose to enable it if you want I am just trying to be safe and not incur any unexpected costs.

The last setting you will be changing is the default root object, type in index.html. This will allow CloudFront to serve your index page to your site visitors.

Leave all other configurations in their default settings, those settings work just fine for what we are trying to do.

Your configuration should look like the screenshots below:

Scroll down to the bottom of the page and click create distribution

Copy the provided policy, you can see it highlighted in yellow on your screen, see screenshot above for the item labelled 1 to know where to locate it.

After copying the policy click on the link to take you to the bucket in order to update the permissions with the policy you just copied.

Update Bucket Policy With Necessary Permission

You could either manually navigate to the S3 bucket we have been using for this project or you could click on the link in the above picture labelled 2 to take you directly to the bucket.

Once the bucket is open, click on the permissions tab.

Scroll down to Bucket Policy and click on edit to add the permissions we copied at the end of the cloudfront distribution creation.

The copied policy is shown below, ensure you use the one you copied from your cloudfront console as it will carry your unique ARN.

{
        "Version": "2008-10-17",
        "Id": "PolicyForCloudFrontPrivateContent",
        "Statement": [
            {
                "Sid": "AllowCloudFrontServicePrincipal",
                "Effect": "Allow",
                "Principal": {
                    "Service": "cloudfront.amazonaws.com"
                },
                "Action": "s3:GetObject",
                "Resource": "arn:aws:s3:::altschool-sem3-site/*",
                "Condition": {
                    "StringEquals": {
                      "AWS:SourceArn": "arn:aws:cloudfront::<redacted>:distribution/EOE3G4O3YYZSA"
                    }
                }
            }
        ]
      }

Now save your changes.

Your bucket policy should now look like this:

Access Your Site

Head back to your cloudfront console and retrieve your distribution domain name as shown in the image below.

This is the address you will enter in your browser to be able to see your website.

We can see from the screenshot below that cloudfront is serving the webpage correctly without have to unblock public access of our bucket.

And that's it!! We have successfully served a static webpage using Amazon S3 and Amazon CloudFront.

Automate the Deployment of a LAMP Stack & Laravel App with a Script and Ansible

ChigozieCO — Thu, 16 May 2024 17:53:07 +0000

Let's dive into one of the most recent projects I did for a cloud engineering endeavour. We were tasked to:

Automate the provisioning of two Ubuntu-based servers, named “Master” and “Slave”, using Vagrant.
On the Master node, create a bash script to automate the deployment of a LAMP (Linux, Apache, MySQL, PHP) stack.
This script should clone a PHP application from GitHub, install all necessary packages, and configure Apache web server and MySQL.
Ensure the bash script is reusable and readable.
Using an Ansible playbook:
- Execute the bash script on the Slave node and verify that the PHP application is accessible through the VM’s IP address (take screenshot of this as evidence)
- Create a cron job to check the server’s uptime every 12 am.

Provision VMs

To begin this project I need to provision two Ubuntu servers named "Master" and "Slave" using vagrant.

I will be provisioning ubuntu 22.04 LTS for the Master and the slave and I will use a multi-machine environment, this basically means that I will define two VMs in one vagrantfile.

The first this I do is initialize the box, the box I'm using is ubuntu/jammy64 and because I am using a multi-machine environment I will pass the -m (minimal) flag so that the VagrantFile is created without all the comments.

vagrant init -m ubuntu/jammy64

When I open the VagrantFile to edit it, we can see that it has just the basic configuration without all the comments.

vi VagrantFile

To configure my VMs, I will enter the below code into my VagrantFile.

Vagrant.configure("2") do |config|

  config.vm.define "master" do |master|
    master.vm.box = "ubuntu/jammy64"
    master.vm.hostname = "master"

    master.vm.network "private_network", type: "dhcp"
    master.vm.network :forwarded_port, guest: 22, host: 2030, id: "ssh"

    master.vm.provider "virtualbox" do |v|
      v.name = "Master"
    end
  end

  config.vm.define "slave" do |slave|
    slave.vm.box = "ubuntu/jammy64"
    slave.vm.hostname = "slave"

    slave.vm.network "private_network", type: "dhcp"
    slave.vm.network :forwarded_port, guest: 22, host: 2032, id: "ssh"

    slave.vm.provider "virtualbox" do |v|
      v.name = "Slave"
    end
  end

end

In the VagrantFile I simply specified the type of base box with which each VM would be created, I ensured they used a private network and they get their IP address from the DHCP server. I also specified the name and hostname of each VM.

Another thing I did was to hard code the host SSH port so that I deal with the SSH clash that will happen, from the jump, another reason I did this is so that the port doesn't conflict with other VMs I have on my laptop.

To provision the VMs I ran the command

vagrant up

From the images below you can see that both my master and slave machines were successfully provisioned.

To access it and start working with it I SSH into them by specifying their names, as shown below

vagrant ssh master
vagrant ssh slave

Bash Script to Deploy LAMP Stack

A LAMP stack is an acronym for the operating system, Linux; the web server, Apache; the database server, MySQL; and the programming language, PHP.

This is a very common stack developers use to build websites and web applications.

The task asked that we write a script that would be used to automate the installation of the LAMP stack. With this script the LAMP stack deployment as well as the Laravel application deployment will be fully automated, this script will be written in a way as not to require any user input at all. I will try to explain my logic along the way.

Find the full script here, to create your script create a new file with the vi editor:

vi deploylamp.sh

Install Apache

To begin, I want this script to stop running whenever it encounters an error at any point, as the successful deployment of the Laravel application is dependent on all the components of this script being present in the server. To accomplish this we will use the set -e command, start you script that way.

#!/bin/bash

# This command will make the script immediately close if any command exits with a non-zero status 
set -e

Next we will update the apt repository so that our packages are up to date.

# Update apt repository
sudo apt update

We finally get to the beginning of the main event, now we will add the command that installs Apache to our script, as earlier mentioned, apache is a very popular webserver used by developers.

To install apache add the command below to your script

# Install Apache and handle any errors if any
echo "Installing Apache ===================================================="
echo

sudo apt install -y apache2 || { echo "Error installing Apache"; exit 1; }
echo "Successfully installed apache ======================================="
echo

The echo commands are there to inform us every step of the way what the script is up to.

The command sudo apt install -y apache2 || { echo "Error installing Apache"; exit 1; } will either install Apache or print an error message depending on whether the installation is successful or not.

Install MySQL

MySQL is a fast, multi-threaded, multi-user, and robust SQL database server. It is intended for mission-critical, heavy-load production systems and mass-deployed software.

Add the below to your bash script.

# Install MySQL and handle any errors if any
echo "Installing MySQL ===================================================="
echo

sudo apt install -y mysql-server || { echo "Error installing MySQL"; exit 1; }
echo "Successfully installed MySQL ======================================="
echo
echo

Once the installation is complete, the MySQL server usually starts automatically.

Configure MySQL

The default configuration of MySQL is not secure, the root has no password and remote access is possible for the root user, it also comes with a test database and an anonymous user, these are the configs that we will be changing.

Ideally we would use the mysql_secure_installation script to secure the database but that would require user input and I'm trying to keep this LAMP deployment process as unattended as possible and so I will be using a here document that would provide all the necessary responses to the prompts that the script usually presents.

💡 NOTE

Another thing to note is that for security purposes I won't be hardcoding my root password in the script. Since this script will be run with Ansible I will save the password in Ansible vault. When we get to the Ansible configuration part of this project you will see how we will walk through the process of adding the password to Ansible vault.

Ansible provides a built-in solution called Ansible Vault for encrypting sensitive data. You can create an encrypted file to store the MySQL root password, and then Ansible will decrypt it when needed during playbook execution.

The way this will work is that ansible will decrypt the password, and pass it as an env variable as MYSQL_ROOT_PASSWORD (as we instruct it to do in the ansible playbook) which the bash script will then read and use while running the script.

This way everything is safe and secure.

Add the below to your script:

# Configure MySQL Server automatically, we won't hardcode the root password, it will be read from ansible vault
echo "Now configuring MySQL Server ======================================="

# Main MySQL configuration
sudo mysql <<EOF
CREATE DATABASE IF NOT EXISTS laravel;
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '$MYSQL_ROOT_PASSWORD';
DELETE FROM mysql.user WHERE User='';
DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1');
DROP DATABASE IF EXISTS test;
DELETE FROM mysql.db WHERE Db='test' OR Db='test\_%';
FLUSH PRIVILEGES;
EOF

The SQL commands in the above part of the script will essentially do what mysql_secure_installation does:

Creates a laravel database that the laravel app would use.
Set a new password for the root user.
Remove anonymous users.
Disallow remote root login.
Remove the test database and access to it.
Flush privileges to apply the changes.

Lastly we will check if the configuration succeeded or failed and print a message based on the exit code. Add the following to the script:

# Check if the configuration was successful or if there was an error and let us know which it is
if [ $? -ne 0 ]; then
    echo "Error: Failed to configure MySQL ====-============================================="
    echo
    exit 1
else
    echo "Successfully configured MySQl ==================================================="
    echo
fi

Install PHP

PHP is a general-purpose scripting language, well-suited for Web development since PHP scripts can be embedded into HTML.

Disable any PHP Module you Might Have Previously Installed

Before installing PHP I will disable any php module I had previously installed, I found this step necessary because while testing my script, after successfully deploying the Laravel application the app didn't come up I kept getting the error shown below.

Upon further investigation after running apachectl -M | grep php I found out that apache was using a lower version of php (7.4) not the latest 8.2 I had just installed.

$ apachectl -M | grep php
 php7_module (shared)

To save you that stress, first disable any old one that might exist on your server before installing the latest version using the if statement below. Add to your script

# Check if any PHP module is enabled
if apachectl -M 2>/dev/null | grep -q 'php'; then
    # Disable all PHP modules
    sudo a2dismod php*
    echo "All PHP modules disabled =========================================================="
else
    echo "No PHP modules found =========================================================="
fi

The script uses apachectl -M to list all enabled Apache modules.
If a PHP module is found, the script disables all PHP modules using sudo a2dismod php*, which disables any module starting with "php". This is a precautionary measure in case there are multiple PHP versions installed.
If no PHP modules are found, it prints a message indicating that no PHP modules were found.

Install the Latest Version

Although PHP is available on Ubuntu Linux Apt repository, to be able to get the latest version I will add the OndreJ PPA and install it from there.

I will also be installing some necessary dependencies that MySQL needs to be able to work with PHP.

Add the following to the script:

# Install dependencies first
sudo apt install -y software-properties-common apt-transport-https ca-certificates lsb-release
# Add the OndreJ PPA to allow us install the latest version of PHP
sudo add-apt-repository -y ppa:ondrej/php
sudo apt update
# Install php and necessary modules
sudo apt install -y php8.2 php8.2-mysql php8.2-cli php8.2-gd php8.2-zip php8.2-mbstring php8.2-xmlrpc php8.2-soap php8.2-xml php8.2-curl php8.2-dom || { echo "Error installing php and php modules"; exit 1; }
echo "Successful installed PHP and necessary modules =================================="
echo

Enable URL Rewriting and the new PHP module

# Enable Apache's URL rewriting and restart Apache
sudo a2enmod rewrite
# Incase mpm_event is enabled disable it and enable mpm_forked as that is what php8.2 needs
sudo a2dismod mpm_event #In one of my tests this was enabled by default but php needed the prefork and it caused errors for me.
sudo a2enmod mpm_prefork 
sudo a2enmod php8.2
sudo service apache2 restart
echo "URL rewrite and PHP module enabled and Apache restarted ========================================================================"
echo

Enabling the rewrite module in Apache (a2enmod rewrite) is typically necessary for Laravel applications and many other web applications that use URL rewriting for routing and clean URLs.

Laravel, like many modern PHP frameworks, relies on URL rewriting to route requests through its front controller (index.php). This allows for cleaner and more expressive URLs without the need for file extensions or query parameters.

The newly installed PHP 8.2 module won't be automatically enabled after installation. After installing PHP 8.2, we'll still need to enable the PHP 8.2 module for Apache to use it. You can do this using the a2enmod command as shown above.

Install Git

We will clone the GitHub repo that has the Laravel application and so we need to ensure that we have Git installed in our server. If Git isn't installed we will install it.

I'd use an if statement for this logic:

# Check if git is installed, if it isn't, install it.
if ! command -v git &> /dev/null; then
  echo "installing git ================================================================="
  sudo apt update
  sudo apt install -y git
  echo "Git Installation complete ======================================================="
  echo
fi

Install Composer

Composer is a PHP dependency manager that facilitates the download of PHP libraries in our projects. Composer both works great with and makes it much easier to install Laravel.

I will first download the Composer installer script from the composer site using curl, pipe it directly to php, and execute it.

Then move it into a location on my server's PATH so that it is globally accessible and lastly make it executable.

# Install composer, composer is required to install laravel dependencies
# Use curl to download the Composer installer script and pipe it directly to php for execution.
curl -sS https://getcomposer.org/installer | php
# Move the script to a location in your path so that it is executable globally
sudo mv composer.phar /usr/local/bin/composer
# Make composer executable
sudo chmod +x /usr/local/bin/composer

Clone the Repository and Install Dependencies

We will clone this repo, this is the official Laravel application repo.

Before cloning the repo I want to delete any content that was in that directory, we will be cloning directly into the /var/www/html directory. You could create a new directory for this but I want to keep this whole process as simple as possible.

Now I navigate into the /var/www/html directory and then I'll go ahead and clone the repo, I will do this in Apache document root and then install the dependencies with composer.

# Before cloning the git repository and installing it's dependencies with composer first remove the content in the /var/www/html directory so we can clone directly into it
echo
sudo rm -rf /var/www/html/*

⚡ SIDE NOTE

I was running into a lot of permission errors when root owned the files in the /var/www/html directory and so I had to change the ownership so that my user would be the owner. Wherever you see vagrant replace that with your username (the one you are using for this deployment).

# Add Vagrant user to the www-data group and correct file permissions
sudo usermod -a -G www-data vagrant
# Set the group ownership of the /var/www/html directory to www-data
sudo chown -R vagrant:www-data /var/www/html
echo "Group ownership of /var/www/html changed ========================================================================"
echo

# Grant write permissions to the www-data group for the /var/www/html directory
sudo chmod -R 775 /var/www/html
echo "Group can now write to the /var/www/html directory =============================================================="
echo

# Navigate to Apache Document root first
cd /var/www/html
# Clone the repo directory in this directory (remember the full stop at the end of the command, it is very important)
git clone https://github.com/laravel/laravel.git .
# Install dependencies with composer
composer install

Update ENV File and Generate an Encryption Key

We need to create a .env file after cloning the git repository or starting a new Laravel project. The .env.example file is typically copied, and the contents of the copied .env file are then updated.

The following commands are what we will use to copy the file from .env.example to .env and generate an encryption key. So add the below to your script:

# Update the env file and generate an encryption key
cd /var/www/html
cp .env.example .env
php artisan key:generate

The php artisan key:generate command is typically used in Laravel projects to generate a new application key. This key is used for encryption and hashing within the Laravel application, such as encrypting session data and generating secure hashes.

When you run php artisan key:generate, Laravel generates a new random key and updates the APP_KEY value in the .env file of your Laravel project with this new key.

Change Permissions on Storage and Bootstrap Directory

Honestly this step could have been carried out right before we create our .env file and generate our APP_KEY, immediately after cloning the repo but whatever, it hurts no one.

If your /var/www/html/storage file and your /var/www/html/bootstrap/cache files do not belong to the www-data user you experience the error message below because the www-data user which Apache uses for web related activities won't have the necessary permissions that Laravel needs to fully function.

Add the below to your script to change the ownership of those directories.

# Set permission for the storage and bootstrap directories
sudo chown -R www-data /var/www/html/storage
sudo chown -R www-data /var/www/html/bootstrap/cache
echo "Permission changed for the storage and bootstrap directories ===================================================="

Update ENV

We also need to update the .env file with our database credentials and update the APP_URL value. Presently it points to the localhost as the APP_URL but I need it to point to my server's IP address so that I can access the Laravel application from my IP address and not just localhost (127.0.0.1).

Like you might have already noticed I am trying to make this script as unattended as possible and so I will use sed to search and replace the APP_URL line in the .env file with the actual value I want and then echo the other database credentials I want in the file.

This will allow my updating the .env file be as unattended as possible. Add the following to your script, ensure to replace with your own credentials where necessary.

# Change the APP_URL value to be the IP address of your server
# Get the IP address of the machine
ip_address=$(hostname -I | awk '{print $2}')

# Update the value of APP_URL in the .env file
sed -i "s/^APP_URL=.*/APP_URL=http:\/\/$ip_address/" /var/www/html/.env

hostname -I | awk '{print $2}' retrieves the IP address of the machine.
hostname -I gets a list of IP addresses associated with the hostname, and awk '{print $2}' selects the second IP address from the list. (I prefer to use the second that comes up on my server as that is the unique one for me, you can change 2 to 1 as you see fit).

We also need to add some database configuration parameters, we will use sed to replace what needs replacing and echo the new items into the file.

Add the following to your script

# Add additional database config

# Update the value of DB_CONNECTION to mysql
sed -i "s/^DB_CONNECTION=.*/DB_CONNECTION=mysql/" /var/www/html/.env

# Add additional database configuration parameters
echo -e "\nDB_HOST=127.0.0.1\nDB_PORT=3306\nDB_DATABASE=laravel\nDB_USERNAME=root\nDB_PASSWORD=\"\$MYSQL_ROOT_PASSWORD\"" >> /var/www/html/.env

Adjust the VirtualHost File

Usually you will find your index.html or index.php file directly in the /var/www/html directory but with laravel it is different, the index.php file is located in the public directory and so we have to tell apache to route the traffic into the public directory so it can find our home page there and serve that page by default.

We just have to add /public at the end of the DocumentRoot and at other places in the virtual host file where we have to define the document root of our project.

Add the below to your script.

SERVER_ADMIN="<your email address>"
DOCUMENT_ROOT="/var/www/html/public"

# Update the 000-default.conf file
sudo sed -i -E "s/#?\s*ServerName .*/ServerName $ip_address/" /etc/apache2/sites-available/000-default.conf
sudo sed -i "s/ServerAdmin .*/ServerAdmin $SERVER_ADMIN/" /etc/apache2/sites-available/000-default.conf
sudo sed -i "s|DocumentRoot .*|DocumentRoot $DOCUMENT_ROOT|" /etc/apache2/sites-available/000-default.conf

# Restart Apache to apply changes
sudo systemctl restart apache2

Run Database Migration

The last step is to run your migrations to build your application’s database tables. This step is necessary if you don't want to get the error message below

Ideally when you run the migrate command, as we will next, your database gets created (if it previously didn't exist) along with the tables and necessary schema.

Note however that this command does not have a built-in -y flag to automatically accept the database creation and so will require user input which is why I went back and added the creation of the database in the MySQL installation (you don't have to sweat it).

Add the following to your script:

# Run outstanding migrations
php artisan migrate
echo "Database migrated successfully =================================================================================="
echo
echo "LAMP stack deployment complete, Laravel Repo fully cloned and configured ========================================"

And that's it for the bash script, save the changes and close the file

Make the Script Executable

To be able to run the script we need to make it executable. Use the command below.

chmod +x deploylamp.sh

Install Ansible

The first part of this task was to automate our deployment with a bash script which we have done up until now, the second part is to run the aforementioned script using an Ansible playbook.

To achieve this, we first need to install Ansible on our server, do this by running the following commands:

sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

⚠️ EXPERT TIP

You need to have a common user amongst your servers so that Ansible can function properly. Go ahead and create matching users on all your servers and ensure that the users have sudo privilege and can run sudo commands without password. If you don't know how to, you can checkout my previous Ansible guide here to see how it's done.

I will be using my vagrant user as I have one on all my servers.

You also need to generate an ssh key on your master server (also called control node, this is the machine from which you will run your Ansible commands) and copy the public key to your slave server(s) (the managed node where you want the final result on). You can also find the steps in this post

Create Inventory

To use Ansible, all our slave servers (managed nodes) has to be listed in a file known as the inventory. You can name this file anything you choose and the file will contain the IP address or URL of the managed nodes.

Create a file, name it anything you want, I will name mine examhost as this project is for my exam and enter the IP address of your managed node(s).

You can retrieve the IP address of your server by running the ip a command on the server you want it's IP address.

vi examhost

Save the IP addresses

<machine IP>

Save your file.

Configure Ansible Vault

Ansible provides a built-in solution called Ansible Vault for encrypting sensitive data. We will create an encrypted file to store the MySQL root password as discussed here, and then decrypt it when needed during playbook execution.

Using Ansible Vault allows you to encrypt sensitive data within Ansible playbooks, roles, or other files. First we will create the file with the below command.

When you run the command you will be prompted for a password. After providing a password, the tool will launch whatever editor you have defined with $EDITOR, and defaults to vim if you haven't defined any. Once you are done with the editor session, the file will be saved as encrypted data.

💡 EXPERT TIP

Take note of the directory you're at while creating the vault file, you will need to enter the file path in your play book. If you already missed it though, no worries you can use the command find / -name [file name] 2>/dev/null to find the path

ansible-vault create mysql_pass.yml

Ensure you do not forget your password as you will need it when you run your playbook.

When the file opens, enter the below:

mysql_root_password: <YourMySQLRootPasswordHere>

To prove that the file has been encrypted, display the content using the cat command and you will see something similar to the below image:

💡 TIP

If for any reason you need to edit an encrypted file in place, use the ansible-vault edit command. This command will decrypt the file to a temporary file and allow you to edit the file, saving it back when done and removing the temporary file:

ansible-vault edit <file name>

Configure Your Ansible Configuration File

The default ansible configuration file is in the ansible.cfg file however when you open that file you will find it almost empty with instructions on how to populate it.

When you run the ansible --version you will see where your current config file is located.

For what we want to do though we want our configuration very basic and so I will create a new ansible.cfg file and populate it with the basic configuration I want to use for this project.

vi ansible.cfg

Enter the below into the file:

[defaults]
inventory = examhost
remote_user = vagrant
host_key_checking = False

[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ack_pass = false

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=3600s

When I first ran my playbook, the execution took a whole lot of time and so I added the ssh_connection parameters to try and speed things up.

Check Connectivity

You can check your connectivity to your server(s) using adhoc commands. So long as you correctly saved the public key of your ssh key in the authorized keys file of your slave node(s) and the slave node IP address in your inventory file is correct your ping should go through.

Check your connection using the command below:

ansible all -m ping

The below image confirms that the connection was established successfully

💡 EXPERT TIP

Because we specified our inventory file in the ansible.cfg file and we are running the command from the same directory as where the ansible config file is saved, we do not need to pass the inventory file path along with the -i flag with the adhoc command.

Assuming you wanted to run the command from somewhere else this is the command to use:

ansible all -i </path/to/inventory> -m ping

Create Playbook

We have finally gotten to the main event. An Ansible playbook is a YAML file that contains a set of instructions or tasks to be executed by Ansible on remote hosts.

Playbooks allow you to define configurations, orchestrate multiple tasks, and automate complex deployments in a structured and repeatable way.

vi deploylaravel.yml

Add the below to your playbook:

---
- name: Deploy Laravel app and create cron job
  hosts: all
  become: false
  vars_files:
    - /home/vagrant/ansible/mysql_pass.yml
  tasks:
    - name: Run bash script to deploy laravel app
      ansible.builtin.script: /home/vagrant/deploylamp.sh
      environment:
        MYSQL_ROOT_PASSWORD: "{{ mysql_root_password }}"

    - name: Ensure the MAILTO variable is present so we can receive the cron job output
      cronvar:
        name: MAILTO
        value: "cnma.devtest@gmail.com"
        user: "vagrant"

    - name: Create a cron job to check the server’s uptime every 12 am
      ansible.builtin.cron:
        name: Check server uptime
        minute: "0"
        hour: "0"
        job: "/usr/bin/uptime | awk '{ print \"[\" strftime(\"\\%Y-\\%m-\\%d\"), \"]\", $0 }' >> /home/vagrant/uptime.log 2>&1"
        #"date && /usr/bin/uptime && echo >> /home/vagrant/uptime.log"

This playbook has 3 tasks, the first task will execute the script we wrote earlier.

The second is optional to this project, it will set MAILTO along with the email address. This simply tells cron to send an email with the output of the cron job to the mentioned email address when ever the job runs. Skip this task if you do not want to receive emails concerning the cron job. If you leave this however, you need to have already configured the ability to send email from your terminal. You can check out this post to see how to achieve this.

The last task will create a cron job that will run at every 12am and send the output to an uptime.log file located in our user's home directory. If you took out the second task, then ensure to add the user parameter as seen in the second task in your 3rd task if not it defaults to root as the user creating the job.

Run Playbook

Run the play book now with the below command:

ansible-playbook -i <path/to/inventory> <path/to/playbook> --ask-vault-pass

The --ask-vault-pass is necessary so that ansible will ask us for our vault password and it can use it to decrypt our secret (our mysql root password) and use it while running our script.

For verbosity, to see the logs as the playbook is executed use the -v flag. -v being the lowest and -vvvvv being the highest.

⚠️ NOTE
The playbook execution might take some time, so sit tight.

What a successful playbook execution looks like:

You can confirm that the cronjob was successfully set by listing the crontab for that user, cronjobs are user specific so ensure you run the command for the user you used.

crontab -l

Laravel App Homepage

After successfully going through this process you should see either of these displayed when you enter your IP address in your browser.

They are both the same page just in light mode and dark mode. This was a very tumultuous ride. Take note of my IP address in the images.

Light mode

Dark mode

Link to a copy of the uptime.log file.

And that's it. Exam done and dusted!!!

Install Apache Web Server and Serve a Custom Webpage Using Ansible

ChigozieCO — Mon, 29 Apr 2024 13:21:20 +0000

As a cloud engineer, automation should be at the center of everything you do. Imagine you had to setup two servers, install some webservers and add some programs for a new staff, easy peasy to do on the two servers. Just log into them one at a time and install what needs to be installed and hand over to the new staff.

Now flip it and you have to setup 100 or 200 servers for 3 departments, it would take you weeks to do this manually and you are prone to making mistakes cos the repetition would become very tiring. However when you automate the process it will take you less than an hour to complete.

⚡ What is Ansible

Ansible is an open-source platform used for automation and for various operations such as configuration management, application deployment, task automation, and IT orchestration. Ansible is easy to set up, and it is efficient, reliable, and powerful.

⚡ Ansible Architecture

Ansible uses the concepts of control and managed nodes. It connects from the control node, any machine with Ansible installed, to the managed nodes sending commands and instructions to them.

⚡ Control node requirements

For your control node (the machine that runs Ansible), you can use nearly any UNIX-like machine with Python installed. This includes Red Hat, Debian, Ubuntu, macOS, BSDs, and Windows under a Windows Subsystem for Linux (WSL) distribution. Windows without WSL is not natively supported as a control node.

⚡ Managed node requirements

The managed node (the machine that Ansible is managing) does not require Ansible to be installed, but requires Python to run Ansible-generated Python code. The managed node also needs a user account that can connect through SSH to the node with an interactive POSIX shell.

⚡ Install Ansible on Ubuntu

To install Ansible on Ubuntu you need to add the PPA to you apt repository and also install the necessary dependencies. Use the below commands:

sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

⚡ Create a Common User

Ansible requires a common user across all the nodes, if the user you are working with is called ansible ensure you have an ansible user on all your nodes.

The user on my control node is named vagrant and so I will create a vagrant user (or any other user you want) with sudo privilege on my other nodes and ensure that the user can run sudo commands without password.

(I already have a vagrant user on all my servers and so I will be using the vagrant user for this walkthrough).

This is useful and necessary so that when we run the commands and SSH into our nodes, we won't be stuck because any of the nodes requires a password.

sudo adduser vagrant

To give this user sudo privilege and ensure that the user can run sudo commands without password, I will edit the /etc/sudoers file using the command below:

sudo visudo

Enter the below into the file

vagrant ALL=(ALL) NOPASSWD:ALL

Do this for all the servers.

⚡ Generate SSH Keys on the Control Node

Before you begin this step switch to the Vagrant user.

Generate an SSH key on the control node. The private key will be on the master and we will copy the public key to the managed nodes (hosts).

Use the command below:

sudo su vagrant
ssh-keygen

Now we will copy the public key to the managed nodes, retrieve the public key and then navigate to the .ssh/authorized_keys/ file and add the copied key into the file:

On the control node

cat .ssh/id_rsa.pub

Copy the outputted public key

On the managed node

vi authorized_keys

⚡ Create Your Inventory

Ansible inventory is a file containing a list of your managed host, these are the servers on which your tasks will be carried out.

You can make use of the default hosts files in the /etc/ansible directory but it's better to create yours and use that so that in a case where you make a mistake you can use the default host file provided by ansible as a reference for the correct configuration syntax.

To that effect I will make an ansible directory and this is here i will be saving my inventory and playbooks.

mkdir ansible
sudo vi myhosts

In the inventory file add the IP addresses of your managed nodes. You can use the IP a command to get the IP address of the nodes.

⚡ Configure your Ansible Configuration file

The default ansible configuration file is in the ansible.cfg file however when you open that file you will find it almost empty with instructions on how to populate it.

When you run the ansible --version you will see where your current config file is located.

For what we want to do in this post though we want our configuration very basic and so I will create a new ansible.cfg file in the ansible directory I created earlier and populate it with the basic configuration I want to use for this post.

vi ansible.cfg

Enter the below into the file:

[defaults]
inventory = myhosts //(the config file is in the same place as the inventory, hence why I don't need to add the full path to the inventory)
remote_user = ansible //(or whatever user you are using)
host_key_checking = false

[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ack_pass = false //(we don't want a password)

Save the file. now if you run the ansible --version command again it will show the new config file path.

⚡ Check Connectivity using adhoc Command

Using ad hoc commands is a quick way to run a single task on one or more managed nodes.

Some examples of valid use cases are rebooting servers, copying files, checking connection status, managing packages, gathering facts, etc.

The pattern for ad hoc commands looks like this:

ansible [host-pattern] -m [module] -a “[module options]”

We will use the ping module to check the connectivity to our servers.

Run the below command:

ansible all -m ping

💡 TIP

Because we specified our inventory file in the ansible.cfg file we do not need to pass the inventory file path along with the -i flag with the adhoc command.

⚡ Prepare Custom Website Files

Apache is a webserver and it serves an apache website by default, this is not the site we want our apache webserver to server and so we would need to retrieve the config file for our custom website.

The files are hosted in my GitHub account and so clone that into a folder on my control node.

First I need to install git on the control node, do that using the command below:

sudo apt update
sudo apt install git

I will create a new directory and it is in this directory that I would clone this repo into (you can use any repo of your choice). The repo contains the html and CSS code for a webpage I designed earlier on.

mkdir webpage

Now I'll navigate into the directory and clone the project there.

cd webpage
git clone https://github.com/ChigozieCO/assignment-03-WP-Pusher.git

This page will be used in our playbook.

⚡ Create Playbook

Create a playbook directory and navigate into the directory

mkdir playbook
cd playbook

Now we will create our playbook, I will name it serverplay as the playbook will be contain tasks to install apache2 web server on my nodes.

sudo vi serverplay.yml

Copy the below and paste it into your playbook, this is the content of your playbook

---
- name: Setup webserver and copy custom webpage
  hosts: all
  become: yes
  tasks:
    - name: Update apt package cache (Ubuntu) / Update yum package cache (CentOS)
      when: ansible_os_family == 'Debian'
      apt:
        update_cache: yes

    - name: Update yum package cache (CentOS) / Update apt package cache (Ubuntu)
      when: ansible_os_family == 'RedHat'
      yum:
        update_cache: yes

    - name: Install Apache (Ubuntu) / Install Httpd (CentOS)
      package:
        name: "{{ 'apache2' if ansible_os_family == 'Debian' else 'httpd' }}"
        state: latest

    - name: Enable and start Apache/Httpd
      ansible.builtin.service:
        name: "{{ 'apache2' if ansible_os_family == 'Debian' else 'httpd' }}"
        enabled: yes
        state: started

    - name: Copy custom webpage files
      ansible.builtin.copy:
        src: "~/ansible/webpage/assignment-03-WP-Pusher/"
        dest: "/var/www/html/"
        mode: '0755'

    - name: Restart Apache2 (Ubuntu) / Httpd (CentOS)
      ansible.builtin.service:
        name: "{{ 'apache2' if ansible_os_family == 'Debian' else 'httpd' }}"
        state: restarted

⚡ Run Playbook

Run the play book now with the below command:

ansible-playbook playbook/serverplay.yml

💡 Note

Because I am running my playbook from the ansible directory we created earlier, I do not need to specify the location of my inventory as I have done that in my config file already.

My playbook ran successfully and we can see it in the screenshot below:

To see your webpage enter your node's IP address in a web browser, from the below screen shots you can see the pages displayed on my browser.

managed node 1

managed node 2

If this post has helped you, consider supporting me:

Configure Postfix to Send Email with Gmail's SMTP From the Terminal

ChigozieCO — Wed, 10 Apr 2024 13:15:51 +0000

Open a tab on your browser, head to Gmail or yahoo mail or whatever other mail agent you use and then compose an email. This is the normal method of sending email from our computer that almost everyone is familiar with, but what if I told you you can send emails directly from your Linux terminal?

There are various ways to go about this but I will be using Mailx and Postfix on an ubuntu OS. Follow along and in no time you would have successfully sent your first mail from your Linux terminal.

Install Mailx and Postfix

Mailx is a UNIX utility program for sending and receiving mail, also known as a Mail User Agent program.

The mailx command in Linux is a powerful tool that lets you send emails directly from your command line. It’s simple, efficient, and incredibly flexible.

It needs to be installed

First update your apt package

sudo apt update

Then use this command to install mailx:

sudo apt install mailutils

Email is sent and received using Postfix, a mail transfer agent (MTA). When you install mailx with the above command postfix is automatically installed as well if you didn't already have it installed.

After successful installation you will see the postfix configuration dialog (as shown below), you can choose the internet site option highlighted below.

Like I said, installing mailx will install postfix automatically however if you want to install postfix manually, use the below command:

sudo apt install postfix

Generate an Application Specific Password

We will need to save our gmail password in a file for this configuration but for security purposes I do not want to save my password in a file and this is why we will be generating an app specific password that we will use for this purpose.

This password will be unique and can only be used by postfix.

This password will be used by Postfix for SMTP authentication instead of your actual Gmail password. Even if someone were to get hold of this password it wouldn't work if it weren't from postfix.

Follow these steps to generate the password:

💡 TIP
You must enable two factor authentication to be able to generate an app specific password on gmail.

⚡ Go to your Google Account settings (https://myaccount.google.com/).

⚡ Navigate to the "Security" section.

⚡ Under "Signing in to Google", select 2-Step Verification.

⚡ At the bottom of the page, select App passwords

⚡ Type a name for the password.

⚡ Click on "Create".

⚡ Copy the generated application-specific password and keep it handy, we will make use of it.

This password will be used in the Postfix configuration.

Create Your `sasl_passwd` File

sasl is simple authentication and security layer.

We will create a sasl_passwd file in the /etc/postfix/sasl directory

sudo vi /etc/postfix/sasl/sasl_passwd

In this file we will add the app specific password that we generated above.

Enter the below code into the file, what you're basically doing here is saving your gmail authentication (username and password) so that it can be used by postfix for authenticating.

💡 Expert Tip

The regular port used for postfix is 587, I started out using port 587 but it turned out that for some reasons my port 587 was closed (either that or my ISP was blocking communication from that port) and so my mails weren't sending so I had to switch to port 465 as you will see me use in this walkthrough.

In the same light, if you follow through this post and your mails do not make it out of the mail queue, try other posts such as 587 and 995.

Enter your password in the format below:

[smtp.gmail.com]:465 <your gmail email address>:<the app specific password>

eg [smtp.gmail.com]:465 example@abc.com:yourAPPspecificpassword

⚠️ NOTE
This is the email address that will be sender for all your outgoing emails.

The next thing to do is to create a hash database file for the password because postfix will only read the hash.

Do this with the command below:

sudo postmap /etc/postfix/sasl/sasl_passwd

This will create a sasl_passwd.db file in the /etc/postfix/sasl directory. You can ls that directory to confirm that it was correctly created.

At this point, for security purposes we will change the permissions of these two files above to ensure that only root (which is the owner of the files) can read them.

Use the command below:

sudo chmod 600 /etc/postfix/sasl/sasl_passwd /etc/postfix/sasl/sasl_passwd.db

Configure Postfix to Use the Gmail Server

If you have your own custom smtp server you would be using it here but we will use the Gmail smtp server instead.

Open the postfix main.cf file:

sudo vi /etc/postfix/main.cf

Locate the relayhost and enter Gmail smtp there

relayhost = [smtp.gmail.com]:465

Add the following to the end of /etc/postfix/main.cf to enable SASL authentication for Postfix.

# Enable SASL authentication
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_wrappermode = yes

Confirm that the certificate file exists. If it doesn't, you can install it using the below command:

sudo apt install ca-certificates

Restart Postfix

To ensure that all these changes take effect, we need to restart Postfix.

Do this with the command below:

sudo systemctl restart postfix

Firewall

Depending on your firewall configurations your firewall could block communications and cause the emails to not be sent so check your firewall status and if need be add rules to ensure that smtp is allowed.

sudo ufw status

If your firewall is inactive you can proceed along without adding any new rules.

To add new rules to allow smtp, use the below commands:

sudo ufw allow "Postfix"
sudo ufw allow "postfix SMTP"
sudo ufw allow "Postfix Submission"

Export SSL Cert File

This step is optional, you might have no need for it.

I needed to do this because I discovered that my openssl in my env could not find the path to my SSL certificate and so my connections kept dropping before the TLS handshake could be completed.

To solve this issue and successfully send a mail from my terminal, I have to export my CAFile path so openssl knows where to look.

Use the command below:

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

To persist it so that every time you boot up your system the changes remain, enter the above command in the ~/.bashrc file or the ~/.bash_profile file.

sudo vi ~/.bashrc
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

To apply the changes immediately, you can close your terminal and open another one or use the below command to apply the change to that same terminal.

source ~/.bashrc

Send Mail

There are various ways you can send a mail from your terminal, I will show a few of them below.

Method 1

The first method is using the echo command with mailx.

echo "Message 1 sent through the terminal" | mailx -s "Echo command mail 1" <email address of recipient>

We do not need to specify the sender as we have already done that in our sasl_passwd file.

The s flag indicates the subject of the mail.

Method 2

In an instance when you want to write an email with a very full body containing multiple lines you can write the body directly in the terminal without the echo command.

Start with the mailx command, the subject and the recipient then click enter. At this point mailx will wait for you to type the body of your email.

Go ahead and write the body of your email.

When you are done, enter a new line and press ctrl D to signify the end of the email and send it.

mailx -s "Terminal email number 2" <recipient's email address>
This is a mail trying out another method of sending an email from the terminal.
This method we are typing multiple times.
You can type multiple lines with the echo command as well.
But when you want multiple lines I prefer to use this method.
That's it for this example.

Thanks guys.

You can see that the message was received.

Method 3

It is equally possible for the body of your mail to be gotten from the contents of a file.

To do this you will redirect the contents of the file to the command to send the mail like so:

mailx -s "Mail body from a file, mail 3" firstperson@mail.com, person2@mail.com, person3@mail.com < path/to/file

You can also send the mail to several people at the same time as shown in the code snippet above.

To demonstrate this, the first thing I will do is to create the file and then add some text to it. I'll do this with one command. Shown below:

echo "This is the body of our mail that will be gotten from this file mail.txt >> mail.txt

Next I will send my mail

mailx -s "Mail body from a file, mail 3" [recipient email address] < ./mail.txt

We can see that the content of the file has now been translated into the body of our mail.

Useful mailx flags

Flag	Description	Example
`-s`	Subject of the mail	echo 'Hello' \| mailx -s 'Subject' user@mail.com
`-c`	Add a CC to the mail, this will add the email address specified after the flag as CC (carbon copy) to the mail	echo 'Hello' \| mailx -s 'Subject' -c user2@mail.com user@mail.com
`-b`	Add a BCC to the mail, this will add the email address specified after the flag as BCC (blind carbon copy) to the mail	echo 'Hello' \| mailx -s 'Subject' -b user2@mail.com user@mail.com
`-r`	Specify recipient email address to the mail	echo 'Hello' \| mailx -r from@mail.com -s 'Subject' user@mail.com
`-a`	Add attachment to the mail	echo 'Hello' \| mailx -s 'Subject' -a file.txt user@mail.com

And there you have it, don't forget to share this post if you found it helpful.

If this post has helped you, consider supporting me:

Vagrant CLI Commands Cheat Sheet

ChigozieCO — Mon, 11 Mar 2024 10:24:35 +0000

In a previous post, I explained what Vagrant is, how to install it, how to use it to install virtual machines and how to connect to installed VM using it. If you haven't already seen that post you can check it out here, check it out first and head back to this post to familiarize yourself with these commands.

If you find this post useful, consider supporting me:

💡 PRO TIP
When running Vagrant commands, you must be in a directory containing a Vagrantfile or you will receive the following error: A Vagrant environment or target machine is required to run this command.

So be sure to navigate to the directory where you have your Vagrantfile saved.

Vagrant CLI Commands

Below are most of the most basic vagrant commands you will likely use in your everyday interaction with vagrant. Nothing too deep or too complicated.

📌 vagrant init : This command initializes vagrant and generate a Vagrantfile with default configurations specifying the box you supplied in the command.

📌 vagrant init --minimal: Generate a Vagrantfile without all the additional instructional comments that vagrant adds by default.

📌 vagrant box add :This will download a box image to your computer.

📌 vagrant up: Start up a virtual machine in vagrant. If this is your first time of running it, it will build the virtual machine using the configurations specified in your Vagrantfile. Subsequently, it just starts up your already created VM.

📌 vagrant ssh: Remotely access your vm in vagrant via ssh.

📌 vagrant ssh-config: Retrieve the IP address and port of your virtual machine (basically to view your ssh configuration just as the name states). This command will only work when your VM is running.

When the command is successfully run, you should see something that resembles the below

📌 vagrant halt: Halt the machine, this attempts a graceful shut down. Here the machine turns off just like when you shut down your physical machine.

📌 vagrant suspend: This command will save the state of your machine, just like when you hibernate your physical computer. A suspended machine is a machine that is turned off but not shut down.

📌 vagrant resume: This resumes a Vagrant managed machine that was previously suspended using vagrant suspend. This command will not start up a machine that was powered off using vagrant halt, if you attempt to start a shut down machine you will get the below error.

When the command is successfully run, you should see something that resembles the below

📌 vagrant reload: This command is very useful when modifications have been made to the Vagrantfile. It is the equivalent of running the vagrant halt and vagrant up command. It is required for changes made in the Vagrantfile to take effect.

📌 vagrant status: This will tell you the state of the machines Vagrant is managing. It comes in very handy when you forget whether you VMs are still running, halted or suspended, trust me it's very easy to forget especially when you start managing a lot of virtual machines with vagrant.

📌 vagrant destroy: stops and deletes all the virtual machine created with vagrant and also destroys all resources that were created during the machine creation process.

Install Virtual Machine With Vagrant & Remotely Access it

ChigozieCO — Mon, 11 Mar 2024 10:23:54 +0000

For this walk through I will be installing a new VM using Vagrant and then remotely accessing the VM using SSH with Vagrant.

What is Vagrant

Installed as a layer of software between a virtualization tool (like VirtualBox, Docker, or Hyper-V) and a virtual machine (VM), Vagrant is an open-source program that makes it simple to create, configure, and manage boxes of virtual machines through an intuitive command interface.

Why is Vagrant Useful

The process of setting up and maintaining virtual machines can be laborious and time-consuming. Even more difficult is replicating the virtual machine on an alternate server as well as if you have to replicate more than one virtual machine (VM).

Vagrant is an effective tool that can make setting up and maintaining your development environment easier. Owing to the fact that it writes your configuration as code, it makes it possible for teams (and every other VM you create with your VagrantFile) have the same virtual machine setup across board.

For a basic cheat sheet of every day vagrant commands that you will use in your day to day operations with vagrant, check out this post.

Install a Hypervisor and Vagrant

🌟 Before you begin, ensure you have a hypervisor such as Oracle Virtual box or VMWare. For this tutorial I will be making use of Oracle Virtual box, you can download it from here.

🌟 To install a VM with Vagrant, first install vagrant in your computer, get it from vagrantup.com

🌟 Vagrant is automatically added to your env path so to check if the installation was successful, launch your terminal and enter the following command to output the installed version number:

vagrant --version

🌟 Once it's installed find the VM (box) you want to install on vagrant cloud and note the name.

🌟 For this installation I will be using ubuntu 20.04-LTS and we can see from vagrant cloud that it is called focal/64.

Initialize Vagrant

🌟 The next thing to do is to initialize vagrant in other to create the Vagrantfile that would be used for our configuration.

🌟 For each box I create with vagrant I create a specific directory for that box to keep things organised, that way the Vagrantfile (which is the configuration) of that box is stored in that directory and so before we initialize vagrant we will create the necessary directories.


mkdir -p vagrant/boxes/ubuntu20.04-LTS
cd vagrant/boxes/ubuntu20.04-LTS

Tip
the p flag will make parent directories as needed and will not result in any error if already existing.

🌟 Now we can go ahead and initialize vagrant by running the command below

vagrant init <box name>

vagrant init ubuntu/focal64

🌟 Your Vagrantfile is now ready to be used to spin up your Virtual machine but before we do this I want to make some changes to our VM configuration in the Vagrantfile. Be sure to read the Vagrantfile so you understand the configurations of your VM.

Edit the Virtual Machine Configuration.

I want my VM to use a private network where the IP will be assigned via DHCP and I also want to configure a shared folder between the Host OS and the Guest OS.

These are the changes we will be making to the Vagrantfile.

🌟 Using any editor of your choice, open the Vagrant file. I will be making use of vim.

vi Vagrantfile

🌟 Scroll to the part of the file that says Create a private network, which allows host-only access to the machine as shown in the image below.

This is what the part looks like and the line highlighted is the line we will change.

🌟 First, uncomment it and edit it as shown below:

  config.vm.network "private_network", type: "dhcp"

This is what it should now look like:

🌟 The next change we will make is to add a shared folder between the host OS and the guest OS. Scroll down in the file and find where it says Share an additional folder to the guest VM. The first argument is

Uncomment the highlighted line and choose a host folder of your choice, I want it in the same directory I am working on and so I will just make a slight change as can be seen below:

  config.vm.synced_folder "./data", "/vagrant_data"

Note: Ensure that the specified host directory already exist if not you will get an error message saying There are errors in the configuration of this machine. Please fix the following errors and try again:

vm:
* The host path of the shared folder is missing: ./data

Save the Vagrantfile and we are ready to spin our VM up.

Create the VM

🌟 After saving the Vangrantfile configuration changes we made, we are ready to run the command we need to create our VM and we do this with the command below:

vagrant up

Note that the above command assumes that you are using Oracle virtual box as your hypervisor. If your were using VMware or any other virtualization tool the command to use is:

vagrant up <hypervisor>

vagrant up VMware

The above command will first download the ubuntu/focal64 VM image (or whichever image you selected) from the vagrant cloud and then it will start the VM.

Also, it will generate an SSH key pair and adds the public key to the VM during this process so that we can SSH into the machine once it is up and running.

If you get this error displayed in the screenshot below, follow the next set of steps to correct the error.

Fix DHCP Error

🌟 Head over to your virtual box (hypervisor), launch it.
🌟 Select the VM that Vagrant just created and on the top left hand click on file.
🌟 Click tools and select network manager.

🌟 Select the host-only network, either right click on it and click on remove or select it and then at the top right click on remove and remove when asked for confirmation.

🌟 When this is done go back to the terminal you are setting up your vagrant VM with and enter the below command.

vagrant reload

The configuration of your VM should complete successfully now.

SSH into the Machine

🌟 After the setup is complete you can now ssh into the machine:

vagrant ssh

Out of date Guest Additions

If you see the message saying your guest additions are out of date, as seen in the image below, and you're like me that can't let things just be. Here's how to update your guest additions.

🌟 If you had previously SSH into the VM, exit the remote access:

exit

🌟 Run the vagrant vb-guest plugin

vagrant plugin install vagrant-vbguest

🌟 Next halt your VM by running the command:

vagrant halt

🌟 Then run the vagrant up command. You will see a lot of installations happening, allow it run its full course.

🌟 Repeat the two commands again if need be:

vagrant halt
vagrant up

🌟 You should now see a message saying GuestAdditions <version> running -- OK

🌟 You are good to go now, SSH into your machine and start enjoying tinkering around the Linux OS.

If this post has helped you, consider supporting me:

Remotely Accessing a Virtual Machine: SSH Key Pair

ChigozieCO — Mon, 19 Feb 2024 10:02:00 +0000

SSH (Secure Shell) is used for remotely accessing your server and it usually comes installed with a lot of Linux OS but where it is not installed, you can install it by installing the application OpenSSH.

The remote system must have a version of SSH installed. The information in this post assumes the remote system uses OpenSSH, see how to install OpenSSH (client and server) below.

SSH authentication can be via password or using private and public key pairs.

While you can create a user with a password to login into any Linux system, sometime accessing the Linux system via that means is not possible either because it is not enabled by your system (in a corporate environment) or for some other reason and the only way is to SSH into the system.

SSH keys come in pairs, the private and the public key. The private keys are always kept in the local machine that needs to connect to the remote system somewhere while the public keys can be shared with sysadmins to add to your corporate server or used in some form of authentication to give you access.

On the remote machine, the public key is stored in a file called authorized keys. This is where the SSH service will check to see if the key on your machine matches the public key on the server before letting you in.

You can generate ssh keys with the command:

ssh-keygen

Keys are stored in the .ssh folder inside a users home directory (/home/$USER/.ssh).

Install OpenSSH (Client & Server)

If you don't have SSH installed follow the steps below to install it, if you already do ignore the next few steps and do continue along to connect.

🌟 Start your Linux machine in a normal start.

🌟 Open the terminal and type the command below and enter your password when prompted for it.

sudo apt update -y

🌟 You can search for the package before installing it with the command:

sudo apt search openssh-client -y

The image below shows the one we want to make use of.

🌟 Next we install it using the command:

sudo apt install openssh-client -y

🌟 In the same way we install the openssh-server using the command

sudo apt install openssh-server -y

🌟 When your installations are complete, confirm that the SSH service is running with the command

systemctl status ssh

Generate SSH Keys

For this tutorial I will be working with two Virtual machines on my local computer, they are both Ubuntu. The remote location is shown on the terminal as anulika@Goz and the local SSH host which is acting as my local computer is my vagrant VM which shows up on the terminal as anulika@ubuntu-focal.

🌟 The first thing I want to do is to check for any existing keys my local user might have. This step is not required but it is recommended, type in the command below:

ls ~/.ssh/id_*

If you do not see any output or if you see an output like that below then you do not have any keys present.

If you see an output listing out keys, then you have existing keys you should back them up so that you don't lose them incase you accidentally delete them.

🌟 To create an SSH key, ensure you are in your local computer (the SSH host) and run the command below:

ssh-keygen

It will let you know that it is generating a public/private rsa key. By default it will use the rsa standard for all systems, if you want to use a different algorithm you can specify it with the -t flag. It is also good practice to add a comment, you do this by using the -C flag as seen below:

ssh-keygen -t rsa -C <"Your comment">

🌟 You will be asked where you want to save the key, by default it will use the id_rsa file, I will leave the default so I press enter.

🌟 When asked for a passphrase I just click enter because I don't want a passphrase. We're trying to avoid using passwords and so adding a passphrase will take me back to having to enter a password every time I want to connect.

Note that adding a passphrase is an additional security measure so that if anybody somehow gets hold of your computer with the private key, without knowing the passphrase they won't be granted access.

🌟 It will then go ahead and generate your keys.

Retrieve your Public Key

🌟 When you run the ls -la command now in the home directory of the user for which you just created an SSH key you should see the .ssh directory.

cd /home/<your user>
ls -la

🌟 Move into the .ssh directory and list it's contents and you will see both the id_rsa and the id_rsa.pub files. The former holds your private key while the latter holds your public key. This is where we will retrieve the public key from.

cd .ssh
ls -la

🌟 Output the contents of the id_rsa.pub file and copy it.

cat id_rsa.pub

This is the key you will add to your remote machine in a file called authorized keys.

Add the Public Key to the Remote Machine

The public key for an SSH key pair needs to be added to a remote machine that you can SSH access. The remote machine will use the public key to decrypt the connection that the SSH host machine—your local computer— used its private key to encrypt.

Transferring your public key to the remote system is a must. As a result, you need to either have an administrator on the remote system add the public key to the ~/.ssh/authorized_keys file in your account or be able to log into the remote system using your established username and password/passphrase.

Note

If you already have an ~/.ssh/authorized_keys file, probably because you have previously remotely accessed that machine using SSH key authentication, all you need to do in this section is to edit the ~/.ssh/authorized_keys file and add your new public key. In the authorized_keys file, add the new key in a new line and then save the file.

🌟 Head over to your remote machine, open your terminal and navigate to the home directory of your user of choice.

cd ~

🌟 List it's contents to check if you have an .ssh directory. If you have one, list its contents to see if it contains an authorized_keys file

ls -la ~
cd .ssh // #if the .ssh file exists
ls -la

If the ~/.ssh/authorized_keys file exists, skip the next step and continue on to edit the file and place you public key in it.

🌟 If your account on the remote system doesn't already contain a ~/.ssh/authorized_keys file, create one; on the command line, enter the following commands:

cd ~
mkdir .ssh
cd .ssh
touch authorized_keys

With those 4 commands above we simply navigated to the user's home directory, created the .ssh directory, entered into the directory and created an authorized_keys file.

🌟 Next we would use a file editor to add our public key to the authorized_keys file.

vi authorized_keys

Paste your public key in the file, save and close it.

Retrieve IP Address of Your Remote machine

🌟 You need to know the IP address of the remote machine, run the below command to obtain it's IP address.

Before you start protesting about needing to have the IP address of the remote machine (I mean if you had access to the remote machine why would you need to connect to it remotely right? Wrong!!)
If you know anything about hacking, the first thing you need to do when to begin hacking any machine or server is to find the IP address of that server.

ip --brief addr show

Copy the IP address of the interface you want to use, leave your machine turned on.

If like me you are working with two VMs on the same host machine, ensure that the IP address you choose is different from that of the VM you are using as your SSH host.
If you use NAT adapter and you notice that the two VMs have the same IP address power them off and add another adapter attached to Host-only Adapter apply the changes and restart your VM. Do this for the VMs, this way they will both get different IP addresses.
Use this unique IP address for the next step.

Access the Remote Machine Using the SSH Key Pair

Now we have everything set up and if you followed the steps correctly, you should too.

Return to your local computer, the SSH host, which has the private key and type the below command in the terminal.

ssh <user>@<ip address>

The local SSH host which is acting as my local computer is my vagrant VM which shows up on the terminal as anulika@ubuntu-focal and the remote location shows on the terminal as anulika@Goz.

If you look at what the arrows in the screenshot above points to you will notice the change when the ssh remote access was successful.

You have now successfully implemented remotely accessing a server using SSH key pairs, go on ahead and brag about your abilities champ.

If this post has helped you, consider supporting me:

Forem: ChigozieCO

Dockerized Deployment of a Full Stack Application with Reverse Proxy, Monitoring & Observability

Project Overview

Prerequisite

Clone Application Repo

Containerization

Dockerize Backend

Version 1

Multi-stage Version 2 Build

Build Backend Image

Dockerize Frontend

Configure Application Docker Compose file

First Draft of Compose File to test locally

Explanation

Spin up Application

Application Frontend

Application's Swagger UI

Adminer Dashboard

Cloud Deployment

Steps to Create an EC2 Instance

Copy project File into Instance

Configure Reverse Proxy - Traefik

Traefik Configuration

Explanation

Docker Compose Configuration for Traefik

Updates Made

Generate Password Hash

Prepare the acme.json File

Encrypt and Decrypt Acme File

Create DNS (A) Record for Domains

Update your Environments

backend/.env

frontend/.env

Build Application

Application Frontend

Application Backend Root

Application's Swagger UI

Adminer Dashboard

Traefik Dashboard Asking for Authentication

Unauthorized Response From Traefik Dashboard

Traefik Dashboard

Monitoring and Observability

Configure Prometheus

Docker Compose Configuration for Prometheus

Prometheus Configuration File

Configure cAdvisor

compose.monitoring.yaml

prometheus.yml

Configure Loki

Docker Compose Configuration for Loki

compose.monitoring.yaml

Loki Configuration File

Configure Promtail

Docker Compose Configuration for Promtail

compose.monitoring.yaml

Promtail Configuration File

Configure Grafana

Docker Compose Configuration for Grafana

Spin up Services

compose.yaml

Prometheus Web UI

Create Dashboards

Grafana Web UI Login

Add Data sources

Grafana welcome page

Prometheus Data source

Prometheus data source

Prometheus server url

Successfully queried Prometheus API

Loki Data Source

Data sources menu

Loki data source addition

Import cAdvisor Dashboard

Create first Dashboard

Import dashboard - cAdvisor

Add cAdvisor Dashboard

cAdvisor Dashboard

Import Loki Dashboard

Import Traefik Dashboard

Conclusion

Prepare the `acme.json` File

`backend/.env`

`frontend/.env`

`compose.monitoring.yaml`

`compose.monitoring.yaml`

`compose.yaml`

Create a `.gitignore` File

Update Your secrets `terraform.tfvars` File