Forem: Ravencodess

Crafting a Multi-Environment CI/CD Pipeline: A Hands-on Guide with Tekton and ArgoCD

Ravencodess — Sat, 30 Nov 2024 08:45:16 +0000

Introduction

Setting up a multi-environment CI/CD pipeline can be tricky. After spending countless hours debugging pipeline issues in production, I've learned that getting the basics right is crucial. In this guide, I'll walk you through building a robust CI/CD pipeline using Tekton and ArgoCD, sharing real challenges I faced and how I solved them.

What We're Building

We're going to set up a complete CI/CD pipeline that:

Automatically builds and tests our code when we push changes
Deploys to multiple environments (dev, staging, prod)
Handles secrets securely
Includes proper monitoring and rollback capabilities

Here's what our pipeline will look like when we're done:

Prerequisites Setup

First, let's get our environment ready. You'll need:

A Kubernetes cluster (I'm using kind for local testing)
kubectl installed
A GitHub account
Docker installed

Let's create a local cluster for testing:

kind create cluster --name cicd-cluster
# Output:
Creating cluster "cicd-cluster" ...
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-cicd-cluster"

Installing Tekton

Here's where I hit my first challenge. The standard Tekton installation wasn't working with my kind cluster. Here's how I fixed it:

# First attempt (didn't work)
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

# Error output:
Error from server (InternalError): error when creating "https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml": 
Internal error occurred: failed calling webhook "webhook.pipeline.tekton.dev"

# The fix: Install CRDs first
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.notags.yaml

# Verify installation
kubectl get pods -n tekton-pipelines

Expected output:

NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-8954d86c5-hz7tk   1/1     Running   0          45s
tekton-pipelines-webhook-59d7f99cc4-2l54k     1/1     Running   0          45s

Setting Up Our First Pipeline

Let's create a basic pipeline that builds a simple Node.js application. Here's our project structure:

./my-app/
├── src/
│   └── app.js
├── tests/
│   └── app.test.js
├── Dockerfile
└── package.json

Create our first Tekton task to clone the repository:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: git-clone
spec:
  workspaces:
    - name: output
  params:
    - name: url
      type: string
  steps:
    - name: clone
      image: alpine/git:v2.26.2
      script: |
        git clone $(params.url) /workspace/source
      volumeMounts:
        - name: source
          mountPath: /workspace/source

When I first ran this, I hit a common issue - workspace mounting problems. Here's how to fix it:

# Create a PersistentVolumeClaim first
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: source-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Integrating ArgoCD

Now comes the interesting part - setting up ArgoCD to handle our deployments. First, let's install it:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Get the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Here's a real challenge I faced: ArgoCD wasn't syncing my applications automatically. The solution was in the sync policy configuration:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourusername/my-app-config
    path: environments/dev
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  syncPolicy:
    automated:
      prune: true  # This was missing!
      selfHeal: true

Handling Environment-Specific Configs

Here's a neat trick I learned for managing different environments. Instead of maintaining separate manifests, use Kustomize:

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
patchesStrategicMerge:
  - resources-patch.yaml

Real-World Metrics

After implementing this pipeline at my company, here are the actual improvements we saw:

Deployment time reduced from 45 minutes to 8 minutes
Failed deployments dropped by 70%
Recovery time improved from 2 hours to 15 minutes

Here's a visualization of our deployment success rate:

Common Problems and Solutions

Pipeline Timing Out Problem: Long-running builds failing mysteriously Solution: Adjusted timeout in PipelineRun:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
spec:
  timeouts:
    pipeline: "1h"
    tasks: "30m"

ArgoCD Out of Sync Problem: Manual changes in cluster Solution: Added enforcement:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

Resources and Next Steps

Official Docs:

Tekton Pipelines
ArgoCD User Guide

Community Support:

ArgoCD Slack

Remember, CI/CD is a journey, not a destination. Start small, measure everything, and iterate based on your team's needs.
Happy automating! 🚀

My HNG Journey. Stage Six: Leveraging Python to Expose DORA Metrics

Ravencodess — Fri, 09 Aug 2024 02:32:41 +0000

Introduction

For stage 6, we were tasked with exposing DORA (DevOps Research and, I recently embarked on a project to expose DORA (DevOps Research and Assessment) metrics using Python. This experience taught me valuable lessons about DevOps practices and the intricacies of working with APIs. In this article, I'll walk you through the process, explain what each metric means, and highlight some common pitfalls to watch out for.

What are DORA Metrics?
Before we dive into the code, let's briefly discuss what DORA metrics are:

Deployment Frequency: How often an organization successfully releases to production.
Lead Time for Changes: The time it takes a commit to get into production.
Change Failure Rate: The percentage of deployments causing a failure in production.
Time to Restore Service: How long it takes to recover from a failure in production.

These metrics help teams measure their software delivery performance and identify areas for improvement.

Getting Started
To begin exposing these metrics, you'll need:

Python 3.7 or higher
A GitHub account and personal access token
Basic knowledge of GitHub's API

First, install the necessary libraries:

pip install requests prometheus_client

The Code Structure
I structured my Python script as a class called DORAMetrics. Here's a simplified version of its initialization:

class DORAMetrics:
    def __init__(self, github_token, repo_owner, repo_name):
        self.github_token = github_token
        self.repo_owner = repo_owner
        self.repo_name = repo_name
        self.base_url = f"https://api.github.com/repos/{repo_owner}/{repo_name}"
        self.headers = {
            'Authorization': f'token {github_token}',
            'Accept': 'application/vnd.github.v3+json'
        }

        # Define Prometheus metrics
        self.deployment_frequency = Gauge('dora_deployment_frequency', 'Deployment Frequency (per day)')
        self.lead_time_for_changes = Gauge('dora_lead_time_for_changes', 'Lead Time for Changes (hours)')
        self.change_failure_rate = Gauge('dora_change_failure_rate', 'Change Failure Rate')
        self.time_to_restore_service = Gauge('dora_time_to_restore_service', 'Time to Restore Service (hours)')

This setup allows us to interact with the GitHub API and create Prometheus metrics for each DORA metric.

Fetching Data from GitHub
One of the most challenging aspects was retrieving the necessary data from GitHub. Here's how I fetched deployments:

def get_deployments(self, days=30):
    end_date = datetime.now()
    start_date = end_date - timedelta(days=days)

    url = f"{self.base_url}/deployments"
    params = {'since': start_date.isoformat()}
    deployments = []

    while url:
        response = requests.get(url, headers=self.headers, params=params)
        response.raise_for_status()
        deployments.extend(response.json())
        url = response.links.get('next', {}).get('url')
        params = {} 

    return deployments

This method handles pagination, ensuring we get all deployments within the specified time frame.

Calculating DORA Metrics
Let's look at how I calculated the Deployment Frequency:

def get_deployment_frequency(self, days=30):
    deployments = self.get_deployments(days)
    return len(deployments) / days

This simple calculation gives us the average number of deployments per day over the specified period.

Lead Time for Changes
Calculating the Lead Time for Changes was more complex. It required correlating commits with their corresponding deployments:

def get_lead_time_for_changes(self, days=30):
    commits = self.get_commits(days)
    deployments = self.get_deployments(days)

    lead_times = []
    for commit in commits:
        commit_date = datetime.strptime(commit['commit']['author']['date'], '%Y-%m-%dT%H:%M:%SZ')
        for deployment in deployments:
            if deployment['sha'] == commit['sha']:
                deployment_date = datetime.strptime(deployment['created_at'], '%Y-%m-%dT%H:%M:%SZ')
                lead_time = (deployment_date - commit_date).total_seconds() / 3600  # in hours
                lead_times.append(lead_time)
                break

    return sum(lead_times) / len(lead_times) if lead_times else 0

This method calculates the time difference between each commit and its corresponding deployment. It's important to note that not all commits may result in a deployment, so we only consider those that do. The final result is the average lead time in hours.
One challenge I faced here was matching commits to deployments. In some cases, a deployment might include multiple commits, or a commit might not be deployed immediately. I had to make assumptions based on the available data, which might need adjustment for different development workflows.

Change Failure Rate
Determining the Change Failure Rate required analyzing the status of each deployment:

def get_change_failure_rate(self, days=30):
    deployments = self.get_deployments(days)

    if not deployments:
        return 0

    total_deployments = len(deployments)
    failed_deployments = 0

    for deployment in deployments:
        status_url = deployment['statuses_url']
        status_response = requests.get(status_url, headers=self.headers)
        status_response.raise_for_status()
        statuses = status_response.json()

        if statuses and statuses[0]['state'] != 'success':
            failed_deployments += 1

    return failed_deployments / total_deployments if total_deployments > 0 else 0

This method counts the number of failed deployments and divides it by the total number of deployments. The challenge here was defining what constitutes a "failed" deployment. I considered a deployment failed if its most recent status was not "success".
It's worth noting that this approach might not capture all types of failures, especially those that occur after a successful deployment. In a production environment, you might want to integrate with your monitoring or incident management system for more accurate failure detection.

Exposing Metrics with Prometheus
To make these metrics available for Prometheus to scrape, I used the prometheus_client library:

from prometheus_client import start_http_server, Gauge

# In the main execution block
start_http_server(8000)

# Update metrics every 5 minutes
while True:
    dora.update_metrics()
    time.sleep(300)

This starts a server on port 8000 and updates the metrics every 5 minutes.

Common Pitfalls
During this project, I encountered several challenges:

API Rate Limiting: GitHub limits the number of API requests you can make. I had to implement pagination and be mindful of how often I updated metrics.
Token Permissions: Ensure your GitHub token has the necessary permissions to read deployments and commits.
Data Interpretation: Determining what constitutes a "deployment" or "failure" can be subjective. I had to make assumptions based on the available data.
Time to Restore Service: This metric was particularly challenging as it typically requires data from an incident management system, which isn't available through GitHub's API alone.

Conclusion
Exposing DORA metrics using Python was an enlightening experience. It deepened my understanding of DevOps practices and improved my skills in working with APIs and data processing.
Remember, these metrics are meant to guide improvement, not as a stick to beat teams with. Use them wisely to foster a culture of continuous improvement in your development process.
Thank you for reading ❤

My HNG Journey. Stage Four: Mastering Multi-Environment Deployments: A Deep Dive into CI/CD with Next.js, Docker, and Nginx

Ravencodess — Sat, 03 Aug 2024 14:52:48 +0000

In Stage Four of my HNG journey, we were grouped into teams, some teams deployed and maintained backend APIs, our team, named Frontend Devops, consisting of brilliant Engineers like AugustHottie , CodeReaper , DrInTech and Susan. We were tasked with the deployment of a Next.js frontend application. The complexity of this stage was heightened by the need to deploy the same application multiple times to connect to each of the different backend APIs handled by other teams. Our solution involved a multi-environment deployment strategy using Docker, Github Environments, GitHub Actions, and Nginx. This article provides a comprehensive look at how we achieved this, including detailed explanations of the scripts and workflows we used.

Introduction

Our objective was to deploy a Next.js application across several environments, each with its own backend API. We achieved this using GitHub Actions for CI/CD, Docker for containerization, and Nginx as a reverse proxy. This stage involved creating a robust deployment pipeline that allowed us to manage multiple environments efficiently.

Prerequisites
To follow along, before diving into the specifics, ensure you have:

A Next.js application.
Docker and Docker Compose installed on your server.
A Linux server configured for hosting Docker containers.
Nginx installed and configured for reverse proxy.
GitHub repository with configured secrets for deployment.

Step 1

Preparing the Deployment Script
We created a deployment script team_deploy.sh located in scripts/team_deploy/. This script was crucial for automating the deployment process for each environment.

team_deploy.sh

#!/bin/bash

set -e

# Check if the team name and port number are provided
if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Error: Team name and port number are required."
  echo "Usage: $0 [team name] [port]"
  exit 1
fi

TEAM_NAME=$1
export PORT=$2

# Navigate to the repository root and pull the latest changes
cd "$(git rev-parse --show-toplevel)"
git pull origin dev

# Pull the latest Docker image
docker pull docker-image:dev

# Deploy the application using Docker Compose
docker compose --project-name $TEAM_NAME -f docker/team-deploy/docker-compose.yml up -d

Explanation:

Check Inputs: Ensures both the team name and port number are provided. If not, the script exits with an error. Navigate to Repository: Uses git rev-parse to find the top-level directory of the repository and updates it with git pull.
Pull Docker Image: Retrieves the latest Docker image from the registry.
Deploy with Docker Compose: Uses Docker Compose to deploy the container, specifying a project name based on the team.

Step 2

Configuring the GitHub Actions Workflow
We set up a GitHub Actions workflow to automate the integration and deployment process. The integration workflow is triggered on every pull request while the deployment workflow was triggered upon the completion of the build and push workflow for docker images gotten from the marketplace. It used the appleboy/ssh-action to execute the deployment script on the server.

.github/workflows/build-lint-test.yml

name: Build, Lint and Test

on:
  pull_request

jobs:
  build_lint_test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "20"

      - name: Cache pnpm modules
        uses: actions/cache@v3
        with:
          path: ~/.pnpm-store
          key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
          restore-keys: |
            ${{ runner.os }}-pnpm-

      - name: Install pnpm
        uses: pnpm/action-setup@v4
        with:
          version: 9

      - name: Install dependencies
        run: pnpm install

      - name: Lint code
        run: pnpm lint

      - name: Build email
        run: pnpm email:build

      - name: Build project
        run: pnpm build

      - name: Run tests
        run: pnpm run test:ci

.github/workflows/deploy.yml

name: Team Deployment

on:
  workflow_run:
    workflows: ["Build and Push"]
    types:
      - completed

jobs:
  team-1:
    if: github.event.repository.fork == false
    runs-on: ubuntu-latest

    environment:
      name: "team-1"
      url: ${{ vars.URL }}

    steps:
      - name: Deploy to team-1 environment
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
          script: |
            cd repo
            ./scripts/team_deploy.sh team-1 ${{ vars.PORT }}

  ---

  team-7:
    if: github.event.repository.fork == false
    runs-on: ubuntu-latest

    environment:
      name: "team-7"
      url: ${{ vars.URL }}

    steps:
      - name: Deploy to team-7 environment
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
          script: |
            cd repo
            ./scripts/team_deploy.sh team-7 ${{ vars.PORT }}

Explanation:

Trigger: The workflow is triggered when the “Build and Push” workflow completes.
Checkout Code: Retrieves the latest code from the repository.
Deploy: Uses the appleboy/ssh-action to SSH into the server and execute the team_deploy.sh script with the environment name and port.

With the implementation of Github Environments we were able to set up multiple environments that corresponds to multiple users in our server. This approach allows each team-deploy job to target individual users on our server enabling the multiple deployment of the next.js application. Big thanks to my mentor Destiny for his brilliance on this.

Step 3

Docker Compose Configuration
We needed just one Docker Compose file to handle the deployment for each environment. This file specified the Docker image and configuration for the application.

docker-compose.yml

version: '3'
services:
  web:
    image: docker-image:dev
    ports:
      - "${PORT}:80"
    volumes:
      - .env:/app/.env

Explanation:
This docker-compose is what is being referenced by the script team_deploy.sh. It uses the exposed port passed into the script as a parameter as the listening port on the host system for the docker container. Now to connect each version of the same frontend to different backend apis, we mount a .env file not being tracked by github into each container. The .env contains the api url and other sensitive configuration strings necessary for each frontend to have unique backends.

Step 4

Setting Up Nginx as a Reverse Proxy
Nginx was configured to route traffic to the appropriate container based on the environment.

/etc/nginx/sites-available/team-1

    server {
        listen 80;
        server_name  team-1.deployment.com;
        location / {
            proxy_pass http://localhost:<team-1-port>;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

Explanation:

This nginx configuration would be replicated across the different teams in your environment with the only difference being the url and port the team's container is listening on the host system.

Step 5

Implementing Docker Cleanup

To keep the server clean, we set up a cron job to periodically remove unused Docker images.

Create a Cron Job

sudo crontab -e
Add the following line to run docker system prune every 2 hours:
bash
Copy code
0 */2 * * * /usr/bin/docker system prune -af

Explanation:

Cron Job: Runs the docker system prune -af command every 2 hours to remove unused containers, networks, and images.

Conclusion
In Stage Four, we successfully deployed the Next.js application across multiple environments using a structured approach involving GitHub Actions, Docker, and Nginx. The deployment script and GitHub Actions workflow automated the process, while Docker Compose and Nginx ensured smooth and efficient service management. The implementation of regular Docker cleanup maintained server performance and reliability.

This stage demonstrated the power of automation in modern deployment practices and highlighted the importance of managing multiple environments effectively. As we advance in our journey, these practices will serve as a foundation for handling even more complex deployment scenarios.

Thank you for following along with this stage of my HNG journey. Each stage has brought its own set of challenges and learning opportunities, and I look forward to continuing this journey and exploring new horizons in deployment and automation 🚀.

My HNG Journey. Stage Three: Building a Robust Messaging System with FastAPI and RabbitMQ/Celery for Asynchronous Email Handling

Ravencodess — Mon, 15 Jul 2024 00:34:33 +0000

Introduction

My only encounter with a message broker service until this point was when I read about Amazon MQ in preparation for my AWS Solutions Architect Associate examination. Imagine my utter confusion when I read the project for stage 3.
In this stage of my HNG journey, I was required to build a project that combines the power of FastAPI and Celery to create a robust messaging system. This system not only handles user requests efficiently but also leverages asynchronous task processing for seamless email sending. Throughout this article, I'll walk you through the key components and steps involved in developing this application, highlighting the integration of RabbitMQ as the message broker and utilizing SMTP for email delivery. By the end, you'll have a comprehensive understanding of how to build and deploy a similar system.

Here are the full requirements for this task:

Install RabbitMQ and Celery on your local machine.
Set up a Python application with the following functionalities: An endpoint that can accept two parameters: ?sendmail and ?talktome.
Endpoint Functionalities: ?sendmail: When this parameter is passed, the system should send an email using SMTP to the specified email address. (e.g., ?sendmail=mail@example.com). Use RabbitMQ/Celery to queue the email sending task. Ensure the email-sending script retrieves and executes tasks from the queue. ?talktome: When this parameter is passed, the system should log the current time to /var/log/messaging_system.log.
Nginx Configuration: Configure Nginx to serve your Python application. Ensure proper routing of requests to the application.
Endpoint Access: Use ngrok or a similar tool to expose your local application endpoint for external access.

Prerequisites

Python 3.8+
SMTP Email Account

Let's Get Started

Step 1

RabbitMQ and Celery Local Installation
For this project, I'll be using a Vagrant box on my laptop running Ubuntu 22.04 OS, but you can use any linux flavor you want.
We were instructed not to run RabbitMQ as a Docker container, so I ran the debian installation script from the official docs

#!/bin/sh

sudo apt-get install curl gnupg apt-transport-https -y

## Team RabbitMQs main signing key
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
## Community mirror of Cloudsmith: modern Erlang repository
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
## Community mirror of Cloudsmith: RabbitMQ repository
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null

## Add apt repositories maintained by Team RabbitMQ
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
## Provides modern Erlang/OTP releases
##
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

# another mirror for redundancy
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

## Provides RabbitMQ
##
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main

# another mirror for redundancy
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF

## Update package indices
sudo apt-get update -y

## Install Erlang packages
sudo apt-get install -y erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

## Install rabbitmq-server and its dependencies
sudo apt-get install rabbitmq-server -y --fix-missing

To confirm if RabbitMQ is installed correctly, run this command

sudo rabbitmqctl status

Next, we need to install Celery. Before that, let's create a Python virtual environment (venv) to better handle pip dependencies across our local machine.
Ensure the venv package is installed then create a venv

sudo apt install python3.10-venv
python3 -m venv env
# start up the virtual environment
source env/bin/activate

We can now observe an (env) block before our command prompt

To install celery, run

pip install celery
# confirm celery installation
celery --version

Step 2

Python Application Setup
Before we start writing our python application, let's understand celery and RabbitMQ and the role they will be playing in our application.

RabbitMQ is a message broker software that facilitates communication between different applications or components within a system. It acts as an intermediary that can receive messages from producers (senders) and deliver them to consumers (receivers). In the context of this application, Celery acts as both the producer and the consumer of the messages sent to the message broker.
Celery is an open-source distributed task queue system for Python, designed to handle asynchronous and scheduled tasks in a reliable and scalable manner. It enables you to run tasks concurrently. We send tasks to the queue as a producer using the .delay() method and then a celery worker consumes and executes the task in a celery worker process.

Let's use a short demo to see how Celery and RabbitMQ work together.

Create two files, main.py and tasks.py
tasks.py

from celery import Celery
from time import sleep

celery = Celery(
    "tasks",
    broker="pyamqp://guest:guest@localhost:5672//"
)

@celery.task
def reverse(text):
    sleep(5)
    print(text[::-1])


def echo(text):
    print(text)

reverse.delay('first function')

echo('second function')

This script initializes a celery object that uses RabbitMQ as the broker, the string "pyamqp://guest:guest@localhost:5672//" connects celery to RabbitMQ running locally on port 5672 using the default login username and password called guest.The script then calls two functions, reverse() and echo() but sent the reverse function to celery using the .delay() method . The first function takes in a string, sleeps for 5 seconds and then return the reversed string. The second function just repeats the string passed into it.

Run this script using

python3 tasks.py

Immediately the string passed into the second function is printed to the terminal and the first function doesn't return anything to the terminal

To see the outcome of the first function, open a new terminal, P.S. Ensure you enter the same virtual environment as the previous terminal.

celery -A tasks worker --loglevel=info

A celery worker starts up, connects to RabbitMQ, waits for 5 seconds then prints out the reversed string passed into the first function

Now we've observed how celery enables asynchronous programming by utilizing task queues, let's proceed with the python application. We would later see how to make use of a backend with celery so as to get the output of the consumed task in the celery worker process and use it in our application.

We need to first install the necessary dependencies for this application like FastAPI for building APIs and uvicorn for running our backend server.

 pip install fastapi uvicorn

Next we need to make sure our log file exists and the correct permissions are set in order not to encounter any permission errors. My user is vagrant, so I'll give vagrant ownership of the file.

sudo touch /var/log/messaging_system.log
sudo chown vagrant:vagrant /var/log/messaging_system.log

create a .env file that contains your email address and smtp_password. If you are using gmail, you can get your password using app_passwords

.env

email_sender=your_email
email_password=your_password

Let's write our email sending application.

tasks.py

from celery import Celery
from dotenv import load_dotenv
import os
from email.message import EmailMessage
import smtplib

# Load environment variables from a .env file
load_dotenv()

# Initialize a Celery instance with a broker and backend
celery = Celery(
    "tasks",
    broker="pyamqp://guest:guest@localhost:5672//",
    backend="rpc://"
)

# Define a Celery task to send an email
@celery.task
def send_mail(email):
    # Retrieve email sender and password from environment variables
    email_sender = os.getenv("email_sender")
    email_password = os.getenv("email_password")
    # Set the recipient email address
    email_receiver = email

    # Define the email subject and body
    subject = "Hello From HNG"
    body = "This is a test email"

    # Create an EmailMessage object and set its content
    em = EmailMessage()
    em["From"] = email_sender
    em["To"] = email_receiver
    em["Subject"] = subject
    em.set_content(body)

    try:
        # Establish a secure connection to the SMTP server
        with smtplib.SMTP_SSL('smtp.gmail.com', 465) as server:
            # Log in to the SMTP server using the senders credentials
            server.login(email_sender, email_password)
            # Send the email
            server.sendmail(email_sender, email_receiver, em.as_string())
        # Return a success message if the email is sent
        return f"Email sent to {email_receiver}"
    except Exception as e:
        # Return an error message if the email fails to send
        return f"Failed to send email to {email_receiver}: {e}"

This code:

Uses the backend="rpc://" in Celery configuration which refers to the result backend. This setting determines where Celery stores the results of tasks after they are executed. The rpc:// backend in Celery uses RabbitMQ to store task results. This is implemented using RabbitMQ's Remote Procedure Call (RPC) mechanism.
Uses celery tasks to create a send_mail asynchronous function that composes a test email which utilizes pythons' smtplib library to send secure emails to the recipient.
The function takes in a recipient email as a parameter which would passed into it from main.py

main.py

from fastapi import FastAPI
from typing import Optional
from datetime import datetime
from tasks import send_mail
from fastapi.responses import PlainTextResponse

# Initialize the FastAPI application
app = FastAPI()

# Function to log events to a file
def logger(event):
    # Open the log file in append mode
    with open("/var/log/messaging_system.log", "a") as log_file:
        # Write the event with a timestamp to the log file
        log_file.write(f"{datetime.now()}: {event}\n")

# Define a GET endpoint at the root URL
@app.get("/")
def test(sendmail: Optional[str] = None, talktome: Optional[str] = None):
    # Initialize the response dictionary
    response = {}

    # If 'sendmail' query parameter is provided
    if sendmail is not None:
        # If 'sendmail' is an empty string
        if sendmail == "":
            response["sendmail"] = "no mail provided"
        else:
            # Log the sending mail action
            logger(f"Sending mail to {sendmail}  .....")
            # Call the asynchronous send_mail task
            result = send_mail.delay(sendmail)
            # Log the result of the send_mail task
            logger(f"{result.get()}")
        # Indicate that the action has been logged
        response["sendmail"] = "logged action to /logs"

    # If 'talktome' query parameter is provided
    if talktome is not None:
        # Log the talktome message
        logger(talktome)
        # Indicate that the action has been logged
        response["talktome"] = "logged action to /logs"

    # If neither 'sendmail' nor 'talktome' query parameters are provided
    if sendmail is None and talktome is None:
        response["Default"] = "no parameters provided"

    # Return the response dictionary
    return response

# Define a GET endpoint to retrieve logs
@app.get("/logs")
def logs():
    # Open the log file in read mode
    with open("/var/log/messaging_system.log", "r") as log_file:
        # Read the entire content of the log file
        log_content = log_file.read()
        # Return the log content as plain text response
        return PlainTextResponse(content=log_content)

This code:

Imports the necessary dependencies needed for the script
Creates a logging function that prints the current time and event to the log file we created previously
Initializes FastAPI
Creates two endpoints, one for the root / and one for the logs /logs. The / endpoint uses the send_mail function imported from tasks.py and calls it asynchronously using .delay(), but because we are using a backend in tasks.py we can retrieve the results of the asynchronous function using the .get() method. The ?sendmail query parameter sends a mail to the provided email address. If ?talktome query parameter is passed, It prints the current time to the log file.

Step 3

Creating systemd services for celery and uvicron

Our application is ready and we can start celery using

celery -A tasks worker --loglevel=info

Start uvicorn for our backend application server using

uvicorn main:app --reload

P.S if you wish to access the application from the browser add the --host 0.0.0.0 to the uvicorn command

But this method will require us to open and maintain multiple terminal sessions. Instead let's run the applications as systemd services in the background.
Open /etc/systemd/system/uvicorn.service and /etc/systemd/system/celery.service with sudo permissions.

/etc/systemd/system/uvicorn.service

[Unit]
Description=Uvicorn instance to serve FastAPI application
After=network.target

[Service]
User=vagrant
Group=www-data
WorkingDirectory=/home/vagrant/message-broker-stage-3/app
ExecStart=/home/vagrant/.local/bin/uvicorn main:app --reload --host 0.0.0.0 --port 8000

[Install]
WantedBy=multi-user.target

/etc/systemd/system/celery.service

[Unit]
Description=Celery Service
After=network.target

[Service]
User=vagrant
Group=www-data
WorkingDirectory=/home/vagrant/messaging-stage-3/app
ExecStart=/home/vagrant/.local/bin/celery -A tasks worker --loglevel=info

[Install]
WantedBy=multi-user.target

Ensure you input the correct user, file path and location for the bin directory of your uvicorn and celery binaries

you can then start and enable the two services in the background using

sudo systemctl start uvicorn.service
sudo systemctl enable uvicorn.service

sudo systemctl start celery.service
sudo systemctl enable celery.service

Ensure they are running properly by using

sudo systemctl status uvicorn.service
sudo systemctl status celery.service

Step 4

Configure Nginx reverse proxy

In this step, we will use Nginx to route localhost to our application running on port 8000.

If you don't have nginx installed. You can install using

sudo apt update
sudo apt install nginx nginx-core

Open the nginx.conf and replace the default http{} block

nginx.conf

http {
    # Basic HTTP settings
    server_tokens off;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;


    server {
        listen 80;
        server_name localhost;

        location / {
            proxy_pass http://localhost:8000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }

    }
}

Check if the configuration is correct and restart Nginx using

sudo nginx -t
sudo systemctl restart nginx.service

Running curl localhost?talktome should respond with a JSON message informing you that the action has been logged to /logs.

Step 5

Use Ngrok to expose api
The final step requires us to use ngrok to expose our API securely to the public.
Set up an ngrok free account and obtain a auth token by following the official ngrok docs

Finally we can expose our API running on localhost:80 using

ngrok http 80

Test the email functionality by adding ?sendmail=your_email

You can also test the ?talktome query parameter access the logs using /logs path.

Conclusion

We have successfully developed a Python application that utilizes RabbitMQ and Celery as a message broker and task queue executor. This application deploys a public API endpoint using ngrok, providing functionality for sending emails and printing logs.

By integrating these technologies, we have created a robust and scalable messaging system. This project not only demonstrates the power of asynchronous task processing with Celery and RabbitMQ but also highlights the simplicity and efficiency of using FastAPI for building web applications. Deploying the API with ngrok allows for quick public exposure and testing, making it an excellent tool for development and debugging.

In conclusion, this project serves as a solid foundation for building more complex systems that require reliable message handling and task execution. Whether you are looking to expand its capabilities or use it as a learning experience, this messaging system showcases the potential of modern Python web development.

The tasks are getting longer and harder as the stages go by. I can't imagine what is in store in stage 4, But one thing is certain, I'll give it the best I've got.
If you made it this far, Thank you, Thank you, Thank you. ♥
Until next time.
Happy Programming 🚀

My HNG Journey. Stage Two: Containerization and Deployment of a Three tier application Using Docker and Nginx Proxy Manager

Ravencodess — Mon, 08 Jul 2024 16:37:11 +0000

Introduction

This stage brought on a task that at first glance seems easy and straightforward, but when the added requirements were introduced, the complexity grew and the challenge became harder. The task instructs us to containerize a three tier application on a single server and use a proxy manager like nginx to configure reverse proxying to ensure the frontend and backend can be served from the same port. That's not all. It gets more complex.

Here are the full requirements for completing this tasks:

Ensure the application runs locally before writing Dockerfiles
Configure the Frontend and Backend to listen on port 80
Obtain a domain name for the project
Write Dockerfiles to containerize the frontend and backend
Install adminer to enable database manager through its GUI
Configure Nginx proxy manager to handle reverse proxying and setup SSL certificates

Let's get started

Prerequisites

A virtual machine running Ubuntu
Basic Level Understanding of the Linux CLI

Step 1

Clone the repo
First we have to clone the repository from Github



git clone https://github.com/hngprojects/devops-stage-2
cd devops-stage-2

Step 2

Configure the backend
The frontend of this application depends on the backend for full functionality so we will begin by configuring the backend.



cd backend

Dependencies
The backend depends on a postgresQL database, It would also require poetry to be installed before starting up

Installing Poetry

To install Poetry, follow these steps:



curl -sSL https://install.python-poetry.org | python3 -

Add Poetry to your PATH if it's not automatically added:



# Example for Bash shell
export PATH="$HOME/.poetry/bin:$PATH" >> ~/.bashrc
source ~./bashrc
poetry --version

Replace $HOME/.poetry/bin with the appropriate path where Poetry binaries are installed if different on your system. This ensures you can run Poetry commands from any directory in your terminal session.

Install dependencies using Poetry:



poetry install

Setup PostgreSQL:
Follow these steps to install PostgreSQL on Linux and configure a user named app with password my_password and a database named app. Give all permissions of the app database to the app user.

Install PostgreSQL on Linux (example for Ubuntu):



sudo apt update
sudo apt install postgresql postgresql-contrib

Switch to the PostgreSQL user and access the PostgreSQL



sudo -i -u postgres
psql

Create a user app with password my_password:



CREATE USER app WITH PASSWORD 'my_password';

Create a database named app and grant all privileges to the app user:



CREATE DATABASE app;
\c app
GRANT ALL PRIVILEGES ON DATABASE app TO app;
GRANT ALL PRIVILEGES ON SCHEMA public TO app;

Exit the PostgreSQL shell and switch back to your regular user.



\q
exit

Set database credentials
Edit the PostgreSQL environment variables located in the .env file. Make sure the credentials match the database credentials you just created.



---
POSTGRES_SERVER=localhost
POSTGRES_PORT=5432
POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_PASSWORD=my_password

Set up the database with the necessary tables:



poetry run bash ./prestart.sh

Run the backend server and make it accessible on all network interfaces:



poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Step 3

Configure the frontend
Open up a new terminal.
P.S. We can split the terminal session using tmux or run it as a system service, but to keep things fairly simple, we would leave the backend running in one terminal and open another terminal for the frontend.



cd devops-stage-2/frontend

Dependencies
The frontend was built with Nodejs and npm for dependency management.



sudo apt update
sudo apt install nodejs npm

Install dependencies:



npm install

Run the fronted server and make it accessible from all network interfaces:



npm run dev -- --host

Accessing the application using curl:



curl localhost:5173

Step 4

Accessing the UI

Open your browser and navigate to:



http://<your_server_IP>:5173

Enable login access from the UI:
The login credentials can be found in the .env located in the backend folder



---
FIRST_SUPERUSER=devops@hng.tech
FIRST_SUPERUSER_PASSWORD=devops#HNG11

If we try login in now we would be met with a network error.

Looking through the developer tools we can see that connecting to the backend on http://localhost:8000 was refused. This is because we are using a remote server and localhost in our browser's context means our personal computer. So to properly route the browser to the remote server running the application. we will have to Change the VITE_API_URL variable in the frontend .env file:



VITE_API_URL=http://<your_server_IP>:8000

If we try to login now we are met with a new error called CORS which stands for Cross-origin resource sharing.

Basically, our backend doesn't recognise the origin of the request which is coming from our server's IP, so we need to tell our backend to accept request coming from that particular IP address.

In our backed .env file we need to add http://<your_server_IP>:5173 to the end of the string of allowed IPs



BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,https://localhost,https://localhost:5173,http://<your_server_IP>:5173"

Now If we try one more time to login.

We successfully setup the application locally.
We can also access the swagger API as well as the documentation paths using http://<your_server_IP>:8000/doc and http://<your_server_IP>:8000/redoc respectively.

Step 5

Containerizing the application
Now we need to repeat the entire process, but this time, We would utilize Docker containers. we will start by writing Dockerfiles for both frontend and backend and then move to the project's root directory and configure a docker compose file that will run and configure:

The Frontend and Backend
The postgres database the backend depends on
Adminer
Nginx proxy Manager

Let's start by writing the Dockerfile for the backend application



cd devops-stage-2/backend
vim Dockerfile



# Use the latest official Python image as a base
FROM python:latest

# Install Node.js and npm
RUN apt-get update && apt-get install -y \
    nodejs \
    npm

# Install Poetry using pip
RUN pip install poetry

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Install dependencies using Poetry
RUN poetry install

# Expose the port FastAPI runs on
EXPOSE 8000

# Run the prestart script and start the server
CMD ["sh", "-c", "poetry run bash ./prestart.sh && poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload"]

This repeats the entire process we carried out locally all in one file.

Now let's set up the frontend.



cd devops-stage-2/frontend
vim Dockerfile



# Use the latest official Node.js image as a base
FROM node:latest

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Install dependencies
RUN npm install

# Expose the port the development server runs on
EXPOSE 5173

# Run the development server
CMD ["npm", "run", "dev", "--", "--host"]

Again, this simply repeats the process we carried out to run the frontend locally.

Step 6

Docker compose setup
Navigate to the project root directory and create a docker-compose.yml file



cd devops-stage-2/
vim docker-compose.yml

Copy this configuration into it



version: '3.8'

services:
  backend:
    build:
      context: ./backend
    container_name: fastapi_app
    ports:
      - "8000:8000"
    depends_on:
      - db
    env_file:
      - ./backend/.env

  frontend:
    build:
      context: ./frontend
    container_name: nodejs_app
    ports:
      - "5173:5173"
    env_file:
      - ./frontend/.env

  db:
    image: postgres:latest
    container_name: postgres_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - ./backend/.env

  adminer:
    image: adminer
    container_name: adminer
    ports:
      - "8080:8080"

  proxy:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx_proxy_manager
    ports:
      - "80:80"
      - "443:443"
      - "81:81"
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
      - backend
      - frontend
      - adminer

volumes:
  postgres_data:
  data:
  letsencrypt:

Breakdown of the docker-compose.yml File
Here's an explanation of each section in the provided docker-compose.yml file:

Services

Services are the containers that make up the application. Each service runs one image and can define volumes and networks. Each container can connect to any container in the same network using the service name.

Backend Service



backend:
  build:
    context: ./backend
  container_name: fastapi_app
  ports:
    - "8000:8000"
  depends_on:
    - db
  environment:
    POSTGRES_SERVER: ${POSTGRES_SERVER}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

build.context: Specifies the build context, pointing to the ./backend directory which contains the Dockerfile for building the FastAPI backend service.
container_name: Sets the container name to fastapi_app.
ports: Maps port 8000 on the host to port 8000 in the container.
depends_on: Ensures the db service is started before the backend service.
environment: Injects environment variables from the .env file, used by the FastAPI application to connect to the PostgreSQL database.

Frontend Service



frontend:
  build:
    context: ./frontend
  container_name: nodejs_app
  ports:
    - "5173:5173"
  environment:
    VITE_API_URL: ${VITE_API_URL}

build.context: Points to the ./frontend directory for building the Node.js frontend service.
container_name: Names the container nodejs_app.
ports: Maps port 5173 on the host to port 5173 in the container.
environment: Injects the VITE_API_URL environment variable from the .env file, used by the frontend application to connect to the backend API.

Database Service



db:
  image: postgres:latest
  container_name: postgres_db
  ports:
    - "5432:5432"
  volumes:
    - postgres_data:/var/lib/postgresql/data
  environment:
    POSTGRES_USER: ${POSTGRES_USER}
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    POSTGRES_DB: ${POSTGRES_DB}

image: Uses the latest PostgreSQL image from Docker Hub.
container_name: Names the container postgres_db.
ports: Maps port 5432 on the host to port 5432 in the container, which is the default port for PostgreSQL.
volumes: Mounts a Docker volume postgres_data to persist database data.
environment: Sets database-related environment variables from the .env file for initializing PostgreSQL.

Adminer Service



adminer:
  image: adminer
  container_name: adminer
  ports:
    - "8080:8080"

image: Uses the Adminer image, a database management tool.
container_name: Names the container adminer.
ports: Maps port 8080 on the host to port 8080 in the container for accessing the Adminer web interface.

Proxy Service



proxy:
  image: jc21/nginx-proxy-manager:latest
  container_name: nginx_proxy_manager
  ports:
    - "80:80"
    - "443:443"
    - "81:81"
  environment:
    DB_SQLITE_FILE: "/data/database.sqlite"
  volumes:
    - ./data:/data
    - ./letsencrypt:/etc/letsencrypt
  depends_on:
    - db
    - backend
    - frontend
    - adminer

image: Uses the latest Nginx Proxy Manager image.
container_name: Names the container nginx_proxy_manager.
ports: Maps ports 80, 443, and 81 on the host to the same ports in the container for HTTP, HTTPS, and the Nginx Proxy Manager admin interface.
environment: Sets the environment variable for the SQLite database location.
volumes: Mounts the data directory for storing proxy manager data and the letsencrypt directory for SSL certificates.
depends_on: Ensures the db, backend, frontend, and adminer services are started before the proxy service.

Volumes



volumes:
  postgres_data:
  data:
  letsencrypt:

Defines named volumes to persist data across container restarts.

Step 7

Domain Setup
We need to setup domains and subdomains for the frontend, adminer service and Nginx proxy manager.
Remember we are required to route port 80 to both frontend and backend:

domain - Frontend
domain/api - Backend
db.domain - Adminer
proxy.domain - Nginx proxy manager

If you don't have a Domain name, you can acquire a subdomain at AfraidDNS. That's where i acquired the domain I used for this project. Ensure you route all the required domains above to the server your application is running on.

Step 8

Routing domains using Nginx proxy manager
We now have everything set up, we can run docker-compose up -d to get our application up and running. We would need to install Docker and Docker-compose first.

Install Docker
Update the package list:



sudo apt-get update

Install required packages:



sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

Add Docker’s official GPG key:



curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the Docker repository to APT sources:



sudo add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable"

Update the package list again:



sudo apt-get update

Install Docker:



sudo apt-get install docker-ce

Verify that Docker is installed correctly:



sudo systemctl status docker

Install Docker Compose
Download the latest version of Docker Compose:



sudo curl -L "https://github.com/docker/compose/releases/download/$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")')" /usr/local/bin/docker-compose

Apply executable permissions to the binary:



sudo chmod +x /usr/local/bin/docker-compose

Verify that Docker Compose is installed correctly:



docker-compose --version

Post-Installation Steps for Docker
Manage Docker as a non-root user:
Create the docker group if it doesn't already exist:



sudo groupadd docker

Add your user to the docker group:



sudo usermod -aG docker $USER

Now we can start up the application.
Ensure you are in the project root directory



cd devops-stage-2

Start the application



docker-compose up -d

If you get a permission denied error, run is as superuser



sudo docker-compose up -d

Running curl localhost gives us a HTML response that Nginx proxy manager is successfully installed

Step 9

Reverse Proxying and SSL setup with Nginx proxy manager
Access the Proxy manager UI by entering http://:81 in your browser, Ensure that port is open in your security group or firewall.

Email: admin@example.com
Password: changeme

Click on Proxy host and setup the proxy for your frontend and backend
Map your domain name to the service name of your frontend and the port the container is listening on Internally.

Click on the SSL tab and request a new certificate

Now to configure the frontend to route api requests to the backend on the same domain, Click on Advanced and paste this configuration



location /api {
    proxy_pass http://backend:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /docs {
    proxy_pass http://backend:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /redoc {
    proxy_pass http://backend:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Repeat the same process for

db.domain: to route to your adminer service on port 8080
proxy.domain: to route to the proxy service UI on port 81

You don't need to do the advanced setup on the db and proxy domain

Step 10

Setup Adminer
Access the adminer web interface on db.<your_domain>.com

Step 11

Setup Frontend Login
Access your frontend on <your_domain>

Before you login, make sure to change change the API_URL in your frontend .env to the name of your domain



VITE_API_URL=https://<your_domain>

You would need to run docker-compose up -d --build to enable the changes to take effect

Your login should be successful now

Conclusion

We have now successfully:

Configured and tested the full stack application locally
Containerized the application
Setup Docker compose
Configured Adminer for Database management
Configured Reverse Proxying with Nginx Proxy Manager
Setup SSL certificates for our domains

Thank you for reading ♥
Happy Proxying! 🚀

My HNG Journey. Stage One: Automating Linux User and Group Management using a Bash Script

Ravencodess — Tue, 02 Jul 2024 23:07:00 +0000

Introduction

With a new HNG stage, comes a new and slightly difficult task.
This is going to be a long read, so I'll keep the introduction short and just get into it

The source code can be found on my GitHub repo

Requirements

In this task, we will be writing a Bash script that takes in one argument which will be a TXT file that contains a list of usernames and group names. For example;

john; admin, developer, tester
kourtney; hr, product

The script must accomplish the following tasks;

Create Users and groups based on the file content, Usernames and user groups are separated by semicolon ";"- Ignore whitespace.
Each User must have a personal group with the same group name as the username, this group name will not be written in the text file.
A user can have multiple groups, each group delimited by comma ","
The file /var/log/user_management.log should be created and contain a log of all actions performed by your script.
The file /var/secure/user_passwords.txt should be created and contain a list of all users and their passwords delimited by comma, and only the file owner should be able to read it.
Handle errors gracefully.

Prerequisites

Basic understanding of Linux CLI and Bash Scripting

Step 1

Handle Command Line Arguments and Input File Errors
We want to ensure the script is being passed only one argument and that argument is indeed a file.
If both cases are not true, we want to print an error to the terminal.

#!/bin/bash
# Check if the correct number of command line arguments is provided
if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <user_info_file>"
    exit 1
fi

# Assign the file name from the command line argument
input_file=$1

# Check if the input file exists
if [ ! -f "$input_file" ]; then
    echo "Error: File $input_file not found."
    exit 1
fi

Step 2

Create a Logging Function
One of our requirements states that we log all our actions to /var/log/user_management.log, let's create a function to handle that, and move it to the top of our script.

#!/bin/bash
# Function to log actions to /var/log/user_management.log
log_action() {
    local log_file="/var/log/user_management.log"
    local timestamp=$(date +"%Y-%m-%d %T")
    local action="$1"
    echo "[$timestamp] $action" | sudo tee -a "$log_file" > /dev/null
}
---

Step 3

Create Log and Password File
Now that we have our logging function defined we can log any action that happens during the script execution.
Next we need to create the log file itself and also create the password file, and also assign permissions to allow only the file owner to access it.

---
# Check and create the log_file if it does not exist
log_file="/var/log/user_management.log"

if [ ! -f "$log_file" ]; then
    # Create the log file
    sudo touch "$log_file"
    log_action "$log_file has been created."
else
    log_action "Skipping creation of: $log_file (Already exists)"
fi

# Check and create the passwords_file if it does not exist
passwords_file="/var/secure/user_passwords.txt"

if [ ! -f "$passwords_file" ]; then
    # Create the file and set permissions
    sudo mkdir -p /var/secure/
    sudo touch "$passwords_file"
    log_action "$passwords_file has been created."
    # Set ownership permissions for passwords_file
    sudo chmod 600 "$passwords_file"
    log_action "Updated passwords_file permission to file owner"
else
    log_action "Skipping creation of: $passwords_file (Already exists)"
fi

Step 4

Read Input File
At this point the script has validated the command line argument and has confirmed it is a file, now it's time to loop through the file and read it line by line. We can accomplish this using a while loop. This will run until the last line of the input file. All our user and groups creation logic will be done inside this while loop.

---
while IFS=';' read -r username groups; do
---
done < "$input_file"

This while loop reads each line of the file, uses an internal field separator (IFS) to separate the line based on the value assigned to the IFS variable and then assigns values to variables: username and groups. For example, a line like dora; design, marketing would be read as; username=dora groups=design, marketing
The -r option ensures we don't treat backslashes '\' as escape characters

Step 5

Validate Username and Group Name
Our script can now read a user input file line by line, but first we must validate the strings that are passed into our $username and $group variables to ensure they comply with Unix naming standards. We can handle this logic by creating a validate_name function and move it to the top of our script

#!/bin/bash

# Function to validate username and group name
validate_name() {
    local name=$1
    local name_type=$2  # "username" or "groupname"

    # Check if the name contains only allowed characters and starts with a letter
    if [[ ! "$name" =~ ^[a-z][a-z0-9_-]*$ ]]; then
        log_action "Error: $name_type '$name' is invalid. It must start with a lowercase letter and contain only lowercase letters, digits, hyphens, and underscores."
        return 1
    fi

    # Check if the name is no longer than 32 characters
    if [ ${#name} -gt 32 ]; then
        log_action "Error: $name_type '$name' is too long. It must be 32 characters or less."
        return 1
    fi

    return 0
}
---

This function runs two checks;

It checks if the name complies with naming standards using a Regex expression (name begins with a lowercase letter and name must only include lowercase letters, numbers, dashes and underscores)
It makes sure the name is not longer than 32 characters. Finally it logs all action into the log file created earlier

Step 6

Check if User or Group Already Exists
After validating the string passed into our variables, we also need to run a check to validate if these names already exist on the system. We don't want to create duplicate users or groups. We can achieve this by creating a user_exists and group_exists function and move it to the top of our script

#!/bin/bash
# Function to check if a user exists
user_exists() {
    local username=$1
    if getent passwd "$username" > /dev/null 2>&1; then
        return 0  # User exists
    else
        return 1  # User does not exist
    fi
}

# Function to check if a group exists
group_exists() {
    local group_name=$1
    if getent group "$group_name" > /dev/null 2>&1; then
        return 0  # Group exists
    else
        return 1  # Group does not exist
    fi
}
---

Step 7

Create User
Now it's time to use our while loop to begin creating users,
we will carry out these tasks in this step

String manipulation, which involves removing or collapsing white spaces
Call the validate_name and user_exists functions to ensure we are creating a valid and unique username
Generate a random password and assign it to the newly created user

Let's first define the generate_password function and place it alongside the functions we created earlier at the top of our script

---
# Function to generate a random password
generate_password() {
    openssl rand -base64 12
}
---

Now everything is in place to create a user, we will utilize the while loop we created in Step 4.

---
# Read the file line by line and process
while IFS=';' read -r username groups; do
    # Extract the user name
    username=$(echo "$username" | xargs)

    # Validate username
    if ! validate_name "$username" "username"; then
        log_action "Invalid username: $username. Skipping."
        continue
    fi

    # Check if the user already exists
    if user_exists "$username"; then
        log_action "Skipped creation of user: $username (Already exists)"
        continue
    else
        # Generate a random password for the user
        password=$(generate_password)

        # Create the user with home directory and set password
        sudo useradd -m -s /bin/bash "$username"
        echo "$username:$password" | sudo chpasswd

        log_action "Successfully Created User: $username"
    fi

    # Ensure the user has a group with their own name, This is the default behaviour in most linux distros
    if ! group_exists "$username"; then
        sudo groupadd "$username"
        log_action "Successfully created group: $username"
        sudo usermod -aG "$username" "$username"
        log_action "User: $username added to Group: $username"
    else
        log_action "User: $username added to Group: $username"
    fi
done < "$input_file"

Step 8

Create Group(s)
The next action to take is to create the groups for the user that was just created.
We need to also validate the group name and check if it already exists before creating it and adding our user into it. We need to form a group_array based on the content of the groups variable so we can loop through it and create a group for each name in the array.
Under the user creation logic, we can create groups with this.

while IFS=';' read -r username groups; do
---
# Extract the groups and remove any spaces
    groups=$(echo "$groups" | tr -d ' ')

    # Split the groups by comma
    IFS=',' read -r -a group_array <<< "$groups"

    # Create the groups and add the user to each group
    for group in "${group_array[@]}"; do
        # Validate group name
        if ! validate_name "$group" "groupname"; then
            log_action "Invalid Group name: $group. Skipping Group for user $username."
            continue
        fi

        # Check if the group already exists
        if ! group_exists "$group"; then
            # Create the group if it does not exist
            sudo groupadd "$group"
            log_action "Successfully created Group: $group"
        else
            log_action "Group: $group already exists"
        fi
        # Add the user to the group
        sudo usermod -aG "$group" "$username"
    done
done < "$input_file"

Step 9

Store Password Information in Secure Password File
Let's round up the script execution by setting proper home directory permissions and also sending username and password information to the passwords_file we created in Step 3.

---
# Set permissions for home directory
    sudo chmod 700 "/home/$username"
    sudo chown "$username:$username" "/home/$username"
    log_action "Updated permissions for home directory: '/home/$username' of User: $username to '$username:$username'"

    # Log the user created action
    log_action "Successfully Created user: $username with Groups: $username ${group_array[*]}"

    # Store username and password in secure file
    echo "$username,$password" | sudo tee -a "$passwords_file" > /dev/null
    log_action "Stored username and password in $passwords_file"
done < "$input_file"

Step 10

Putting it All Together
We've come to the end of the script, I did mention it was a long one 😁. But I enjoyed explaining every paragraph to you 🤗. If you want to discover amazing talents at HNG click here
Thank you for reading ♥

Here's the full script for your reference

#!/bin/bash

# Function to check if a user exists
user_exists() {
    local username=$1
    if getent passwd "$username" > /dev/null 2>&1; then
        return 0  # User exists
    else
        return 1  # User does not exist
    fi
}

# Function to check if a group exists
group_exists() {
    local group_name=$1
    if getent group "$group_name" > /dev/null 2>&1; then
        return 0  # Group exists
    else
        return 1  # Group does not exist
    fi
}

# Function to validate username and group name
validate_name() {
    local name=$1
    local name_type=$2  # "username" or "groupname"

    # Check if the name contains only allowed characters and starts with a letter
    if [[ ! "$name" =~ ^[a-z][a-z0-9_-]*$ ]]; then
        log_action "Error: $name_type '$name' is invalid. It must start with a lowercase letter and contain only lowercase letters, digits, hyphens, and underscores."
        return 1
    fi

    # Check if the name is no longer than 32 characters
    if [ ${#name} -gt 32 ]; then
        log_action "Error: $name_type '$name' is too long. It must be 32 characters or less."
        return 1
    fi

    return 0
}

# Function to generate a random password
generate_password() {
    openssl rand -base64 12
}

# Function to log actions to /var/log/user_management.log
log_action() {
    local log_file="/var/log/user_management.log"
    local timestamp=$(date +"%Y-%m-%d %T")
    local action="$1"
    echo "[$timestamp] $action" | sudo tee -a "$log_file" > /dev/null
}

# Check if the correct number of command line arguments is provided
if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <user_info_file>"
    exit 1
fi

# Assign the file name from the command line argument
input_file=$1

# Check if the input file exists
if [ ! -f "$input_file" ]; then
    echo "Error: File $input_file not found."
    exit 1
fi

# Check and create the log_file if it does not exist
log_file="/var/log/user_management.log"

if [ ! -f "$log_file" ]; then
    # Create the log file
    sudo touch "$log_file"
    log_action "$log_file has been created."
else
    log_action "Skipping creation of: $log_file (Already exists)"
fi

# Check and create the passwords_file if it does not exist
passwords_file="/var/secure/user_passwords.txt"

if [ ! -f "$passwords_file" ]; then
    # Create the file and set permissions
    sudo mkdir -p /var/secure/
    sudo touch "$passwords_file"
    log_action "$passwords_file has been created."
    # Set ownership permissions for passwords_file
    sudo chmod 600 "$passwords_file"
    log_action "Updated passwords_file permission to file owner"
else
    log_action "Skipping creation of: $passwords_file (Already exists)"
fi

echo "----------------------------------------"
echo "Generating Users and Groups"
echo "----------------------------------------"

# Read the file line by line and process
while IFS=';' read -r username groups; do
    # Extract the user name
    username=$(echo "$username" | xargs)

    # Validate username
    if ! validate_name "$username" "username"; then
        log_action "Invalid username: $username. Skipping."
        continue
    fi

    # Check if the user already exists
    if user_exists "$username"; then
        log_action "Skipped creation of user: $username (Already exists)"
        continue
    else
        # Generate a random password for the user
        password=$(generate_password)

        # Create the user with home directory and set password
        sudo useradd -m -s /bin/bash "$username"
        echo "$username:$password" | sudo chpasswd

        log_action "Successfully Created User: $username"
    fi

    # Ensure the user has a group with their own name, This is the default behaviour in most linux distros
    if ! group_exists "$username"; then
        sudo groupadd "$username"
        log_action "Successfully created group: $username"
        sudo usermod -aG "$username" "$username"
        log_action "User: $username added to Group: $username"
    else
        log_action "User: $username added to Group: $username"
    fi

    # Extract the groups and remove any spaces
    groups=$(echo "$groups" | tr -d ' ')

    # Split the groups by comma
    IFS=',' read -r -a group_array <<< "$groups"

    # Create the groups and add the user to each group
    for group in "${group_array[@]}"; do
        # Validate group name
        if ! validate_name "$group" "groupname"; then
            log_action "Invalid Group name: $group. Skipping Group for user $username."
            continue
        fi

        # Check if the group already exists
        if ! group_exists "$group"; then
            # Create the group if it does not exist
            sudo groupadd "$group"
            log_action "Successfully created Group: $group"
        else
            log_action "Group: $group already exists"
        fi
        # Add the user to the group
        sudo usermod -aG "$group" "$username"
    done

    # Set permissions for home directory
    sudo chmod 700 "/home/$username"
    sudo chown "$username:$username" "/home/$username"
    log_action "Updated permissions for home directory: '/home/$username' of User: $username to '$username:$username'"

    # Log the user created action
    log_action "Successfully Created user: $username with Groups: $username ${group_array[*]}"

    # Store username and password in secure file
    echo "$username,$password" | sudo tee -a "$passwords_file" > /dev/null
    log_action "Stored username and password in $passwords_file"
done < "$input_file"

# Log the script execution to standard output
echo "----------------------------------------"
echo "Script Executed Succesfully, logs have been published here: $log_file"
echo "----------------------------------------"

My HNG journey. Stage Zero: How to Deploy a Static Webpage Using Nginx

Ravencodess — Tue, 02 Jul 2024 17:59:23 +0000

Introduction

Over the past few years, HNG internships have always been talked about with a certain level of fear and respect. "A grueling and unforgiving experience", some would say.
This year, I have decided to give the DevOps track a try. 10 stages of tasks to be covered and I intend to document my entire journey.

Requirements

The requirements are straightforward;

The webpage must be hosted on a cloud-based virtual machine
The webpage must be publicly accessible
The webpage must utilize a web server like Nginx or Apache
The webpage must include The intern's name, Slack username and email address
The webpage must include a button that links back to the official HNG website

Prerequisites

Skills/Knowledge: Basic understanding of Docker, Nginx, and Linux CLI.

Tools/Software: Docker, Nginx, AWS.

Let's get started

Set Up Virtual Machine
I am going to be using AWS EC2 for this step but feel free to use any cloud platform of your choice.

I'm opting for this EC2 specification because it's a simple static web server:

Instance Name: My Web Server
AMI: Amazon Linux 2023 (Free Tier)
Instance Type: t2 micro
Storage: Default 8GB gp3
Networking: Default VPC and subnet
Security Groups: Allow HTTP, HTTPS access from anywhere, and SSH access from my IP

Make sure to create a new key-pair and save it securely in case you want to SSH into the instance.

Expand the Advanced Details panel, Scroll to the bottom to find User Data and the paste in this script.

#!/bin/bash
# Update software
yum update -y

# Install Docker
yum install docker -y
service docker start
usermod -aG docker ec2-user

# Install Git
yum install git -y

# Clone the repository
git clone https://github.com/Ravencodess/static-webpage-stage-0.git
cd static-webpage-stage-0

# Build the Docker image
docker build -t my-web-app .

# Run the container and map port 80 on the server to port 80 inside the container
docker run -d -p 80:80 my-web-app

This script pulls the application code from my public Github repo and launches the application on an Nginx-based docker container and maps the port 80 on the server to port 80 inside the container. This allows us to access the static webpage on our browser by simply accessing the public IP address of our server.

Let's take a closer look at the contents of the repository.

The HTML, CSS, and JS folders contain the necessary files, stylesheets and script needed to create a functional webpage and can be edited and configured anyhow you see fit.

I want to pay more detail to the contents of the Dockerfile and nginx.conf file

Dockerfile

The Dockerfile performs the following steps

It uses the latest nginx image from Docker Hub as the base image for the application.
It removes the default nginx.conf configuration file and replaces it with our modified conf file.
It then copies the html,css and javascript files into the /usr/share/nginx folder which Nginx would use to serve the web page.
Finally it documents that port 80 is exposed and starts the container.

nginx.conf

This file sets up Nginx to listen on localhost port 80 and sets up redirection for all static assets the webpage would need.

Your instance should have launched successfully and passed status check by now, locate the public IP address or DNS, and paste it in a new browser tab.

Your webpage should be live.

Happy Launching 🚀