Forem: Lionel♾️☁️

End to End CI/CD pipeline using GitHub Actions for Android Application

Lionel♾️☁️ — Fri, 13 Dec 2024 18:39:19 +0000

In this article, you will get a brief idea about how to create an End to End CI/CD Pipeline using GitHub Actions for an Android Application.

Project Source Code : LINK

Here, we will be covering some use cases like how to trigger one workflow from another workflow, how to run 2 jobs which depends upon each other, how to add public ip of GitHub Actions in security groups of Jfrog which is running in an EC2 , on port 8082 — so that GitHub Actions can access Jfrog to upload the .apk file into repository, how to integrate SonarQube , Teams with GitHub Actions , how to create a cron job in workflow, how to delete artifacts which are created during the workflow in GitHub Actions, how to clean caches in GitHub Actions which gets created every time your run a workflow.

Step by Step Process :

To create a workflow, go to Actions in your GitHub Repository and choose a template yml file or click “set up a workflow yourself” — for me I have chosen Android CI as my application is android application.

This basic template you will get to do CI part of Android Application.

Let me explain you some of the terms used in workflow file:

name : The name of the workflow as it will appear in the “Actions” tab of the GitHub repository. Like here it is “Android CI”

on: Specifies the trigger for this workflow. So here the workflow will be triggered when there is a push event in “main” branch and pull_request event in “main” branch

jobs: A workflow job is a set of steps that execute on the same runner. We can have multiple jobs in a single workflow yml file. Groups together all the jobs that run in the Android CI workflow. Here, in the example there is a single job whose name is build

runs-on: Configures the job to run on the latest version of an Ubuntu Linux runner. This means that the job will execute on a fresh virtual machine hosted by GitHub. You can use windows and macOS runner too.

steps: Groups together all the steps that run in the build job. Each item nested under this section is a separate action or shell script.

uses: actions/checkout@v3 : The uses keyword specifies that this step will run v3 of the actions/checkout action. This is an action that checks out your repository onto the runner, allowing you to run scripts or other actions against your code (such as build and test tools). You should use the checkout action any time your workflow will run against the repository's code.

uses: actions/setup-java@v3 : This step uses the actions/setup-java@v3 action to install the specified version of the JDK (this example uses v11) of distribution: 'temurin'

run: chmod +x gradlew: The run keyword tells the job to execute a command on the runner. In this case, you are granting execute permission for gradlew

run: ./gradlew build: In this case you are building the code using gradle

After this , Click on Start Commit and add comment and click Commit. This will create a basic android CI workflow in GitHub Action.

To create a secret in GitHub Actions  Go to Settings and then click on Secrets and then to Actions and create different secrets .

Let’s modify the android.yml file : Full Workflow

Let’s discuss part by part

Here, the workflow is getting triggered whenever we push into the branches “main” or “qa” or “develop” and whenever we pull request into the branches “main” or “qa”.

You can use environment variables to store information that you want to reference in your workflow. You reference environment variables within a workflow step or an action, and the variables are interpolated on the runner machine that runs your workflow. Commands that run in actions or workflow steps can create, read, and modify environment variables.

You can define environment variables that are scoped for:

The entire workflow, by using env at the top level of the workflow file.
The contents of a job within a workflow, by using jobs.<job_id>.env.
A specific step within a job, by using jobs.<job_id>.steps[*].env.

Here in the workflow we can have created a env at the top level with variable name as “AWS_DEFAULT_REGION” and assigned value as “ap-south-1”

uses: actions/setup-java@v3 : This step uses the actions/setup-java@v3 action to install the specified version of the JDK (this example uses v11) of distribution: 'temurin'

run: chmod +x gradlew: The run keyword tells the job to execute a command on the runner. In this case, you are granting execute permission for gradlew

run: ./gradlew clean: Gradle clean will delete a build directory if already present.

run: ./gradlew lint: It will detect poorly structured code that can impact the reliability and efficiency of your Android apps and make your code harder to maintain

run: ./gradlew build: A process of building a Gradle project

run: ./gradlew jacocoTest: The JacocoReport task can be used to generate code coverage reports in different formats

In this step we are integrating GitHub Actions with SonarQube. To make your workflows faster and more efficient, you can create and use caches for dependencies and other commonly reused files.

In the Cache SonarQube Package Step and Cache Gradle Package we are caching SonarQube packages and Gradle Packages with given path where runner stores the cache. The new cache will use the key you provided and contains the files you specify in path and alternative restore key used if no cache hit occurs for key, these restore keys are used sequentially in the order provided to find and restore a cache.

In the next step we are doing code analysis using SonarQube. We have added environment variables whose values are as secrets which you can use in your workflows as environment variables.

The secrets in GitHub Actions are defined as {{ secrets.secret_name }}. Here we have added SonarQube Token and SonarQube URL as secrets. Then we are running the command in runner as “./gradlew sonarqube” and passing the projectkey.

In build.gradle we have added the plugins for SonarQube

We can see that SonarQube analysis is passed and coverage report is greater than 80%

In the next step “Date and Time” , we have evaluated date and time using Linux command and created outputs in the step by writing to stdout in the format of ::set-output name=<name>::<value>. A step can have multiple outputs. Steps that create outputs must have unique ids.

current_date_time::$(date +”%d-%m-%Y-%H-%M-%S”)  Here current_date_time variable is in the format of %d : Day of the month , %m : Month , %Y : Year , %H : Hour, %M : Minutes, %S : Seconds

Here, the output name is “current_date_time” and id of the step of “Date and Time” is “date” which is unique name in the workflow.

To use this parameter in the job we can use in the way {{ steps.<step-id>.outputs.<output-name> }} . Here in the example it is {{ steps.date.outputs.current_date_time }}

In the next step “Copy APK files to a directory” we are creating a directory structure to store the Debug and Release APK files in the format

apk-files > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > release> app-release-unsigned-11–11–2022–09–09–12–36.apk

In the last step “Upload apk-files” Directory we are uploading artifactory — APK Files which we created so that in the deploy “job” we can download this artifactory as it will be using a new ubuntu runner. Here {{ github.workspace }} is default path for the checkout action. The path which we want to upload is the apk-files directory and if-no-files is present in the path , then ignore this step.

In this step we are integrating teams with GitHub Actions. This action take your GitHub token and the webhook URL which is generated during the configuration part .

Create a teams channel add people who should be notified for workflow success and fail steps . Click Connectors in the channel and then choose “Incoming Webhook” and add and then configure and then add a name and copy the URL and paste it as a secret in your GitHub Secrets and use it in your workflow.

If you see the last part, we are creating an outputs variable name as “CURRENT_DATE_TIME” and passing the date time value . Since this variable we want to use in another job “deploy” . To pass any variables from 2 different jobs we need to create output values like this.

Then we are creating another job named as “deploy” where we are adding our CD part of the Workflow.

Here “needs” : build means after successfully executing the “build” job only this “deploy” job will run.

Then the if condition is telling that if branches are “qa” or “master” then only this steps will run inside the runner.

Then again in the steps checkout repository is happening and in the next step “Download apk-files Artifactory” , we are downloading the artifact which we just uploaded in “build” job. We have mentioned the path where we need to download the artifact

Then in “Display structure of downloaded files” step we are checking the directory structure of downloaded artifacts.

In the next step “Public IP of GitHub Hosted Runner”  we are generating the Public IP of the GitHub Hosted Runner by using the action haythem/public-ip@1.3

Then in the next step we are adding the Public IP to Security Group in which Jfrog is running in the EC2 Instance so that GitHub Actions can access the Jfrog Page at 8082 using AWS CLI Commands “authorize-security-group-ingress”. For this we need to create a user with having EC2 Full Access permission and give programmatic access to get access_key_id and secret_access_key.

In the first step we are downloading the Jfrog CLI with latest version. Add the JF_URL which is the URL of the artifactory where we are storing the .apk files and access token which we can create in Admin → User Management → Access Token .

Also set the “Password Encryption Policy” to Unsupported for demo purpose.

Then in the next step we are creating folders for QA and Master Branch. Where in script we are using if and else condition , that if GitHub branch is qa then we will create a directory QA in apk-files directory else master. Now the directory structure will be like:

apk-files > qa > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > qa> release> app-release-unsigned-11–11–2022–09–09–12–36.apk

apk-files > master > debug > app-debug-11–11–2022–09–09–12–36.apk

apk-files > master > release> app-release-unsigned-11–11–2022–09–09–12–36.apk

In the step “Upload APK files to Jfrog” we are using Jfrog CLI commands to upload apk files from Ubuntu Runner to Jfrog Artifactory –> “android-artifact”

jf rt u — url ${{ secrets.JF_URL }} — user ${{ secrets.JF_USER }} — password ${{ secrets.JF_PASSWORD }} apk-files/qa/debug/app-debug-${{ needs.build.outputs.CURRENT_DATE_TIME }}.apk android-artifact/

Here, u means upload , — url means Artifactory Repository URL (android-artifact) one, — user means Username of Jfrog UI and –password means Password of Jfrog UI , <path of the file want to upload> <artifact-repo-name>/

This URL to file is the JF_URL which we need to add a secret in GitHub Actions.

Then in the next step , we are removing the GitHub Actions Public IP from the security group of Jfrog EC2 Instance . if: always() make sure that this step runs always although if any steps fails.

Then in the last step we are sending notifications to teams. You can see all these information will go to teams.

Use Case : If we want to delete the caches which are formed after running each workflow

Here, we are using workflow_run command  It allows you to execute a workflow based on execution or completion of another workflow. So, here we are telling that “Clear Cache” Workflow will run only when workflow “Android CI and CD” workflow will complete (type of activity) successfully. Then we add permissions as write in a top-level key, to apply to all jobs in the workflow. When you add the permissions key within a specific job, all actions and run commands within that job that use the GITHUB_TOKEN gain the access rights you specify.

After that we run a script where we are first listing all Caches using JavaScript command and then we are deleting caches using their ID.

You can see all the caches got deleted which got created in the previous workflow.

Use Case : If we want to delete the artifacts which are formed when we uploaded the artifacts (apk-files) so that we can pass that directory from build job to deploy job using cron job — Every HOUR

Here, we are running a cron job which will run every hour and delete all artifacts which got created . Here we are passing the GitHub token in purge-artifacts action.

Use Case : How to create Self Hosted Runner and how to configure the self-hosted runner application as a service.

To create self-hosted runner , Go to Settings  Actions  Runner and click create and select the type of OS you have . For me I am choosing Linux OS.

Run the commands which you get here.

Add all the values when asking about Runner Registration.

To connect to the Runner we need to start the run.sh file

We can see that the runner is up and running now.

If you want to configure the self-hosted runner application as a service so that the runner is up and running if your Linux machine is up and running

Run these commands :

Installing the service

1. Stop the self-hosted runner application if it is currently running.

2. Install the service with the following command:

sudo ./svc.sh install

3. Alternatively, the command takes an optional user argument to install the service as a different user.

./svc.sh install USERNAME

Starting the service

Start the service with the following command:

sudo ./svc.sh start

Checking the status of the service

Check the status of the service with the following command:

sudo ./svc.sh status

If you want test with Android Application and want all codes : Check this out

Buy me a coffee :) ← — — If you like my articles

DevOps from 0 to Hero - for Freshers

Lionel♾️☁️ — Sun, 20 Oct 2024 08:32:44 +0000

Introduction

DevOps is a transformative culture and set of practices that bring together software development (Dev) and IT operations (Ops). It aims to shorten the development lifecycle, deliver continuous integration and continuous delivery (CI/CD), and ensure high software quality. If you're a fresher with zero knowledge in DevOps, this guide will help you get started on your journey to becoming a proficient DevOps engineer.

📚 Step-by-Step Guide

1. Understand the Basics

1.1 What is DevOps?

DevOps is a set of practices that combines software development and IT operations. It emphasizes collaboration, communication, and integration between developers and IT operations teams. DevOps aims to automate and streamline the processes of building, testing, and deploying software.

1.2 Core DevOps Principles

Continuous Integration (CI): Regularly merging code changes into a central repository to detect and fix integration issues early.
Continuous Delivery (CD): Automating the process of deploying code changes to production after passing rigorous automated tests.
Infrastructure as Code (IaC): Managing and provisioning infrastructure through code, enabling version control and automated deployment of infrastructure.
Microservices Architecture: Breaking down applications into smaller, independently deployable services for improved scalability and maintainability.
Monitoring and Logging: Implementing robust systems to track application performance and quickly identify issues.

2. 🔧 Learn the Foundation Skills

2.1 Basic Programming

Learning a programming language is essential for automating tasks and writing scripts. Some widely used languages in DevOps are:

Python: Known for its simplicity and readability, Python is great for scripting and automation.
- Learn Python
Go: Gaining popularity in DevOps for its performance and concurrency features.
- Learn Go
JavaScript: Often used in web development and automation tasks.
- Learn JavaScript

2.2 Operating Systems

Understanding operating systems, especially Linux, is crucial as most DevOps tools and environments run on Linux. Learn basic commands, file systems, process management, and networking in Linux.

Linux Tutorial

2.3 Networking Basics

Understanding networking fundamentals is important for configuring and managing servers, containers, and applications. Learn about IP addresses, DNS, HTTP/HTTPS, firewalls, and load balancers.

Networking Basics

3. 🌿 Dive into Version Control

3.1 Git

Git is a version control system that tracks changes in source code, allowing multiple developers to work on a project simultaneously without conflicts. Learn the basics of Git commands and workflows.

3.2 GitHub

GitHub is a platform for hosting Git repositories, providing tools for collaborative development, code review, and project management.

4. 🔄 Master Continuous Integration and Continuous Delivery (CI/CD)

4.1 Jenkins

Jenkins is an open-source automation server that helps automate parts of the software development process, including building, testing, and deploying code.

4.2 GitLab CI/CD

GitLab CI/CD is a powerful tool integrated with GitLab for automating the entire DevOps lifecycle. Learn how to create and manage CI/CD pipelines.

5. Explore Configuration Management

5.1 Ansible

Ansible is an open-source automation tool used for configuration management, application deployment, and task automation. It uses simple, human-readable YAML templates to define automation jobs.

5.2 Puppet

Puppet is a configuration management tool that helps automate the provisioning and management of infrastructure.

6. 🐳 Understand Containerization and Orchestration

6.1 Docker

Docker is a platform for developing, shipping, and running applications inside containers. Containers are lightweight, portable, and consistent environments that ensure applications run the same way regardless of where they are deployed.

6.2 Kubernetes

Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It helps manage containerized applications in a clustered environment.

7. ☁️ Explore Cloud Platforms

7.1 AWS (Amazon Web Services)

AWS is a comprehensive cloud computing platform offering a wide range of services, including compute, storage, and databases. Learn the basics of AWS services such as EC2 (virtual servers), S3 (object storage), RDS (relational databases), and Lambda (serverless computing).

7.2 Azure

Azure is Microsoft's cloud computing platform that provides a variety of cloud services, including those for compute, analytics, storage, and networking. Familiarize yourself with Azure's offerings and capabilities.

7.3 Google Cloud Platform (GCP)

GCP is Google's cloud computing service, offering a range of services such as compute, storage, and machine learning. Learn about GCP's infrastructure and services.

8. Learn Infrastructure as Code (IaC)

8.1 Terraform

Terraform is an open-source tool for building, changing, and versioning infrastructure safely and efficiently. It allows you to define and provision infrastructure using a high-level configuration language.

9. 📊 Implement Monitoring and Logging

9.1 Prometheus

Prometheus is an open-source monitoring system and time-series database that is well-suited for monitoring containerized applications.

9.2 ELK Stack (Elasticsearch, Logstash, Kibana)

The ELK Stack is a powerful set of tools for searching, analyzing, and visualizing log data. Elasticsearch stores and indexes log data, Logstash processes it, and Kibana visualizes it.

10. Get Hands-On Experience

10.1 Build Projects

Apply what you've learned by working on real projects. Here are some ideas:

Build a complete CI/CD pipeline for a sample application
Deploy a microservices architecture on Kubernetes
Implement a multi-cloud disaster recovery solution

Practical experience is crucial for mastering DevOps.

11. 🤝 Join the Community

Participate in DevOps communities, forums, and meetups to learn from others, share your experiences, and stay updated on industry trends.

12. 📚 Continuous Learning

12.1 Online Courses

Enroll in online courses to deepen your understanding and keep your skills up-to-date. Many platforms offer comprehensive DevOps courses taught by industry experts.

12.2 Books

Read books on DevOps practices, tools, and methodologies. Some highly recommended books are:

The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford: A novel about IT, DevOps, and helping your business win.
The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois, and John Willis: How to create world-class agility, reliability, and security in technology organizations.

DevOps is not just about tools, but also about fostering a culture of collaboration, continuous improvement, and shared responsibility. Embrace the DevOps mindset in your work and interactions with team members.

📚 Security in DevOps (DevSecOps)

As you progress in your DevOps journey, don't forget to integrate security practices into your workflows. DevSecOps emphasizes the importance of building security into every stage of the development process.

Career Paths in DevOps

DevOps offers various career paths and specializations. Some roles you might consider as you progress in your career include:

DevOps Engineer
Site Reliability Engineer (SRE)
Cloud Architect
Automation Specialist
DevSecOps Engineer

DevOps Podcasts and YouTube Channels

Stay up-to-date with the latest in DevOps through these popular podcasts and YouTube channels:

Podcasts:

YouTube Channels:

A Day in the Life of a DevOps Engineer

To give you a practical perspective of what it's like to work as a DevOps engineer, here's a typical day:

Time	Activity
8:00 AM	Start the day by checking monitoring dashboards for any overnight issues
9:00 AM	Attend the daily stand-up meeting with the development team to discuss ongoing projects and potential blockers
10:00 AM	Work on automating a deployment process using Jenkins and Ansible
12:00 PM	Lunch break and catch up on the latest DevOps news and trends
1:00 PM	Troubleshoot a production issue reported by the operations team
3:00 PM	Collaborate with developers to optimize a Docker container for a new microservice
4:00 PM	Review and merge pull requests for infrastructure-as-code changes
5:00 PM	Document the day's work and plan for tomorrow's tasks

This schedule can vary greatly depending on the organization and current projects, but it gives you an idea of the diverse tasks a DevOps engineer might handle in a day.

Conclusion

Learning DevOps from scratch may seem daunting, but with the right approach and resources, you can master the essential skills and become a proficient DevOps engineer. Follow this step-by-step guide, practice consistently, and engage with the DevOps community to accelerate your learning journey.

Stay curious, and embrace the DevOps mindset to drive innovation and efficiency in software development and operations! 🎉

Thank you for reading our blog …:)

Join Our Telegram Community || Follow us for more DevOps Content.

Deploying Django Application on AWS with Terraform - Part 1

Lionel♾️☁️ — Mon, 01 Apr 2024 05:25:32 +0000

Hi everyone, I am back with another project. This time we will be working on a CICD pipeline, deploying our application to our AWS account. In this project we will use Terraform to deploy our Django application to our AWS account.

Below are the tools and services that we will use and their categories.

https://gist.github.com/apotitech/eefebc53ce4c154f180defdc758d385f

Before we get into today's hands on writeup, let us start with the different parts of our project from start to finish.

Parts of Project

Minimal Working Setup
Connecting PostgreSQL RDS
GitLab CI/CD
Namecheap Domain + SSL
Celery and SQS
Connecting to Amazon S3
ECS Autoscaling

We are starting with the first part today

Minimal Working Setup

In this part, we will go through first - the setup of our AWS account, then we will create our Terraform project, and lastly we will define resources for our web application. At the end, we will deploy our Django application on our AWS ECS. We will access our app using our Load Balancer URL.

Creating Django project

Let’s start with our Django application. Create a new folder and initialize a default Django project.



$ mkdir django-aws && cd django-aws
$ mkdir django-aws-backend && cd django-aws-backend
$ git init --initial-branch=main
$ python3.10 -m venv venv
$ . ./venv/bin/activate
(venv) $ pip install Django==3.2.13
(venv) $ django-admin startproject django_aws .
(venv) $ ./manage.py migrate
(venv) $ ./manage.py runserver

Now that our Django server is setup, let's check our Django greeting page at http://127.0.0.1:8000, ensure that Django is running, and kill the development server.

Next, we are going to dockerize our application. First, we will add a requirements.txt file to the Django project:



Django==3.2.13

For testing purposes, enable debug mode and allow all hosts in our settings.py



DEBUG = True

ALLOWED_HOSTS = ['*']

To containerize our app, we need to have a dockerfile. So next, we add a Dockerfile in our working directory:



FROM python:3.10-slim-buster

# Open http port
EXPOSE 8000

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV DEBIAN_FRONTEND noninteractive

# Install pip and gunicorn web server
RUN pip install --no-cache-dir --upgrade pip
RUN pip install gunicorn==20.1.0

# Install requirements.txt
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

# Moving application files
WORKDIR /app
COPY . /app

Now let us go ahead and build and run our docker container locally.



$ docker build . -t django-aws-backend
$ docker run -p 8000:8000 django-aws-backend gunicorn -b 0.0.0.0:8000 django_aws.wsgi:application

Let's go to pur localhost port http://127.0.0.1:8000 page and verify that we have successfully built and run the docker image with a Django application. You should see exactly the same greeting page as for the runserver command.

There may be some files that we do not want to push to our git repo. For that. let’s add a .gitignore file:



*.sqlite3
.idea
.env
venv
.DS_Store
__pycache__
static
media

Next, we will continue with our git steps. Now we commit our changes:



$ git add .
$ git commit -m "initial commit"

Good job making it this far.

For now, we are done with the Django part. In the next steps, we deploy our application on AWS. But first, we need to configure our AWS account.

We need to create credentials for AWS CLI and Terraform. So we’ll create a new user with administration access to the AWS account. This user will be able to create and change resources on your AWS account.

First, we will go to the IAM service, select the “Users” tab, and click “Add Users”.

Enter Username and choose the ‘Access key — Programmatic access’ option. This option means that your user will have ‘Access key’ to use AWS API. Also, this user won’t be able to sing in the AWS web console.

Select the “Attach existing policies directly” tab and select “AdministratorAccess.” Then click next and skip the “Add tags” step.

Review user details and click “Create user.”

Yay, we have successfully created our user!

Now we need to save your Access key ID and Secret access key in some safe place. Be aware of committing these keys in public repositories or other public places. Anybody who owns these keys can manage your AWS account.

Now we can configure AWS CLI and check our credentials. We will use the us-east-2 region in this guide. Feel free to change it.



$ aws configure
AWS Access Key ID [None]: AKU832EUBFEFWICT
AWS Secret Access Key [None]: 5HZMEFi4ff4F4DEi24HYEsOPDNE8DYWTzCx
Default region name [us-east-2]: us-east-2
Default output format [table]: table
$ aws sts get-caller-identity
-----------------------------------------------------
|                 GetCallerIdentity                 |
+---------+-----------------------------------------+
|  Account|  947134793474                           |  <- AWS_ACCOUNT_ID
|  Arn    |  arn:aws:iam::947134793474:user/admin   |
|  UserId |  AIDJEFFEIUFBFUR245EPV                  |
+---------+-----------------------------------------+

Remember your AWS_ACCOUNT_ID. We'll use it in the next steps.

Now we are all set up to create Terraform project!

Creating Terraform Project

Let’s create a new folder django-aws/django-aws-infrastructure for our Terraform project.



cd .. 
mkdir django-aws-infrastructure && cd django-aws-infrastructure  
git init --initial-branch=main

Let us add our provider.tf:



provider "aws" {
  region = var.region
}

Here, we defined our AWS provider. We use Terraform variable for specifying an AWS region. Let’s define region and project_name variables in the variables.tf file:



variable "region" {
  description = "The AWS region to create resources in."
  default     = "us-east-2"
}

variable "project_name" {
  description = "Project name to use in resource names"
  default     = "django-aws"
}

Now, we will run terraform init to create a new Terraform working directory, download the AWS provider, and everything else that will be needed.

Now we are ready to create resources for our infrastructure.

Resources we will use

Here is the plan of the services we wull use to configure our project

https://gist.github.com/apotitech/6c1ac26ee528eacb7dbeae7635253058

Elastic Container Repository, ECR



resource "aws_ecr_repository" "backend" {
  name                 = "${var.project_name}-backend"
  image_tag_mutability = "MUTABLE"
}

Next, we will run terraform plan. We'll see that Terraform is going to create an ECR repository.



Terraform will perform the following actions:
  # aws_ecr_repository.backend will be created
  + resource "aws_ecr_repository" "backend" {
      ...
    }
Plan: 1 to add, 0 to change, 0 to destroy.

If the plan looks good, we will go ahead and run terraform apply. We will be prompted to accept or refuse the plan. Type yes to confirm changes.



aws_ecr_repository.backend: Creating...
aws_ecr_repository.backend: Creation complete after 1s [id=django-aws-backend]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Great, our repository is created. Now, let’s push our Django image to this new registry. Before we do that, we will need to build an image with tag, below is mine ${AWS_ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/django-aws-backend:latest, authorize in the ECR, and push an image:

In my own case, my account ID is 94713479347 and I will use the us-east-2 region



$ cd ../django-aws-backend
$ docker build . -t 947134793474.dkr.ecr.us-east-2.amazonaws.com/django-aws-backend:latest
$ aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 947134793474.dkr.ecr.us-east-2.amazonaws.com
$ docker push 947134793474.dkr.ecr.us-east-2.amazonaws.com/django-aws-backend:latest

Network

Now, let’s create a network for our application. First we create our variables.tf file. Add this block to the variables.tf file:



variable "availability_zones" {
  description = "Availability zones"
  default     = ["us-east-2a", "us-east-2c"]
}

Next, we will create a network.tf file with the following content:

https://gist.github.com/apotitech/ba6d140d902cef8e7038118829335d7e

Below are the resources that we will be creating. We’ve defined the following resources with our code:

Virtual Private Cloud.
Public and Private subnets in different Availability zones
Internet Gateway for internet access for our public subnets.
NAT Gateway for internet access for our private subnets.

Next we will apply what we just coded. Run terraform apply command to apply changes on AWS.

Load Balancer

We continue our infrastructure building. Next we will create our load balancer load_balancer.tf file with the following content:

https://gist.github.com/apotitech/da3f7ba10640b6c1809c2894c7525d62

Let us look at the resources that the code below will create:

Application Load Balancer
LB Listener to receive incoming HTTP requests.
LB Target group to route requests to the Django application.
Security Group to control incoming traffic to load balancer.

Before we proceed, we want to know the load balancer URL. For that we need our code to output it in our terminal. For that we will add a outputs.tf file with the following code and run terraform apply to create load balancer and see its hostname.



output "prod_lb_domain" {
  value = aws_lb.prod.dns_name
}

We will see our ALB domain in output.



Outputs:
prod_lb_hostname = "prod-57218461274.us-east-2.elb.amazonaws.com"

We can now check our this domain in our browser. It should respond with 503 Service Temporarily Unavailable error because there are no targets associated with the target groups we created yet.

In the next step, we'll deploy the Django application that will be accessible by this URL.

Application

Last but not least, we’ll create the application using the ECS Service. For this, we will add a ecs.tf file with following content:

https://gist.github.com/apotitech/bfb372e92d2b88fee935ee39a8f68ed7

Also, we will add the ecs_prod_backend_retention_days variable to the variables.tf file:



variable "ecs_prod_backend_retention_days" {
  description = "Retention period for backend logs"
  default     = 30
}

then we add a container definition in a new file templates/backend_container.json.tpl and run terraform apply.



[
  {
    "name": "${name}",
    "image": "${image}",
    "essential": true,
    "links": [],
    "portMappings": [
      {
        "containerPort": 8000,
        "hostPort": 8000,
        "protocol": "tcp"
      }
    ],
    "command": ${jsonencode(command)},
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "${log_group}",
        "awslogs-region": "${region}",
        "awslogs-stream-prefix": "${log_stream}"
      }
    }
  }
]

Our code will be creating:

ECS Cluster
ECS Task Definition
ECS Service to run tasks with the specified definition in the ECS cluster
IAM Policies to allow tasks access to resources.
Cloudwatch Log group and stream for log collection.

Now, go to the our ECS AWS Console and look at our running service and tasks.

Now we will go ahead and Check the Load Balancer domain in a browser again, to ensure that our setup works. Now we will see the Django’s starting page.

Awesoome, great job making it this far, our setup is working. It’s time to commit our changes in the django-aws-infrastructure repo. We will add another file, a .gitignore file

Great job making it this far. Now we will go ahead and apply all of our terraform configuration terraform apply . We will st



output "prod_lb_domain" {
  value = aws_lb.prod.dns_name
}

We will see our ALB domain in output.



Outputs:
prod_lb_hostname = "prod-57218461274.us-east-2.elb.amazonaws.com"

We can now check our this domain in our browser. It should respond with 503 Service Temporarily Unavailable error because there are no targets associated with the target groups we created yet.

https://gist.github.com/apotitech/5f4afbe7826322f8cf6de112d150cd8f

Our code is all ready to go. Now we will save, commit our changes and push our changes:



$ git add .

$ git commit -m "initialize infrastructure"

Congratulations

Yay! We have now deployed our Django web application with ECS Service + Fargate on AWS. But now it works with SQLite file database. This file will be recreated on every service restart. So, our app cannot persist any data for now. In the next article we’ll connect Django to AWS RDS PostgreSQL.

If you need technical consulting on your project or have any questions or suggestions, please comment below or connect with me directly on LinkedIn.

Do not forget the 👏❤️ and share if you like this content!

Thank you for joining me, and best of luck with your AWS endeavors!

Thank you so much everyone!

Lionel♾️☁️ — Sat, 21 Oct 2023 19:23:14 +0000

Hi fam, I'm so excited to announce that I've reached 30,000 views and 6K+ followers! This is such a huge milestone for me, and I couldn't have done it without your support.

I'm so grateful for all of your comments, shares, and feedback. It means the world to me that you're interested in what I have to say about DevOps, Cloud, SRE and AI.

Your support is what keeps me going and helps me write even more articles.

I love hearing from you and learning from your experiences as well.

One of my favourite things about being a part of the dev.to community is the people I've met along the way. I've learned so much from all of you, and I'm so grateful for your support.

I remember one time when I was writing an article on a new DevOps tool. I was struggling to understand a particular concept, so I asked for help in the dev.to community. Within minutes, I had several helpful responses from other developers. I was so impressed by their willingness to help me out.

This is just one example of the many ways that the dev.to community has helped me. I'm so grateful to be a part of such a supportive and knowledgeable group of people.

I'm so glad to be a part of the dev.to community. Thank you for being a part of my journey! I couldn't have reached this milestone without you. I'm excited to continue sharing my knowledge and experience with the dev.to community in the years to come.

I'm also open to feedback on my existing articles, please don't hesitate to let me know what you think.

What topics would you like to see me write about in the future? Please share your suggestions in the comments below.

How Netflix Uses the Cloud - AWS

Lionel♾️☁️ — Sun, 24 Sep 2023 05:33:01 +0000

How they Use AWS Services

Binge-watching 🍿has become more and more of a phenomena. Netflix 🎬 has transformed the way we watch shows. Beneath its interface 🔍 is a really sophisticated system that effortlessly introduces new shows, movies 🎥 and beams them to countless devices 📱💻 worldwide.

Today, we'll take a deep dive into the magic ✨ behind Netflix's content orchestration and distribution, and unravel the AWS cloud ☁️ mechanics that support its beautiful framework. 🌐🛠️

1. The Netflix Magic 🎬🍿

Step 1: Metadata and Shows ✨

We start our Netflix adventure with its administrators 🧙‍♂️ uploading fresh episodes and movies, with metadata such as tags 🏷️, titles 📜, and descriptions 🖋️. The metadata is the backbone of organizing and categorizing shows, making your search for the next binge-watch, sleek and smooth. The metadata will be stored in the Elasticsearch/OpenSearch Database 🗃️, renowned for their flash ⚡ speed search !

Step 2: Uploading and Process Videos! 🎥➡️☁️

Once our Admin hits the upload button, the video 🎞️ is uploaded to an Amazon S3 bucket 🪣. To make sure everyone, no matter their device 📱💻🖥️ or internet speed 🌐, gets a flawless viewing 🍿 experience, once uploaded, the video gets a a complete makeover 💄. Thanks to AWS Elemental MediaConvert 🌀, the video is converted into various sizes (4K, 1080p, 720p), perfectly fitting all screen sizes.

Step 3: AI Content Analysis in Action! 🔍🤖

Next, let's get into quality checks. For that, Netflix uses its AWS Rekognition 🧠⚙️. It meticulously scans 🕵️‍♂️ all uploaded videos to spot any sensitive 🚫 content 📼. Thanks to this tool 🤖🔍, Netflix ensures that their platform remains a safe and wholesome space for all viewers. After all it is safety first, binge later! 🛡️🍿

Step 4: AWS Step Functions for Parallel Processing 🤹⚙️

We all have our shows that we are impatiently waiting to watch on our favorite streaming platform. So the Netflix team cannot wait and upload one movie then another. To fix this, Netflix amped up its efficiency by using AWS Step Functions for speedy parallel processing 🌌⏩of videos. Think of AWS Step Functions as a series 🎼of tasks, from content scanning 🔄 to AI content analysis 🔍🤖 (Step 1 to 3), all at the same beat. With all these processes happening side by side, not only does the waiting time ⏳ drop drastically, but in addition the whole Netflix ecosystem runs even smoother 🚀✨.

Step 5: Smart Storage Frequent vs Infrequent Access 📦🔄

Once the processing is completed, our videos are stored in a frequently used Amazon S3 bucket 🥂💼, where it's all set for prime time. Here's the smart bit: any movie or episode that is not frequently watched 🤷‍♂️📉, gets transferred to a cozy spot in the 'chill-zone' S3 Glacier 🛋️🪣. This storage switch-ups 🕺💃 not only ensure that costs are kept low 💲⚖️ but also that resources are used in the best way possible 🧠⚡

Efficiency meets elegance! 🎩✨

Step 6: Deep Freeze Dive for Rare Gems 🏔️❄️

What about those shows and episodes 🎬 that only a select few cherish 💎? Netflix keeps them Amazon S3 Glacier. Think of AWS S3 Glacier as a vault 🗝️ where cinematic rarities are safely kept, until needed. This multi-layered storage strategy 🍰 ensures that all Netflix's content is cost-effective.

A balance of budget 💸 and value 💡 with flair! 🎉🌌

Next -> The Nextflix Express Lane

Step 7: Netflix and AWS CDN (Open Connect) 🚀🎥

In 2020, Netflix became and international phenomenon. They needed to provide similar seamless quality to their international customers. As a solution, Netflix used AWS' Content Delivery Network, Open Connect 🌐⚡ Open Connect uses AWS' Edge Locations located internationally to slash wait times 🕐 making sure their next binge-worthy shows are just a blink away 📺✨.

Lightning-quick movie magic! 🍿🎬🎉

Step 8: Diving Deep with Data Magic 🔍✨

The one thing which too many people seem not really that important but which makes all the difference to all companies, is data. To unlock the mysteries of its user habits 🕵️‍♂️💡 and system prowess 🖥️🚀, Netflix uses ELK stack trio: Elasticsearch, Logstash, and Kibana.

How do they all work together to make Netflix better ?

First Elasticsearch the detective, sorts and stashes away logs (clues) 📁🔍; next, Logstash the craftsman, processes and channels the data 🌊⚙️; and Kibana? The artist 🎨, paints beautiful data visualizations for Netflix engineers to use. With this dynamic trio, Netflix ensures every move is backed by data 📊 and strategy🪄!

2. Netflix and us, viewers 🌍✨

Step 1: One Service, Many Screens 📺📱💻🎮

Now let us step into our shoes. Whether you're on a smartphone 📱, TV 📺, laptop 💻, or even a gaming console 🎮, you can access your favorite streaming platform. This chameleon-like adaptability ensures that no matter your device, you always have access to 🍿🎬the same responsive, seamless and beautiful user experience.

Step 2: A Dazzling Dance of Design and Delivery 🌐💃

Whenever you open Netflix, you're actually viewing their website built using the magical React.js 🪄🖥️. React.js is a JavaScript library and is what gives you a dazzling experience. When you halt on a movie, a video snippet plays which captivates 🌀🎨 your attention. And who ties it all together? The CDN (Open Connect), acting as the bridge 🌉, ensuring that every video and frame is delivered in perfect harmony 🎶🎬.

Step 3: Searching for a show 🔍💎

Looking for your next show? Go to the search page of Netflix and type away! 🎹✨ As you search, the website communicates with the CDN 🌐, which connects with Netflix's backstage — AWS API Gateway and microservices 🎩⚙️. This triggers a Lambda function (a magic spell) if you will 🪄⚡, which will go and search / query the database, using the metadata you provided.

Your personal concierge for your binge-watching journey. 🍿🌟

Step 4: Using CDN for Video Delivery🚂✨

Once the search is complete, the CDN (Open Connect), like a cinematic librarian 🤓📚, will quickly grab the video's metadata from the digital archives -> Database and, at the same time, connects to the S3 bucket where the shows🎬🪣 are kept. Both the movie's metadata (details) and the movie itself will be cached / stashed within the CDN 🌐🎥, prepping for future movie nights.

Step 5: Seamless Playback 🍿🌌

Thanks to CDN's (Open Connect) 🌐 smart caching / stashing skills 🧠🔐, frequently accessed content are very smooth (minimal latency)! Regardless of a user's location 🌍, this caching mechanism ensures a lag-free experience 🎥💨. So sit back, relax, and enjoy a seamless cinema spree! 🛋️🎉

Conclusion 🌟

I am still to study the backend of other streaming platforms. However from my research, Netflix's behind-the-scenes magic ✨ is nothing short of tech wizardry 🎩🔮. First with AWS's unique tools and power trio - Elemental MediaConvert, Rekognition, and Step Functions, Netflix consistently concocts a potion of optimized content 🏰. In addition to that, AWS' storage solutions 🗄️, Open Connect CDN 🌐⚡, and ELK 🔍 ensures that viewers like you and I get a cost-friendly, lightning-quick 🚀, and always-on 🌍 streaming experience.

As all our screens connect to Netflix 📺📱💻, one thing is very clear - their promise of delivering seamless content, remains unwavering 🤘🎬!

Every Project Deserves its CI/CD pipeline, no matter how small

Lionel♾️☁️ — Wed, 30 Aug 2023 20:30:00 +0000

TL;DR

In today's tech industry, setting up a CI/CD pipeline is quite easy. Creating a CI/CD pipeline even for a simple side project is a great way to learn many things. For today we will be working on one of my side projects using Portainer, Gitlab and Docker for the setup.

My sample project

As the founder of Apoti Development Association (A.D.A.) an NGO, I like organizing technical events in the Buea area (SW Region of Cameroon, Africa). I was frequently asked if there is a way to know all the upcoming events (the meetups, the Jugs, the ones organized by the local associations, etc.) After taking some time to look into it, I realized there was no single place which listed them all. So I came up with https://apotidev.org/events, a simple web page which tries to keep an update of all the events. This project is available in Gitlab.

Disclaimer: Even though this is a simple project, the complexity of this project is not important here. The different components of our CI/CD pipeline we will detail can also be used in almost the same way for more complicated projects. However they are a nice fit for micro-services.

A look at the code

To make things as simple as possible, we have an events.json file in which all new events are added. Let's look at a snippet of it

{
  "events": [
    {
      "title": "Let's Serve Day 2018",
      "desc": "Hi everyone! We're back with 50, 60-minute practitioner-led sessions and live Q&A on Slack. Our tracks include CI/CD, Cloud-Native Infrastructure, Cultural Transformations, DevSecOps, and Site Reliability Engineering. 24 hours. 112 speakers. Free online.",
      "date": "October 17, 2018, online event",
      "ts": "20181017T000000",
      "link": "https://www.alldaydevops.com/",
      "sponsors": [{"name": "all-day-devops"}]
    },
    {
      "title": "Creation of a Business Blockchain (lab) & introduction to smart contracts",
      "desc": "Come with your laptop! We invite you to join us to create the first prototype of a Business Blockchain (Lab) and get an introduction to smart contracts.",
      "ts": "20181004T181500",
      "date": "October 4 at 6:15 pm at CEEI",
      "link": "https://www.meetup.com/en-EN/IBM-Cloud-Cote-d-Azur-Meetup/events/254472667/",
      "sponsors": [{"name": "ibm"}]
    }
    …
  ]
}

Our mustache template is applied to this file. It will help us to generate the final web assets.

Docker multi-stage build

Once our web assets have been generated, they are copied into an nginx image — the image that is deployed on our target machine.

Thanks to Gitlab's multi-stage build, our build is in two parts:

creation of the assets
generation of the final image containing the assets

Let's look at the Dockerfile used for the build

# Generate the assets  
FROM node:8.12.0-alpine AS build  
COPY . /build  
WORKDIR /build  
RUN npm i  
RUN node clean.js  
RUN ./node_modules/mustache/bin/mustache events.json index.mustache > index.html


# Build the final Docker image used to serve them
FROM nginx:1.14.0  
COPY --from=build /build/*.html /usr/share/nginx/html/  
COPY events.json /usr/share/nginx/html/  
COPY css /usr/share/nginx/html/css  
COPY js /usr/share/nginx/html/js  
COPY img /usr/share/nginx/html/img

Local testing

Before we proceed, we need to test the generation of our site. Just clone the repository and run the test script. This script will create an image and run a container out of it.

# First Clone the repo
$ git clone git@gitlab.com:lucj/ada.events.git

# Next, cd into the repo
$ cd sophia.events

Now let us run our test script

$ ./test.sh

This is what our output looks lke

Sending build context to Docker daemon  2.588MB  
Step 1/12 : FROM node:8.12.0-alpine AS build  
 ---> df48b68da02a  
Step 2/12 : COPY . /build  
 ---> f4005274aadf  
Step 3/12 : WORKDIR /build  
 ---> Running in 5222c3b6cf12  
Removing intermediate container 5222c3b6cf12  
 ---> 81947306e4af  
Step 4/12 : RUN npm i  
 ---> Running in de4e6182036b  
npm notice created a lockfile as package-lock.json. You should commit this file.  
npm WARN www@1.0.0 No repository field.added 2 packages from 3 contributors and audited 2 packages in 1.675s  
found 0 vulnerabilitiesRemoving intermediate container de4e6182036b  
 ---> d0eb4627e01f  
Step 5/12 : RUN node clean.js  
 ---> Running in f4d3c4745901  
Removing intermediate container f4d3c4745901  
 ---> 602987ce7162  
Step 6/12 : RUN ./node_modules/mustache/bin/mustache events.json index.mustache > index.html  
 ---> Running in 05b5ebd73b89  
Removing intermediate container 05b5ebd73b89  
 ---> d982ff9cc61c  
Step 7/12 : FROM nginx:1.14.0  
 ---> 86898218889a  
Step 8/12 : COPY --from=build /build/*.html /usr/share/nginx/html/  
 ---> Using cache  
 ---> e0c25127223f  
Step 9/12 : COPY events.json /usr/share/nginx/html/  
 ---> Using cache  
 ---> 64e8a1c5e79d  
Step 10/12 : COPY css /usr/share/nginx/html/css  
 ---> Using cache  
 ---> e524c31b64c2  
Step 11/12 : COPY js /usr/share/nginx/html/js  
 ---> Using cache  
 ---> 1ef9dece9bb4  
Step 12/12 : COPY img /usr/share/nginx/html/img  
 ---> e50bf7836d2f  
Successfully built e50bf7836d2f  
Successfully tagged registry.gitlab.com/ada/ada.events:latest  
=> web site available on [http://localhost:32768](http://localhost:32768/)

We can now access our webpage using the URL provided at the end

Our target environment

Provisioning a virtual machine on a cloud provider

As you have probably noticed, this web site is not critical (only a few dozen visits a day), and as such it only runs on a single virtual machine. This one was created with Docker Machine on AWS, the best cloud provider.

Given the scale of our project, you must have noticed that it is not that critical (barely a few visits a day), so we will only one one virtual machine for it. For that we created ours with Docker on Exoscale, a nice European cloud provider.

Using Docker swarm

We configured our VM (virtual machine) above so it runs the Docker daemon in Swarm mode so that it would allow us to use the stack, service, secret primitives, config and the great (very easy to use) orchestration abilities of Docker swarm

The application running as a Docker stack

The file below -> ada.yaml defines the service which runs our Nginx web server that contains the web assets.

version: "3.7"

services:
  www:
    image: registry.gitlab.com/lucj/sophia.events
    networks:
      - proxy
    deploy:
      mode: replicated
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

networks:
  proxy:
    external: true

Let's break this down

The docker image is in our private registry on gitlab.com.
The service is in replicated mode with 2 replicas, this means that 2 tasks / 2 containers of the service are always running at the same time. A virtual IP address (VIP) will be associated to this service by Docker Swarm. That way, each request targeted at the service is easily load balanced between our two replicas.
Every time that an update is done to our service (like deploying a new version of the website), one of our replicas is updated and the 2nd is updated 10 secs after. This makes sure our website is always available even during the update process.
Our service is also attached to the external proxy network. This makes it so our TLS termination service (which runs in another service which is deployed on docker swarm, but out of our project) can always send requests to our www service.

Our stack is executed with the command:

$ docker stack deploy -c ada.yml ada_events

Portainer: One tool to manage them all

Portainer is a really great web UI which will help us to manage all our Docker hosts and Docker Swarm clusters very easily. Let's take a look at its interface where it lists all our stacks available in the swarm.

As you can see above, our current setup has 3 stacks:

First we have Portainer itself
Then we have ada_events or in this case sophia_events which contains the service which runs our web site
Last we have tls, our TLS termination service

If we go ahead and list the details of the www service, in the ada_events stack, we can easily see that the Service webhook is activated. This is a new feature, available since Portainer version 1.19.2. This update allows us to define a HTTP Post endpoint which we can call to trigger an update of our service. As you will notice later on, our Gitlab Runner is in charge of calling this webhook.

Note: As you see in the screenshot above, I use localhost:8888 to access Portainer. Since I don't want to expose our Portainer instance to the external world, access is done through an ssh tunne; which we open with the command below

ssh -i ~/.docker/machine/machines/labs/id_rsa -NL 8888:localhost:9000 $USER@$HOST

Once we have done this, all requests targeted at our local machine on port 8888 -> localhost:8888 are sent to port 9000 on our VM through ssh. Port 9000 is the port where Portainer is running on our VM but this port is not opened to the outside world. We used a security group in our AWS config to block it.

NB: Note that in the command above, the ssh key that was used to connect to the VM was the one generated by our Docker Machine during the creation of the VM.

GitLab runner

A gitlab runner is a continuous integration tool that helps automate the process of testing and deploying any and all applications. It works in with GitLab CI to run any job defined in the project's .gitlab-ci.yml file.

So in our project, our GitLab runner is in charge of executing all the actions we defined in the .gitlab-ci.yml file. On Gitlab, you have a choice of using your runners or using the shared runners available. In this project, we used a VM on AWS as our runner.

First we register our runner providing a couple of commands:

CONFIG_FOLDER=/tmp/gitlab-runner-config

$ docker run — rm -t -i \  
 -v $CONFIG_FOLDER:/etc/gitlab-runner \  
 gitlab/gitlab-runner register \  
   --non-interactive \  
   --executor "docker" \  
   —-docker-image docker:stable \  
   --url "[https://gitlab.com/](https://gitlab.com/)" \  
   —-registration-token "$PROJECT_TOKEN" \  
   —-description "AWS Docker Runner" \  
   --tag-list "docker" \  
   --run-untagged \  
   —-locked="false" \  
   --docker-privileged

As you see above, we have $PROJECT_TOKEN as one of the needed options. We will get that from the project page on Gitlab, used to register new runners

Once we have registered our gitlab runner, we can now start it

CONFIG_FOLDER=/tmp/gitlab-runner-config

$ docker run -d \  
 --name gitlab-runner \  
 —-restart always \  
 -v $CONFIG_FOLDER:/etc/gitlab-runner \  
 -v /var/run/docker.sock:/var/run/docker.sock \  
 gitlab/gitlab-runner:latest

Once our VM has been setup as a Gitlab runner, it will be listed in the CI/CD page, under settings of our project

Now that we have a runner, it can now receive work to do every time that we have a new commit and push in our git repo. It will sequentially run the different stages in our .gitlab-ci.yaml file. Let's look at our .gitlab-ci.yaml file -> the file that configures our Gitlab CI/CD pipeline

variables:
  CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH
  DOCKER_HOST: tcp://docker:2375
stages:
  - test
  - build
  - deploy

test:
  stage: test
  image: node:8.12.0-alpine
  script:
    - npm i
    - npm test

build:
  stage: build
  image: docker:stable
  services:
    - docker:dind
  script:
    - docker image build -t $CONTAINER_IMAGE:$CI_BUILD_REF -t $CONTAINER_IMAGE:latest .
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - docker image push $CONTAINER_IMAGE:latest
    - docker image push $CONTAINER_IMAGE:$CI_BUILD_REF
  only:
    - master

deploy:
  stage: deploy
  image: alpine
  script:
    - apk add --update curl
    - curl -XPOST $WWW_WEBHOOK
  only:
    - master

Let's break down the stages

First, the test stage starts by running some pre-checks ensuring that the events.json file is well formed and also makes sure that there is no images missing..
Next, the build stage uses docker to build the image and then pushes it to our GitLab registry.
Lastly, the deploy stage triggers the update of our service via a webhook sent to our Portainer app. > Note that the WWW_WEBHOOK variable is defined in the CI/CD settings of our project page on GitLab.com.

Some Notes

Our Gitlab runner is running inside a container in our Docker swarm. Like mentioned before, we could have instead used a shared runner (publicly available runners which share their time between the jobs needed by different projects hosted on GitLab) — but, in our case, since the runner must have access to our Portainer endpoint (to send the webhook), and also because I don't want our Portainer app to be publicly accessible, I preferred setting it this way, having it run inside the cluster. It is also more secure this way.

In addition to that, because our runner is in a docker container, it is able to send the webhook to the IP address of the Docker0 bridge network, to connect with Portainer through port 9000 which it exposes on the host. Thus, our webhook has following format: http://172.17.0.1:9000/api[…]a7-4af2-a95b-b748d92f1b3b

The Deployment

Updating a new version of our app follows the workflow below

It starts with a developer pushing some changes to our GitLab repo. The changes in the code mainly involve adding or updating one or more events in our events.json file while also adding some sponsors' logo.
After this, the GitLab runner performs all the actions that we defined in the .gitlab-ci.yml file.
Then the GitLab runner calls our webhook that is defined in Portainer.
Finally, upon the webhook reception, Portainer deploys the newest version of the www service. It does this, calling the Docker Swarm API. Portainer can access to the API because the /var/run/docker.sock socket is bind mounted once it is started.
Now our users can access the newest version of our events website.

Let's Test

Let's test our pipeline by doing a couple changes in the code and then committing / pushing those changes.

$ git commit -m 'Fix image'  

$ git push origin master

As you can see in this screenshot below, our changes triggered our pipeline

Breaking down the steps

On Portainer side, the webhook was received and the service update was performed. Also, although we cannot see it clearly here, one replica has been updated. Like we mentioned before, that still left the website accessible through the other replica. The other replica also was updated a couple of seconds later.

Summary

Although this was a tiny project, setting up a CI/CD pipeline for it was a good exercise. First it helped me get more familiar with GitLab (which has been on my To-Learn list for quite some time). Having done this project, I can say that it is an excellent, professional product. Also, this project was a great opportunity for me to play with the long awaited webhook feature available in updated versions of Portainer. Lastly, choosing to use Docker Swarm for this project was a real no-brainer - so cool and easy to use!

Hope you found this project as interesting as I did. No matter how small your project is, it would be a great idea to build it using CI/CD.

What projects are you working on and how has this article inspired you? Please comment below.

Create your first Web-app using ChatGPT

Lionel♾️☁️ — Fri, 25 Aug 2023 19:24:00 +0000

Introduction

Language translation is essential in our globalized world, bridging language gaps. Web apps are becoming more important for seamless communication due to the rising need for language translation services.

This article discusses creating a web-based language translation application utilizing ChatGPT and Streamlit. ChatGPT, an advanced OpenAI language model, can generate natural-sounding text responses from inputs provided. On the other hand, Streamlit is a powerful open-source framework for fast, easy data science web application development.

TL ; DR

Use Chatgpt to create a Translator Web-app

Here we go....

I. Starting ChatGPT and Streamlit

Before building our language translation web application, we must set up our development environment and install the necessary packages. This section explains how to configure and install OpenAI with Streamlit.

ChatGPT's language model was created by OpenAI, which powers our web-based translation tool. To use OpenAI's resources, we will need to get an API key from the website. With the API key, we can install OpenAI using pip in our command prompt or terminal:



pip install openai

Next, we need to setup Streamlit, which is the framework we will be using to build our web app. We can also use pip to install Streamlit:



pip install streamlit

Now that OpenAI and Streamlit have been installed, we can now start building our translation web-based app.

For more information on ChatGPT, Streamlit, and the OpenAI API key acquisition procedure, check out my in-depth tutorial, Building a ChatGPT Web Application using Streamlit and OpenAI: A Step-by-Step Tutorial.

II. Developing the Translation Web-App

After installing the necessary packages, let us now see how to create the user interface for our language translation web-based application. Please follow this full guide:

Name a new file app.py in your favourite Python editor. Please copy the code snippets into your file and save changes.
Import the necessary packages to start coding:



# Importing required packages  
import streamlit as st  
import openai

We will use ChatGPT model engine text-davinci-003. This model can mimic human responses and even translate across languages. This model requires registration and an OpenAI api_key, as explained in step (V.5) of this article. Now, add the following two lines to your script:



# Set the model engine and your OpenAI API key  
model_engine = "text-davinci-003"  
openai.api_key = "your_secret_key"

Next we will create the translate_tex function which will take care of the translation process. This function will take whatever the user selects as target language and text input and return the translated text. ```python

Define a function to handle the translation process

def translate_text(text, target_language):

Define the prompt for the ChatGPT model

prompt = f"Translate '{text}' to {target_language}"

Generate the translated text using ChatGPT

response = openai.Completion.create(

engine=model_engine,

prompt=prompt,

max_tokens=1024,

n=1,

stop=None,

temperature=0.7,

)

Extract the translated text from the response

translated_text = response.choices[0].text.strip()

return translated_text


5. Now to bring it all together, we will create the `main()` function that will call the needed functions to create our language web-based translation app. Let us setup this app, so the user will be able to translate text into `Arabic, English, Spanish, French, German, Japanese, Russian, Korean, Chinese, Yoruba` :

```python


# Define the main function that sets up the Streamlit UI and handles the translation process  
def main():  
# Set up the Streamlit UI  
st.sidebar.header('Language Translation App')  
st.sidebar.write('Enter text to translate and select the target language:')  

# Create a text input for the user to enter the text to be translated  
text_input = st.text_input('Enter text to translate')  

# Create a selectbox for the user to select the target language  
target_language = st.selectbox('Select language', ['Arabic', 'English', 'Spanish', 'French', 'German', 'Japanese', 'Russian', 'Korean', 'Chinese', 'Yoruba'])  

# Create a button that the user can click to initiate the translation process  
translate_button = st.button('Translate')  

# Create a placeholder where the translated text will be displayed  
translated_text = st.empty()  

# Handle the translation process when the user clicks the translate button  
if translate_button:  
translated_text.text('Translating...')  
translated_text.text(translate_text(text_input, target_language))

Now for our last step, we need to call the main function that we created



# Call the main function  
if __name__ == '__main__':  
main()

All done, now go ahead and save the file. We can give any name to our python file. I will call it Translator_GPT.py.

We can now run our translator app in our terminal.



streamlit run Translator_GPT.py

Once you run this command, a webpage in your default web browser will open where we will see our web-app.

Please go ahead and test the app for yourself. This is what it will look like.

III. Making our Web-app to the World

To make our app public, there are 3 easy ways.

We will look at how to make our available to everyone, using streamlit

III.1 Before Deploying to the Cloud…

We are now ready to deploy our translation app to the rest of the world. Before doing that, we need to do some house cleaning.

III.2 Install git

We need to install Git, our Version Control Tool which will allow us to run git commands on the terminal to upload our app.

III.3 Add a ‘requirements.text’ file

The cloud platform will need to know the python packages to install before they can start your app. We will specify that in our requirements.txtfile.



streamlit


pandas

III.4 Deploy your App to Streamlit Cloud

We will be using Streamlit to make our website public

III.4.1 Set up a GitHub Account

First, create a GitHub account here: Click here

III.4.2 Create a New GitHub Repository

In the upper-right corner of any page, use the drop-down + menu, and select New repository.

Give the repo name you want repository name, -> click on Create repository. For now, no need to change any parameters, we will go with the default parameters.

III.4.3 Upload Files to your GitHub Repository

Now click on ‘uploading an existing file’.

Select your app.py and requirements.txt files inside the following page.

III.4.4 Set up a Streamlit Cloud Account

First we need to create a streamlit account. To do create a Streamlit cloud account here: Click

III.4.5 Create a New App and Link your GitHub Account

You will see a prominently visible "New app" button after logging in, which is self-explanatory once you see it.
Also, you will get a prompt asking you to "Connect to GitHub". Click on it and log into your previously created GitHub account to continue.

III.4.6 Deploy your App

On the next screen, search for the GitHub repository that you just created. Type the repo name in the area -> Repository, which is specifically for this purpose.
In the field -> Main file path change to app.py
Click the 'Deploy!' button.

III.4.7 Your Public App is Now Live on Streamlit Cloud!

After waiting for some time, your app will appear. Congrats! Here’s mine: https://share.streamlit.io/apotitech/translatorGPT/main/TranslatorGPT.py

Congratulations!!!!!! You have created your very first app.

Conclusion

Great job creating your very first app. Please go ahead and share your new app to others and let them test your app.

With that said ----------------> Happy Building!!

Thank you for taking the time to read this! If you like the article, please clap (up to 50 times!) and connect with me on LinkedIn , dev.to and Medium to remain up to speed on my future articles. 😅

LocalStack: Emulate AWS Services for Local Development & Testing

Lionel♾️☁️ — Thu, 24 Aug 2023 07:06:19 +0000

It can be time-consuming, difficult, and even dangerous to create and test cloud-based apps in a production setting. This is when the significance of regional growth becomes apparent. Because it takes place on the developer's own system, local development helps keep costs down, makes debugging easier, and shortens development times.

-> Introducing LocalStack

The best, if not one of the best local development tools is LocalStack, which is based on AWS (Amazon Web Services). LocalStack creates a fully working local AWS cloud stack, which can do some offline development and testing of our cloud and Serverless apps.

Local stack is user-friendly and provides a testing environment on your local system that mimics the APIs and behaviors of AWS cloud Services. Now you can create and test your AWS applications without spending money or requiring access to the internet.

Docker's Importance

Docker is an essential part of this infrastructure. Docker is an open-source technology that employs containerized software distribution through virtualization at the operating system level.

This will not only guarantee that the application will function correctly in any setting because each container is self-contained and includes all necessary software, libraries, and system utilities but also that it is all working locally. Docker offers the containerization used by LocalStack to simulate the AWS cloud on your local workstation.

Getting started

Initializing Docker

We need to make sure Docker is functional before we can launch LocalStack. Starting up Docker varies from one Operating system to another.

MacOS: Docker can be launched in the background on MacOS by typing open --background -a Docker. Docker be launched in the background.
Linux: Docker is typically deployed on Linux as a service. The command sudo service docker start can be used to start it.
Windows: On Windows, Docker can be initialized from the Start menu or with a command like Start-Process -NoNewWindow "C:\Program Files\Docker\Docker\Docker Desktop.exe" in PowerShell.

It's crucial to verify Docker's continued operation after starting it. The command docker system info can be used for this purpose. Information about the Docker system should be returned from this command if Docker is functioning properly. If Docker is not running, the command will not return any output.

How to Get LocalStack Up and Running

LocalStack can be installed and launched once Docker is up and running. But the installation procedure is different for each OS.

Using the brew install localstack command, the Homebrew package manager may be used to set up LocalStack on MacOS.
Using the sudo apt-get install localstack command, LocalStack may be set up on Linux machines with the help of the APT package management.
The LocalStack package manager is available for Windows and may be installed using the command choco install localstack.

Once LocalStack has been installed, you can launch it with the command localstack start.

Using a Shell Script to Fully Automate Everything

To simplify matters, we can write a shell script to determine the OS, launch Docker, deploy LocalStack, and start it up. The following script must be run with chmod +x localstack.sh &&./localstack.sh:

#!/bin/bash

# Function to start Docker and ensure it's running on macOS
start_docker_mac() {
    echo "Starting Docker on macOS..."
    open --background -a Docker
    while ! docker system info > /dev/null 2>&1; do sleep 1; done
    echo "Docker is running."
}

# Function to start Docker and ensure it's running on Linux
start_docker_linux() {
    echo "Starting Docker on Linux..."
    sudo service docker start
    while ! docker system info > /dev/null 2>&1; do sleep 1; done
    echo "Docker is running."
}

# Function to start Docker and ensure it's running on Windows
start_docker_windows() {
    echo "Starting Docker on Windows..."
    Start-Process -NoNewWindow "C:\Program Files\Docker\Docker\Docker Desktop.exe"
    while (!(Test-Connection localhost -count 1)) { Start-Sleep -Seconds 1 }
    echo "Docker is running."
}

# Function to install LocalStack on macOS
install_mac() {
    echo "Installing LocalStack on macOS..."
    brew install localstack
}

# Function to install LocalStack on Linux
install_linux() {
    echo "Installing LocalStack on Linux..."
    sudo apt-get install localstack
}

# Function to install LocalStack on Windows
install_windows() {
    echo "Installing LocalStack on Windows..."
    choco install localstack
}

# Detect the operating system
OS="$(uname)"
case $OS in
  'Linux')
    start_docker_linux
    install_linux
    ;;
  'Darwin') 
    start_docker_mac
    install_mac
    ;;
  'WindowsNT')
    OS='Windows'
    start_docker_windows
    install_windows
    ;;
  *) ;;
esac

# Start LocalStack
localstack start

This script above defines functions that,

Depending on the operating system, will either begin running Docker or install LocalStack.
It then determines the operating system and calls the functions that are relevant to that system.
Finally, it activates the LocalStack.

Please be aware that running this script requires that you already have Docker and the necessary package manager (Homebrew for MacOS, APT for Linux, and Chocolatey for Windows) installed on your computer. In the event that they are not already installed, you will be required to do so before running this script.

Conclusion

The combination of LocalStack with Docker makes for a great tool for developing locally on AWS. Emulating the environment of Amazon Web Services (AWS) locally enables developers to improve their productivity, cut expenses, and avoid the dangers that are associated with building software directly in a live cloud environment.

You can make it even simpler to get your local AWS cloud environment up and running by automating the process of starting Docker, installing LocalStack, and running LocalStack with the help of a straightforward shell script. This will allow you to save time.

K8S Quickstart & Helm

Lionel♾️☁️ — Mon, 21 Aug 2023 21:16:00 +0000

Today, Kubernetes becomes a must for DevOps Engineers, SRE and others for orchestrating containers. Once you have a Docker image of your application, you have to code some YAML manifests to define Kubernetes workloads after which, you deploy them with the kubectl command.

This deployment way is when you’ve only one application. When you start to have many applications and multiple environments it becomes overwhelmed. Often you define the same YAML files 90% of the time.

Here, we are going to focus on how to manage applications smartly with Helm.

What Is Helm?

Helm is a package manager for Kubernetes. Helm is an open-source project originally created by DeisLabs and donated to the Cloud Native Foundation (CNCF). The CNCF now maintains and has graduated the project. This means that it is mature and not just a fad.

Package management is not a new concept in the software industry. On Linux distros, you manage software installation and removal with package managers such as YUM/RPM or APT. On Windows, you can use Chocolatey or Homebrew on Mac.

Helm lets you package and deploy complete applications in Kubernetes. A package is called a “Chart”. Helm uses a templating system based on Go template to render Kubernetes manifests from charts. A chart is a consistent structure separating templates and values.

As a package, a chart can also manage dependencies with other charts. For example, if your application needs a MySQL database to work you can include the chart as a dependency. When Helm runs at the top level of the chart directory it installs whole dependencies. You have just a single command to render and release your application to Kubernetes.

Helm charts use versions to track changes in your manifests — thus you can install a specific chart version for specific infrastructure configurations. Helm keeps a release history of all deployed charts in a dedicated workspace. This makes easier application updates and rollbacks if something wrong happens.

Helm allows you to compress charts. The result of that is an artifact comparable to a Docker image. Then, you can send it to a distant repository for reusability and sharing.

What Are the Benefits of Using Helm?

Helm provides you the ability to install applications with a single command. A chart can contain other charts as dependencies. You can consequently deploy an entire stack with Helm. You can use Helm like docker-compose but for Kubernetes.
A chart includes templates for various Kubernetes resources to form a complete application. This reduces the microservices complexity and simplifies their management in Kubernetes.
Charts can be compressed and sent to a distant repository. This creates an application artifact for Kubernetes. You can also fetch and deploy existing Helm charts from repositories. This is a strong point for reusability and sharing.
Helm maintains a history of deployed release versions in the Helm workspace. When something goes wrong, rolling back to a previous version is simply — canary release is facilitated with Helm for zero-downtime deployments.
Helm makes the deployment highly configurable. Applications can be customized on the fly during the deployment. By changing parameters, you can use the same chart for multiple environments such as dev, staging, and production.
Streamline CI/CD pipelines — Forward GitOps best practices.

Quick Look On The Problem Helm Solves

Basic Kubernetes practice is to write YAML manifests manually. We’ll create minimum YAML files to deploy NGINX in Kubernetes.

Here is the Deployment that will create Pods:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: nginx  
spec:  
  selector:  
    matchLabels:  
      app: nginx  
  replicas: 1  
  template:  
    metadata:  
      labels:  
        app: nginx  
    spec:  
      containers:  
      - name: nginx  
        image: nginx:1.21.6  
        ports:  
        - containerPort: 80

The Service exposes NGINX to the outside. The link with pod is done via the selector:

apiVersion: v1  
kind: Service  
metadata:  
  name: nginx  
spec:  
  selector:  
    app: nginx  
  ports:  
    - protocol: TCP  
      port: 80  
      targetPort: 8080

Kubernetes service for NGINX: service.yaml

Now we have to create the previous resources with the kubectl command:

$ kubectl create -f deployment.yaml  
$ kubectl create -f service.yaml

We check all resources are up and running:

$ kubectl get deployment -l app=nginx  
NAME    READY   UP-TO-DATE   AVAILABLE   AGE  
nginx   1/1     1            1           8m29s

$ kubectl get pods -l app=nginx                                                                                        
NAME                     READY   STATUS    RESTARTS   AGE  
nginx-65b89996ff-dcfs9   1/1     Running   0          2m26s

$ kubectl get svc -l app=nginx   
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE  
nginx   ClusterIP   10.106.79.171   <none>        80/TCP    4m58s

Issues with this method

Specific values in YAML manifests are hardcoded and not reusable.
Redundant information to specify such as labels and selectors leads to potential errors.
Kubectl does not handle potential errors after execution. You’ve to deploy each file one after the other.
There’s no change traceability.

Create a Helm Chart From Scratch

Helm can create the chart structure in a single command line:

$ helm create nginx

Understand the Helm chart structure

Chart.yaml: A YAML file containing information about the chart.
charts: A directory containing any charts upon which this chart depends on.
templates: this is where Helm finds the YAML definitions for your Services, Deployments, and other Kubernetes objects. You can add or replace the generated YAML files for your own.
templates/NOTES.txt: This is a templated, plaintext file that gets printed out after the chart is successfully deployed. This is a useful place to briefly describe the next steps for using the chart.
templates/_helpers.tpl: That file is the default location for template partials. Files whose name begins with an underscore are assumed to not have a manifest inside. These files are not rendered to Kubernetes object definitions but are available everywhere within other chart templates for use.
templates/tests: tests that validate that your chart works as expected when it is installed
values.yaml: The default configuration values for this chart

Customize the templates

The values.yaml is loaded automatically by default when deploying the chart. Here we set the image tag to 1.21.5 :

Please note that You can specify a specific values.yaml file to customize the deployment for environment-specific settings.

Install The Helm Chart

Good advice before deploying a Helm chart is to run the linter if you made an update:

$ helm lint nginx  
==> Linting nginx  
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

Run Helm to install the chart in dry-run and debug mode to ensure all is ok:

$ helm install --debug --dry-run nginx nginx

Using helm linter and dry-run install with debug mode will save you precious time in your development.

To install the chart, remove the --dry-run flag:

You can see the templated content of the NOTES.txt explaining how to connect to the application.

Now, you can retrieve the release in the Helm workspace:

Upgrade The Helm Release

Imagine you want to upgrade the container image to 1.21.6 for testing purposes.

Instead of creating a new values.yaml, we'll change the setting from the command line.

The pod is using the new container image as well:

The upgrade is visible in the chart history:

Change is inspectable with helm diff:

$ helm diff revision nginx 1 2  
default, nginx, Deployment (apps) has changed:  
  # Source: nginx/templates/deployment.yaml  
  apiVersion: apps/v1  
  kind: Deployment  
  metadata:  
    name: nginx  
    labels:  
      helm.sh/chart: nginx-0.1.0  
      app.kubernetes.io/name: nginx  
      app.kubernetes.io/instance: nginx  
      app.kubernetes.io/version: "1.0.0"  
      app.kubernetes.io/managed-by: Helm  
  spec:  
    replicas: 1  
    selector:  
      matchLabels:  
        app.kubernetes.io/name: nginx  
        app.kubernetes.io/instance: nginx  
    template:  
      metadata:  
        labels:  
          app.kubernetes.io/name: nginx  
          app.kubernetes.io/instance: nginx  
      spec:  
        serviceAccountName: nginx  
        securityContext:  
          {}  
        containers:  
          - name: nginx  
            securityContext:  
              {}  
-           image: "nginx:1.21.5"  
+           image: "nginx:1.21.6"  
            imagePullPolicy: IfNotPresent  
            ports:  
              - name: http  
                containerPort: 80  
                protocol: TCP  
            livenessProbe:  
              httpGet:  
                path: /  
                port: http  
            readinessProbe:  
              httpGet:  
                path: /  
                port: http  
            resources:  
              {}

Rollback The Helm Release

The upgrade was not conclusive and you want to go back. As Helm keeps all the changes, rollback is very straightforward:

$ helm rollback nginx 1  
Rollback was a success! Happy Helming!

The pod is now back to 1.21.5 container image:

Uninstall The Helm Chart

Uninstalling a Helm chart is trivial as the installation:

$ helm uninstall nginx

Reuse Existing Helm Charts

A lot of famous projects provide Helm chart to make the integration more user-friendly. They provide the charts through a repository. You have just to add it on your side:

$ helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami)

Once added, update your local cache to synchronize info with remote repositories:

$ helm repo update

You can now install the chart on your Kubernetes cluster:

$ helm install nginx bitnami/nginx

Charts are deployed with default values. You can inspire and specify a custom values.yamlto match your needs!

$ helm install my-release bitnami/nginx -f values.yaml

That’s all folks. Today we have looked at how to use Helm.

Please stay tuned and subscribe for more articles and study materials on DevOps, Agile, DevSecOps and App Development.

If you’d like to learn more about Infrastructure as Code, or other modern technology approaches, Please read or other articles.

🚀 Your Guide to Prometheus Monitoring on Kubernetes with Grafana

Lionel♾️☁️ — Sun, 20 Aug 2023 01:54:06 +0000

Introduction:

Hey fam🌟 In the fast-changing tech world of today, keeping an eye on the health of your apps has become the key to a smooth user experience. What do you know? Kubernetes is here to help you handle containers at scale as your trusted helper. But how do you keep track of all these bits of code that are flying around? Here come Prometheus and Grafana, a powerful pair that turns data into superhero insights.

Prometheus 📈

It is your measurements guru. It's an open-source wizard that not only gets data from your apps and services, but also adds some alerting magic. It's like having your own treasure chest full of info.

Open source monitoring tool
Out-of-the-box monitoring capabilities for the Kubernetes
It collects and stores metrics as time-series data, recording information with a timestamp
Works by pulling and collecting metrics from targets by scraping metrics HTTP endpoints.

Next, meet

Grafana 📊

It is your visual storyteller. It uses Prometheus data to create eye-catching visualizations. Think of your data as vibrant graphs that tell the story of your apps' performance and alert you to potential problems before they arise.

Open source visualization and analytics software.
It helps you to query, visualize, alert on, and even explore your metrics.

In this guide, we will set up Prometheus to watch your Kubernetes cluster and invite Grafana to the party. Whether you're an experienced Kubernetes user or just starting out, we've got you covered with everything you need to know to set up a rock-solid tracking system.

So our Key Components are

Key components:

Prometheus server — Processes and stores all your metrics data
Alert Manager — The manager sends alerts to any systems/channels
Grafana — Visualize all scraped data in your UI

Let's Go!!!!!!!!!!!!!!!!!

Installing

There are many ways to setup Prometheus and Grafana. Let us look at some of them:

Set up your Prometheus and Grafana configuration files and run them in the correct order.
Prometheus Operator: Used to streamline and automate the administration of your Prometheus monitoring stack in your Kubernetes environment.
Helm Chart (Recommended) : Using Helm Chart, set up and Prometheus Operator which includes Grafana.

Why Helm ?

Helm is a package manager for Kubernetes. In other words, it simplifies the setting up and installation of all the components of an installation, all in one command. It is recommended to use Helm as it will take care of all the config steps and you would not miss any.

Helm has three significant benefits that it adds to the process of app deployments to the Kubernetes cluster:

Speed of Deployment – It helps us speed up our speed of app deployments with a single command.
Using prebuilt application configurations – Whatever the config that you need for your infrastructure, someone else already has a prebuilt application configurations that can be used.
Easy rollbacks – Last but not least, Helm makes it more easy for us to upgrade and rollback the versions of our apps.

So let us look at our prerequisites

Pre-requisites

Setup Kubernetes (Using Kubeadm)
Install Helm Package manager
Download the Helm charts for setting up Prometheus and Grafana

Let us Get Started!

Creating our Infrastructure

We will be setting up our Kubernetes cluster on AWS, using Kubeadm. I will be using two (2) t2.medium ec2 instances for this installation.

Installing Kubernetes using Kubeadm

Step 1

We will ssh into the 2 VMs (the one we will use as MASTER & WORKER node) and run some commands there to configure the environment.

apt-get update -y  

apt-get install docker.io -y  

service docker restart  

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -  

echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" >/etc/apt/sources.list.d/kubernetes.list  

apt-get update

Step 2

Now we need to configure the MASTER NODE. So now we will ssh into it and run some commands

kubeadm init --pod-network-cidr=192.168.0.0/16  

# If above one fails then run below command  

kubeadm token create --print-join-command

Step 3

Let us proceed with our installation in the MASTER NODE

mkdir -p $HOME/.kube  

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  

sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 4

Next, we need to run some kubectl commands on our Master Node to complete the installation

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/calico.yaml  


kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.49.0/deploy/static/provider/baremetal/deploy.yaml  

(after that check "kubectl get nodes" command on master node you can see the worker node configured with master)

Let us look at what it looks like now

Installing Helm

Now we can install Helm on our master node. For that, you can run these commands.

curl -fsSL -o get_helm.sh \ https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 

chmod 700 get_helm.sh 

./get_helm.sh

Setting up our Monitoring Environment

Now that we have installed Helm Package Manager we need to add Helm Stable Charts for our local machine.

helm repo add stable https://charts.helm.sh/stable

Next, we will be adding the Helm repository of prometheus to our local machine

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Next we need to create the namespace where we will install prometheus

kubectl create namespace prometheus

We can now go ahead and install our kube-prometheus stack. The kube-stack-prometheus will be installed with Helm, with a grafana deployment embedded.

helm install stable prometheus-community/kube-prometheus-stack --version 48.3.1 -n prometheus

-n is added to specify the namespace where you want the installation to be done.

Having installed it ... this is the screen we will have next

Great Job!!!!! It is successfully installed. Now let us go ahead and check our pod resources

kubectl get pods -n prometheus

kubectl get svc -n prometheus

These all show that broth Prometheus and Grafana have been successfully installed.

Enabling external access to our Infrastructure

When a Kubernetes cluster has been created, Cluster IP is usually created. However, for us to be able to enable external access, we need to have a LoadBalancer or NodePort service.

For that, we will therefore need to edit the Prometheus and Grafana service. Run the following command

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

kubectl edit svc stable-grafana -n prometheus

Once you finish editing both services, make sure that you close and save the file -> Escape :wq

Now we should have 2 loadbalancers in our Cluster. Let us check that

Now we can access Grafana. So copy the Loadbalancer link.

Configuring our Grafana server

Grafana dashboards give you instant visual insights, making complex data easier to understand. They also let you watch in real time and help you make decisions based on data.

Discovering the password 🔍

To login to your grafana account, we will use the default username and password. The default user is admin and the password can be found with the following command

kubectl get secrets

You will be see different secrets as below

Mine is stable-grafana. Replace that with yours.

kubectl get secret stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Please note that stable-grafana is the name of the grafana secret on my deployment. Yours could be different.

We can create different kinds of dashboards, depending on our needs.

Grafana dashboards are very easy to create. Let us go through the steps to create our dashboard. In your Grafana server, go ahead and create our dashboards.

1. Creating Kubernetes Monitoring Dashboard

Click + button on left panel and select ‘Import’
Enter 12740 dashboard id under Grafana.com Dashboard
Click ‘Load’
Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
Click ‘Import’

This DASHBOARD will show monitoring dashboard for all the nodes in our cluster.

2. Creating Kubernetes Cluster Monitoring Dashboard

Click + button on left panel and select ‘Import’
Enter 3119 dashboard id under Grafana.com Dashboard

Click ‘Load’
Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
Click ‘Import’

This DASHBOARD will show monitoring dashboard for all the nodes in our cluster.

3. Creating POD Monitoring Dashboard

Click + button on left panel and select ‘Import’
Enter 6417 dashboard id under Grafana.com Dashboard
Click ‘Load’
Select ‘Prometheus’ as the endpoint under Prometheus Data Sources drop down.
Click ‘Import’

Congratulations!!!!!!!!!

Conclusion

Your journey to master Prometheus Monitoring on Kubernetes using Grafana is complete! 🚀 You can effortlessly gather, visualise, and analyse metrics from your dynamic Kubernetes system thanks to this hands-on tutorial.

By combining Prometheus' data collection strength with Grafana's stunning dashboards, you can confidently sail containerized application into the seas. From resource utilization to anomaly detection and alert setup, you've learned key skills for monitoring system health and performance.

Prometheus and Grafana help you optimise, debug, and innovate with Kubernetes and monitoring. Hope your monitoring trip is informative and your applications sparkle! 🌟🌐📊

Docker Security: Clair

Lionel♾️☁️ — Tue, 08 Aug 2023 19:30:00 +0000

Docker offers an amazing solution for packaging applications along with their dependencies. You can find the application itself, its supporting elements like Maven or NPM packages, the base Operating System, and other necessary tools like Java and NodeJS in Docker images. By creating a build pipeline, you can very easily create a Docker image whenever there are code changes. The best part is that docker images contain all the dependencies, allowing you to deploy them on any platform that supports Docker, be it an on-prem Kubernetes setup or a cloud-based platform like AWS ECS.

Docker Image Scanning

Since docker images contain all the required OS files for the application to run, it is very essential to carefully examine all the packages installed in the docker image and be sure that there are no vulnerabilities. Docker's philosophy is creating lightweight and minimal containers that serve a specific purpose. In other words, it is recommended to always create separate images for different applications. This approach perfectly aligns with the microservice ecosystem, where each microservice has its own dedicated docker image.

Image Scanning in Registries

The easiest way of scanning docker images is scanning them inside of registries. Both Dockerhub and Quay offer built-in image scanning capabilities, but there are a few limitations to keep in mind. Currently, DockerHub's scanning feature is only available for private repositories by default. On the other hand, Quay's image security scanning is exclusive to Quay Enterprise.

Clair - Open Source Image Scanner

Clair, developed by CoreOS, is a fantastic open source vulnerability scanner specifically designed for docker images. It is capable of gathering vulnerabilities from various vulnerability databases for different operating systems like Debian, Ubuntu, Red Hat, Alpine, and Oracle Linux. The best part is that Clair can be easily obtained as a docker image, enabling one-off scans to be seamlessly integrated into the build pipeline. However, when running Clair for the first time, it needs to download vulnerability information, which can be quite time-consuming, taking around 20-30 minutes. This delay becomes problematic when implementing a continuous integration and continuous deployment (CICD) pipeline.

The arminc/clair-db image on Docker Hub solves this. It runs a daily build of the vulnerability database and creates pre-populated database images that are ready to use right away. By utilizing this pre-built database, the need for time-consuming downloads during the initial setup of Clair is eliminated. This makes it incredibly convenient for seamless integration into CICD pipelines.

To run the pre-built database, simply follow these steps:



docker run -d --name db arminc/clair-db

Clair requires a config.yaml file that contains config like the DB password to get started. Another docker image, still from arminc solves this, providing a default config which is embedded in Clair. The command below can be run to connect with our db.



docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan

How Clair works

Clair exposes an API to scan the individual docker layers. To be able to run a scan, a Clair client is needed which can do the job. A useful tool for this is Klar, which is a popular CLI client written in Go language which can run point and shoot scans. However, it needs the image to be scanned to be inside of a registry already and cannot scan images that are locally stored. To use image scanning in a CICD pipeline, it is always better to run scans locally on an image before pushing it to the registry. Having results available before release allows us to choose whether we can still proceed and push that image or break the build based on the number and severity of the vulnerabilities in the docker image. An alternative, that can still help us run local scans is Clair-Scanner.

We can run clair-runner tool by providing it the IP of the docker bridge gateway to run a local scan, by running this command:



# get docker gateway IP
DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
# download scanner cli
wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 && chmod +x clair-scanner
# run scan on local image - dubu in this case
./clair-scanner --ip="$DOCKER_GATEWAY" dubu:latest

Running a local scan

To be able to run a local scan on a docker image before pushing it. You can use this docker-compose.yml script



version: '3'

services:
  db:
    image: arminc/clair-db
    container_name: clairdb
    restart: always

  clair:
    image: arminc/clair-local-scan
    container_name: clairlocal
    depends_on:
      - db
    ports:
      - "6060:6060"
    restart: always

By using docker-compose, it streamlines the process and ensures that the linked containers (if we create more than 1) can communicate seamlessly. Now let's look at our bash script that can automate this



#!/usr/bin/env bash

# Function to check if a command executed successfully
check_command() {
    if [ $? -ne 0 ]; then
        echo "Error executing the previous command!"
        exit 1
    fi
}

# Start the Clair and database containers
docker-compose up -d
check_command

# Give some time for the database to initialize. 
# Note: It's better to have a health check to confirm when the db is ready, but for simplicity, we use sleep here.
sleep 20

# Download and prepare the clair-scanner
if [ ! -f clair-scanner ]; then
    wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
    check_command
    chmod +x clair-scanner
fi

# Get Docker's bridge network gateway
DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
check_command

# Scan the specified Docker image
./clair-scanner --ip="$DOCKER_GATEWAY" dubu:latest
check_command

Let us break down the bash script

Function check_command: This function checks the exit status of the last executed command. If the command failed (i.e., didn't return a status of 0), it outputs an error message and exits the script. This provides better error handling throughout your script.
Download Check: The script checks if clair-scanner already exists before attempting to download it. This can save time if you're running the script multiple times.
Health Check Note: I've added a note in the script to emphasize that using a health check for the database would be a more robust solution than just waiting a set amount of time with sleep.

Please note that after copying this script, to execute it, you need to make sure it has execute permissions (chmod +x your_script_name.sh) and then run it (./your_script_name.sh).

Conclusion: CICD Integration

The whole plan is for us to be able to integrate image scanning with out CICD pipeline. To do this, you can use this script below as part of your Jenkins pipeline.



stage("docker_scan"){
    steps {
        script {
            // Ensure the containers are not already running (cleanup from a previous run).
            sh 'docker rm -f db || true'
            sh 'docker rm -f clair || true'

            // Start the Clair database and wait for it to initialize.
            sh 'docker run -d --name db arminc/clair-db'
            sh 'sleep 15' // Consider a more robust health check here.

            // Start the Clair container.
            sh 'docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan'
            sh 'sleep 1'

            // Download the clair-scanner if not already present.
            sh '''
                if [ ! -f clair-scanner ]; then
                    wget -qO clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
                    chmod +x clair-scanner
                fi
            '''

            // Get Docker's bridge network gateway and run the scanner.
            sh '''
                DOCKER_GATEWAY=$(docker network inspect bridge --format "{{range .IPAM.Config}}{{.Gateway}}{{end}}")
                ./clair-scanner --ip="$DOCKER_GATEWAY" myapp:latest || exit 0
            '''
        }
    }
    post {
        always {
            // Clean up, remove the Clair and database containers.
            sh 'docker rm -f db'
            sh 'docker rm -f clair'
        }
    }
}

clair-scanner by default will break the build if any issues are found. In this code we ignore the exit code by appending || exit 0.

Let us take a look at out Jenkins build

Jenkins Security: Storing Secrets

Lionel♾️☁️ — Fri, 04 Aug 2023 19:20:00 +0000

Jenkins is by far one of my favorite open source tools. Jenkins is an open source automation that that helps and provides support for building and deploying apps. Jenkins Plugins are what enable it to be customized and extended.

You are new to Jenkins? Quickly try it out by running its docker image, with the command

docker run -p 8080:8080 jenkins/jenkins

CI/CD

Automation is fundamental to Continuous Integration and Countinuous Delivery. Jenkins build can be setup for Continuous Integration tasks such as running integration testing. And Continuous Delivery activities such as building a final jar and pushing it to a repository or building a Docker image and pushing it to a registry. However all such integrations require credentials for authorization. Applications tend have a lot of other secrets such as Certificates for authentication for MASSL, credentials for upstream systems, databases and API tokens.

We cannot talk of Continuous Integration and Continuous Delivery (CI/CD) and not talk about automation. Jenkins is a CI tool and its builds can be configured to run Continuous Integration tasks such software integration testing. Not only that but Continuous Delivery actions such as creating a final jar (java app artifact) and pushing it to a repository or even creating a Docker image and pushing it to a registry are examples of Continuous Delivery activities. All of these integrations however need credentials to be able to work. Certificates for authentication for MASSL credentials for upstream systems, databases, and API tokens are common secrets in applications.

Jenkins Plugins for Secret Management

Below are some plugins that allow basic credential management:

Credentials Binding Plugin: These allow credentials to be bound to environment variables for use in build steps.
Credentials Plugin: On the other hand, this plugin allows you to store your credentials in Jenkins.

N.B. These plugins are available in default installation of Jenkins.

Below are secrets which are supported by default:

Using secrets in Freestyle Job

For our test, let’s create a Free style Jenkins Project and use Credentials plugins to store our Dockerhub credentials. As seen below, select Username and password (separated) and enter bindings for the credentials. Basically, these are the environment variables that username and password will be available as:

Once that is done, next click on Add and select the right scope and enter Dockerhub credentials:

Now here is an example of how we use our credentials

Using Credentials in Jenkins Pipeline

Now let us look how we can use our credentials in Jenkins Pipeline. In the following stage of our Jenkinsfile we are will be using artifactory-credentials to login to the artifactory and push our docker images to it. Also note that usernameVariable and passwordVariable are available in the withCredentials block:



    stage("docker_push") {
      withCredentials([usernamePassword(credentialsId: 'artifactory-credentials', 
        passwordVariable: 'ARTIFACTORY_KEY', 
        usernameVariable: 'ARTIFACTORY_USER')]) 
      {
        sh "echo $ARTIFACTORY_KEY | docker login -u $ARTIFACTORY_USER --password-stdin ${REGISTRY_URL}"
        sh "docker tag myapp:latest ${REGISTRY_URL}/myapp:${shortCommit}"
        sh "docker push ${REGISTRY_URL}/myapp:${shortCommit}"
      }

Retrieving Secrets

Jenkins tries to provide some sense of security by masking the credentials in logs. It does that by looking for an exact match and replaces them with asterisks (*****). But note also that this control can simply be bypassed by a user with edit permission by encoding the credentials in the pipeline and printing them in logs.

Please note that the first echo results in our password being masked since it matches our password string.

Now let us decode the encoded password and see what it gives us:



dev@ubu:~ $ echo YWRtaW4xMjM0Cg== | base64 -d
admin1234

Let us explain this. Actually, Jenkins credentials are stored in Jenkins master encrypted by Jenkins instance id. This file usually exists in your linux server at /var/lib/jenkins/credentials.xml. It is therefore very easy to decrypt credentials saved in Jenkins for Jenkins administrators. Moreover, do note that any user that can SSH to Jenkins can read the encrypted credentials file since it has read permissions for everyone. Let us look at that:



dev@ubu:/var/lib/jenkins$ ls -la credentials.xml 
-rw-r--r-- 1 jenkins jenkins 4650 Mar 26 09:28 credentials.xml

This file has passwords encrypted with Jenkins Id:



<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
    <scope>GLOBAL</scope>
    <id>artifactory-credentials</id>
    <description></description>
    <username>admin</username>
    <password>{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>

Our favorite CI-tool, Jenkins provides a handy utility at /script of the URL that can be used to decrypt passwords with the following script:



encryptedPassword = '{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}'
passwd = hudson.util.Secret.decrypt(encryptedPassword)
println(passwd)

In Conclusion, it is very clear that secrets entered in Jenkins will not be disclosed. Any user that can use a secret in a build can decode and see it. All and any user that can SSH into Jenkins master server can decrypt all secrets using /script utility if they can login to Jenkins Web UI.

Management Issues with Hardcoding Credentials

As we just saw, the principle of least privilege cannot be effectively employed when just using Jenkins Credentials Plugin. Therefore, there has to be a better solution for auditable credential sharing within team, credential rotation and automatic build provisioning that we can use. For that, I would suggest a number of credential management products such as Hashicorp Vault, Cyberark Vault and AWS Secrets Manager. Some open source products like CredStash for AWS help with credential management.

Forem: Lionel♾️☁️

End to End CI/CD pipeline using GitHub Actions for Android Application

Project Source Code : LINK

Step by Step Process :

Installing the service

Starting the service

Checking the status of the service

DevOps from 0 to Hero - for Freshers

Introduction

📚 Step-by-Step Guide

1. Understand the Basics

1.1 What is DevOps?

1.2 Core DevOps Principles

2. 🔧 Learn the Foundation Skills

2.1 Basic Programming

2.2 Operating Systems

2.3 Networking Basics

3. 🌿 Dive into Version Control

3.1 Git

3.2 GitHub

4. 🔄 Master Continuous Integration and Continuous Delivery (CI/CD)

4.1 Jenkins

4.2 GitLab CI/CD

5. Explore Configuration Management

5.1 Ansible

5.2 Puppet

6. 🐳 Understand Containerization and Orchestration

6.1 Docker

6.2 Kubernetes

7. ☁️ Explore Cloud Platforms

7.1 AWS (Amazon Web Services)

7.2 Azure

7.3 Google Cloud Platform (GCP)

8. Learn Infrastructure as Code (IaC)

8.1 Terraform

9. 📊 Implement Monitoring and Logging

9.1 Prometheus

9.2 ELK Stack (Elasticsearch, Logstash, Kibana)

10. Get Hands-On Experience

10.1 Build Projects

11. 🤝 Join the Community

12. 📚 Continuous Learning

12.1 Online Courses

12.2 Books

📚 Security in DevOps (DevSecOps)

Career Paths in DevOps

Further Reading

DevOps Podcasts and YouTube Channels

Podcasts:

YouTube Channels:

A Day in the Life of a DevOps Engineer

Conclusion

Join Our Telegram Community || Follow us for more DevOps Content.

Deploying Django Application on AWS with Terraform - Part 1

Parts of Project

Minimal Working Setup

Creating Django project

Creating Terraform Project

Resources we will use

Elastic Container Repository, ECR

Network

Load Balancer

Application

Congratulations

Thank you so much everyone!

How Netflix Uses the Cloud - AWS

How they Use AWS Services

1. The Netflix Magic 🎬🍿

Step 1: Metadata and Shows ✨

Step 2: Uploading and Process Videos! 🎥➡️☁️

Step 3: AI Content Analysis in Action! 🔍🤖

Step 4: AWS Step Functions for Parallel Processing 🤹⚙️

Step 5: Smart Storage Frequent vs Infrequent Access 📦🔄

Step 6: Deep Freeze Dive for Rare Gems 🏔️❄️

Step 7: Netflix and AWS CDN (Open Connect) 🚀🎥

Step 8: Diving Deep with Data Magic 🔍✨

2. Netflix and us, viewers 🌍✨

Step 1: One Service, Many Screens 📺📱💻🎮

Step 2: A Dazzling Dance of Design and Delivery 🌐💃

Step 3: Searching for a show 🔍💎