Forem: Christian Ameachi

[Boost]

Christian Ameachi — Sun, 22 Feb 2026 11:51:59 +0000

Building a Secure and Observable E-commerce Microservices Platform on AWS EKS

Christian Ameachi ・ Feb 22

Building a Secure and Observable E-commerce Microservices Platform on AWS EKS

Christian Ameachi — Sun, 22 Feb 2026 11:50:16 +0000

Introduction

Building a production-ready microservices architecture involves more than just writing code. It requires a robust delivery pipeline, automated infrastructure, and deep observability. In my latest project, ShopMicro-Production, I set out to build a fully automated e-commerce engine deployed on Amazon EKS.

The Stack

The platform follows a classic microservices pattern:

Frontend: A sleek React/Vite interface served via Nginx.
Backend: A Node.js/Express API handling the core logic.
ML Service: A Python/Flask recommendation engine for intelligent product suggestions.
Data Layers: PostgreSQL for persistent storage and Redis for high-performance caching.

Infrastructure as Code (IaC)

One of the core principles of this project was "Everything as Code." I used Terraform to provision the entire AWS EKS cluster, including managed node groups and all necessary IAM roles.

Early on, I also experimented with Ansible to bootstrap self-managed Kubernetes nodes on EC2, which provided a deep understanding of control plane orchestration before moving to the managed EKS experience.

The CI/CD Engine

The automation is powered by GitHub Actions with four distinct pipelines:

App CI: Automatically runs linting and unit tests on every PR, then builds and pushes Docker images to GHCR.
App CD: Sequentially deploys services to EKS, ensuring dependencies like Redis and Postgres are ready before the apps start.
IaC CI: Validates Terraform code using tflint and ensures compliance with OPA (Open Policy Agent) policies.
Drift Detection: A daily automated check to ensure no manual changes have deviated from our Terraform source of truth.

Zero-Downtime Reliability

To ensure the system stays healthy:

HPA (Horizontal Pod Autoscaler): Automatically scales the backend and ML services based on CPU/Memory thresholds.
Rollback Proof: Implemented a "fail-safe" procedure where failed deployments can be reverted instantly using kubectl rollout undo.
Persistence: Fixed complex volume binding issues on EKS by implementing the AWS EBS CSI driver and custom PGDATA pathing.

Observability: The Full Stack

You can't fix what you can't see. I implemented the full LGTM stack (Loki, Grafana, Tempo, Metrics):

Metrics: Prometheus scraping service endpoints.
Logs: Loki aggregating distributed container logs.
Traces: Tempo providing end-to-end request tracing.
Visualize: Custom Grafana dashboards for a single pane of glass monitoring.

Conclusion

This project was a deep dive into the realities of cloud-native engineering. From handling stateful persistence in Kubernetes to enforcing policy-as-code, it taught me that the best systems are the ones that are both automated and transparent.

Check out the repo here: GitHub

[Boost]

Christian Ameachi — Fri, 09 Jan 2026 20:48:51 +0000

From Linux Primitives to Docker Swarm: A Deep Dive into Container Networking 🚀

Christian Ameachi ・ Jan 9

From Linux Primitives to Docker Swarm: A Deep Dive into Container Networking 🚀

Christian Ameachi — Fri, 09 Jan 2026 20:47:36 +0000

Have you ever wondered what actually happens under the hood when you run docker run? How do containers "talk" to each other while staying isolated?

Lately, I've been taking a "from scratch" approach to understanding container networking. I didn't start with Docker. I started with the Linux Kernel.

Here is the story of my journey through the layers of modern infrastructure.

🏗️ Phase 1: The Hard Way (Linux Primitives)

Before touching a single Dockerfile, I built a microservices environment using raw Linux commands.

The Ingredients:

Network Namespaces: To create isolated network stacks for each service.
Veth Pairs: Think of these as virtual "ethernet cables" connecting namespaces.
Linux Bridges: Acting as virtual switches to manage traffic between services.
Iptables & Routing: Manually configuring NAT and firewall rules to secure the frontend, backend, and database tiers.

The Lesson: Building the network manually teaches you exactly how much "magic" Docker does for us. It’s brittle and complex, but the performance is incredibly raw and fast.

🐳 Phase 2: Optimization with Docker

Next, I migrated the stack to Docker. But I didn't just wrap the apps; I optimized them for production-grade reliability.

Key Highlights:

Multi-Stage Builds: Using python:3.11-slim to keep image sizes small (under 150MB).
Embedded Healthchecks: Ensuring the API Gateway only routes traffic to "ready" services.
Resource Constraints: Pinning CPU and Memory (e.g., 0.5 vCPU, 128MB RAM) to prevent noisy neighbor scenarios.
Internal DNS: Leveraging Docker's embedded DNS for seamless service discovery.

🌐 Phase 3: The Distributed Horizon (Multi-Host Networking)

The final challenge was scaling across two separate Linux hosts. This is where things got really interesting.

The Strategy:

VXLAN Overlay: I manually established a VXLAN tunnel to bridge two different VM network segments.
Docker Swarm: I initialized a Swarm cluster and deployed our stack using the overlay network driver.

The Result: A resilient, self-healing system. Even when my Manager node faced storage issues, Swarm's orchestration automatically failed over the services to the healthy Worker node. That is the power of distributed systems.

📊 Performance Benchmark: Linux vs. Docker

I ran load tests using Apache Bench (AB) and the results were eye-opening:

Linux Namespaces: ~54 Requests Per Second (RPS)
Docker Compose: ~29 Requests Per Second (RPS)

While the raw Linux setup was ~46% faster, Docker's portability, ease of scaling, and management tools make it the clear winner for modern software engineering.

💡 Key Takeaway

Understanding the "low-level" doesn't mean you should always build from scratch—it means you know exactly what to do when your high-level tools break.

If you're a DevOps or Backend Engineer, I highly recommend spending a day building a bridge and a namespace from scratch. It will change how you look at your docker-compose.yml forever.

I've documented the entire process, including setup scripts and architecture diagrams, in my GitHub repo.

Check out the full project here: GitHub

Very simple and useful cli

Christian Ameachi — Sun, 23 Nov 2025 08:41:58 +0000

Building a Simple Ticket Tracker CLI in Go

Christian Ameachi ・ Nov 22

#cli #go #productivity #tooling

Building a Simple Ticket Tracker CLI in Go

Christian Ameachi — Sat, 22 Nov 2025 21:49:28 +0000

A lightweight, command-line alternative to complex ticketing systems.

Using golang and cobra-cli, I built a simple command-line interface for managing support tickets. Tickets are stored locally in a CSV file.

Introduction

As developers, we often find ourselves juggling multiple tasks, bugs, and feature requests. While tools like Jira, Trello, or GitHub Issues are powerful, sometimes you just need something simple, fast, and local to track your daily work without leaving the terminal.

That's why I built Ticket CLI—a simple command-line tool written in Go to track daily tickets and store them in a CSV file. No servers, no databases, just a binary and a text file.

The Tech Stack

For this project, I choose:

Go: For its speed, simplicity, and ability to compile into a single binary.
Cobra: The industry standard for building modern CLI applications in Go. It handles flag parsing, subcommands, and help text generation effortlessly.
Standard Library (encoding/csv): To keep dependencies low, I used Go's built-in CSV support for data persistence.

How It Works

The project follows a standard Go CLI structure:

ticket-cli/
├── cmd/            # Cobra commands (add, list, delete)
├── internal/       # Business logic
│   └── storage/    # CSV handling
└── main.go         # Entry point

1. The Command Structure

Using Cobra, I defined commands like add, list, and delete. Here's a snippet of how the add command handles flags to create a new ticket:

// cmd/add.go
var addCmd = &cobra.Command{
    Use:   "add",
    Short: "Add a new ticket",
    Run: func(cmd *cobra.Command, args []string) {
        // default values logic...

        t := storage.Ticket{
            ID:          fmt.Sprintf("%d", time.Now().UnixNano()),
            Title:       flagTitle,
            Customer:    flagCustomer,
            Priority:    flagPriority,
            Status:      flagStatus,
            Description: flagDescription,
        }

        if err := storage.AppendTicket(t); err != nil {
            fmt.Println("Error saving ticket:", err)
            return
        }
        fmt.Println("Ticket saved with ID:", t.ID)
    },
}

2. Data Persistence (The "Database")

Instead of setting up SQLite or a JSON store, I opted for CSV. It's human-readable and easy to debug. The internal/storage package handles reading and writing to tickets.csv.

// internal/storage/storage.go
func AppendTicket(t Ticket) error {
    // ... (file opening logic)
    w := csv.NewWriter(f)
    defer w.Flush()

    rec := []string{t.ID, t.Date, t.Title, t.Customer, t.Priority, t.Status, t.Description}
    return w.Write(rec)
}

Installation & Usage

You can clone the repo and build it yourself:

git clone https://github.com/yourusername/ticket-cli
cd ticket-cli
go mod tidy
go build -o ticket-cli .

Adding a Ticket

./ticket-cli add --title "Fix login bug" --priority high --customer "Acme Corp"

Listing Tickets

./ticket-cli list

Filter by date

./ticket-cli list --date 2025-11-15

CSV Storage

A CSV file is created automatically at the Project directory, and it keeps getting updated, once a new ticket is added using the .ticket-cli add --flags

Columns :

ID, Date, Title, Customer, Priority, Status, Description

Future Improvements

This is just an MVP. Some ideas for the future include:

JSON/SQLite Storage: For more complex querying.
TUI (Text User Interface): Using bubbletea for an interactive dashboard.
Cloud Sync: Syncing tickets to a Gist or S3 bucket.

Conclusion

Building CLI tools in Go is a rewarding experience. Cobra makes the interface professional, and Go's standard library handles the rest. If you're looking for a weekend project, try building your own developer tools!

Check out the code on GitHub.

Experience Continuous Integration with Jenkins | Ansible | Artifactory | SonarQube | PHP

Christian Ameachi — Sun, 25 Feb 2024 00:07:34 +0000

Experience Continuous Integration with Jenkins | Ansible | Artifactory | SonarQube | PHP

IMPORTANT NOTICE - This project has some initial theoretical concepts that must be well understood before moving on to the practical part. Please read it carefully as many times as needed to completely digest the most important and fundamental DevOps concepts. To successfully implement this project, it is crucial to grasp the importance of the entire CI/CD process, roles of each tool and success metrics - so we encourage you to thoroughly study the following theory until you feel comfortable explaining all the concepts (e.g., to your new junior colleague or during a job interview).

In previous projects, you have been deploying a tooling website directly into the /var/www/html directory on dev servers. Well, even though that worked fine, and we were able to access the website, it is not the best way to do it. Real world web application code written on Java, .NET or other compiled programming languages require a build stage to create an executable file. The executable file (e.g., jar file in case of Java) contains all the codes embedded, and the necessary library dependencies, which the application needs to run and work successfully.

Some other programs written languages such as PHP, JavaScript or Python work directly without being built into an executable file - these languages are called interpreted. That is why we could easily deploy the entire code from git into var/www/html and immediately the webserver was able to render the pages in a browser. However, it is not optimal to download code directly from Git onto our servers. There is a smarter way to package the entire application code, and track release versions. We can package the entire code and all its dependencies into an archive such as .tar.gz or .zip, so that it can be easily unpacked on a respective environment's servers.

For a better understanding of the difference between compiled vs interpreted programming languages - read this short article.

In this project, you will understand and get hands on experience around the entire concept around CI/CD from applications perspective. To fully gain real expertise around this idea, it is best to see it in action across different programming languages and from the platform perspective too. From the application perspective, we will be focusing on PHP here; there are more projects ahead that are based on Java, Node.js, .Net and Python. By the time you start working on Terraform, Docker and Kubernetes projects, you will get to see the platform perspective of CI/CD in action.

What is Continuous Integration?

In software engineering, Continuous Integration (CI) is a practice of merging all developers' working copies to a shared mainline (e.g., Git Repository or some other version control system) several times per day. Frequent merges reduce chances of any conflicts in code and allow to run tests more often to avoid massive rework if something goes wrong. This principle can be formulated as Commit early, push often.

The general idea behind multiple commits is to avoid what is generally considered as Merge Hell or Integration hell. When a new developer joins a new project, he or she must create a copy of the main codebase by starting a new feature branch from the mainline to develop his own features (in some organization or team, this could be called a develop, main or master branch). If there are tens of developers working on the same project, they will all have their own branches created from mainline at different points in time. Once they make a copy of the repository it starts drifting away from the mainline with every new merge of other developers' codes. If this lingers on for a very long time without reconciling the code, then this will cause a lot of code conflict or Merge Hell, as rightly said. Imagine such a hell from tens of developers or worse, hundreds. So, the best thing to do, is to continuously commit & push your code to the mainline. As many times as tens times per day. With this practice, you can avoid Merge Hell or Integration hell.

CI concept is not only about committing your code. There is a general workflow, let us start it...

Run tests locally: Before developers commit their code to a central repository, it is recommended to test the code locally. So, Test-Driven Development (TDD) approach is commonly used in combination with CI. Developers write tests for their code called unit-tests, and before they commit their work, they run their tests locally. This practice helps a team to avoid having one developer's work-in-progress code from breaking other developers' copy of the codebase.
Compile code in CI:

After testing codes locally, developers commit and push their work to a central repository. Rather than building the code into an executable locally, a dedicated CI server picks up the code and runs the build there. In this project we will use, already familiar to you, Jenkins as our CI server. Build happens either periodically - by polling the repository at some configured schedule, or after every commit. Having a CI server where builds run is a good practice for a team, as everyone has visibility into each commit and its corresponding builds.

Run further tests in CI:

Even though tests have been run locally by developers, it is important to run the unit-tests on the CI server as well. But, rather than focusing solely on unit-tests, there are other kinds of tests and code analysis that can be run using CI server. These are extremely critical to determining the overall quality of code being developed, how it interacts with other developers' work, and how vulnerable it is to attacks. A CI server can use different tools for Static Code Analysis, Code Coverage Analysis, Code smells Analysis, and Compliance Analysis. In addition, it can run other types of tests such as Integration and Penetration tests. Other tasks performed by a CI server include production of code documentation from the source code and facilitate manual quality assurance (QA) testing processes.

Deploy an artifact from CI:

At this stage, the difference between CI and CD is spelt out. As you now know, CI is Continuous Integration, which is everything we have been discussing so far. CD on the other hand is Continuous Delivery which ensures that software checked into the mainline is always ready to be deployed to users. The deployment here is manually triggered after certain QA tasks are passed successfully. There is another CD known as Continuous Deployment which is also about deploying the software to the users, but rather than manual, it makes the entire process fully automated. Thus, Continuous Deployment is just one step ahead in automation than Continuous Delivery.

Continuous Integration in The Real World

To emphasize a typical CI Pipeline further, let us explore the diagram below a little deeper.

Version Control: This is the stage where developers' code gets committed and pushed after they have tested their work locally.
Build: Depending on the type of language or technology used, we may need to build the codes into binary executable files (in case of compiled languages) or just package the codes together with all necessary dependencies into a deployable package (in case of interpreted languages).
Unit Test: Unit tests that have been developed by the developers are tested. Depending on how the CI job is configured, the entire pipeline may fail if part of the tests fails, and developers will have to fix this failure before starting the pipeline again. A Job by the way, is a phase in the pipeline. Unit Test is a phase, therefore it can be considered a job on its own.
Deploy: Once the tests are passed, the next phase is to deploy the compiled or packaged code into an artifact repository. This is where all the various versions of code including the latest will be stored. The CI tool will have to pick up the code from this location to proceed with the remaining parts of the pipeline.
Auto Test: Apart from Unit testing, there are many other kinds of tests that are required to analyse the quality of code and determine how vulnerable the software will be to external or internal attacks. These tests must be automated, and there can be multiple environments created to fulfil different test requirements. For example, a server dedicated for Integration Testing will have the code deployed there to conduct integration tests. Once that passes, there can be other sub-layers in the testing phase in which the code will be deployed to, so as to conduct further tests. Such are User Acceptance Testing (UAT), and another can be Penetration Testing. These servers will be named according to what they have been designed to do in those environments. A UAT server is generally be used for UAT, SIT server is for Systems Integration Testing, PEN Server is for Penetration Testing and they can be named whatever the naming style or convention in which the team is used. An environment does not necessarily have to reside on one single server. In most cases it might be a stack as you have defined in your Ansible Inventory. All the servers in the inventory/dev are considered as Dev Environment. The same goes for inventory/stage (Staging Environment) inventory/preprod (Pre-production environment), inventory/prod (Production environment), etc. So, it is all down to naming convention as agreed and used company or team wide.
Deploy to production: Once all the tests have been conducted and either the release manager or whoever has the authority to authorize the release to the production server is happy, he gives green light to hit the deploy button to ship the release to production environment. This is an Ideal Continuous Delivery Pipeline. If the entire pipeline was automated and no human is required to manually give the Go decision, then this would be considered as Continuous Deployment. Because the cycle will be repeated, and every time there is a code commit and push, it causes the pipeline to trigger, and the loop continues over and over again.
Measure and Validate: This is where live users are interacting with the application and feedback is being collected for further improvements and bug fixes. There are many metrics that must be determined and observed here. We will quickly go through 13 metrics that MUST be considered.

Common Best Practices of CI/CD

Before we move on to observability metrics - let us list down the principles that define a reliable and robust CI/CD pipeline:

Maintain a code repository
Automate build process
Make builds self-tested
Everyone commits to the baseline every day
Every commit to baseline should be built
Every bug-fix commit should come with a test case
Keep the build fast
Test in a clone of production environment
Make it easy to get the latest deliverables
Everyone can see the results of the latest build
Automate deployment (if you are confident enough in your CI/CD pipeline and willing to go for a fully automated Continuous Deployment)

Why are we doing everything we are doing? - 13 DevOps Success Metrics

After all, DevOps is all about continuous delivery or deployment, and being able to ship out quality code as fast as possible. This is a very ambitious thing to desire; therefore, we must be careful not to break things as we are moving very fast. By tracking these metrics, we can determine our delivery speed and bottlenecks before breaking things. Ultimately, the goals of DevOps are enhanced Velocity, Quality, and Performance. But how do we track these parameters? Let us have a look at the 13 metrics to watch out for.

1 Deployment frequency: Tracking how often you do deployments is a good DevOps metric. Ultimately, the goal is to do more smaller deployments as often as possible. Reducing the size of deployments makes it easier to test and release. I would suggest counting both production and non-production deployments separately. How often you deploy to QA or pre-production environments is also important. You need to deploy early and often in QA to ensure enough time for testing.

2 Lead time: If the goal is to ship code quickly, this is a key DevOps metric. I would define lead time as the amount of time that occurs between starting on a work item until it is deployed. This helps you know that if you started on a new work item today, how long would it take on average until it gets to production.

3 Customer tickets: The best and worst indicator of application problems is customer support tickets and feedback. The last thing you want is your users reporting bugs or having problems with your software. Because of this, customer tickets also serve as a good indicator of application quality and performance problems.

4 Percentage of passed automated tests: To increase velocity, it is highly recommended that the development team makes extensive usage of unit and functional testing. Since DevOps relies heavily on automation, tracking how well automated tests work is a good DevOps metrics. It is good to know how often code changes break tests.

5 Defect escape rate: Do you know how many software defects are being found in production versus QA? If you want to ship code fast, you need to have confidence that you can find software defects before they get to production. Defect escape rate is a great DevOps metric to track how often those defects make it to production.

6 Availability: The last thing we ever want is for our application to be down. Depending on the type of application and how we deploy it, we may have a little downtime as part of scheduled maintenance. It is highly recommended to track this metric and all unplanned outages. Most software companies build status pages to track this. Such as this Google Products Status Page

7 Service level agreements: Most companies have some service level agreement (SLA) that they promise to the customers. It is also important to track compliance with SLAs. Even if there are no formally stated SLAs, there probably are application non-functional requirements or expectations to be met.

8 Failed deployments: We all hope this never happens, but how often do our deployments cause an outage or major issues for the users? Reversing a failed deployment is something we never want to do, but it is something you should always plan for. If you have issues with failed deployments, be sure to track this metric over time. This could also be seen as tracking **Mean Time To Failure* (MTTF).

9 Error rates: Tracking error rates within the application is super important. Not only they serve as an indicator of quality problems, but also ongoing performance and uptime related issues. In software development, errors are also known as exceptions, and proper exception handling is critical. If they are not handled nicely, we can figure it out while monitoring the rate of errors.

Bugs – Identify new exceptions being thrown in the code after a deployment
Production issues – Capture issues with database connections, query timeouts, and other related issues

Presenting error rate metrics like this simply gives greater insights into where to focus attention.

10 Application usage & traffic: After a deployment, we want to see if the number of transactions or users accessing our system looks normal. If we suddenly have no traffic or a giant spike in traffic, something could be wrong. An attacker may be routing traffic elsewhere, or initiating a DDOS attack

11 Application performance: Before we even perform a deployment, we should configure monitoring tools like Retrace, DataDog, New Relic, or AppDynamics to look for performance problems, hidden errors, and other issues. During and after the deployment, we should also look for any changes in overall application performance and establish some benchmarks to know when things deviate from the norm.

It might be common after a deployment to see major changes in the usage of specific SQL queries, web service or HTTP calls, and other application dependencies. These monitoring tools can provide valuable visualizations like this one below that helps make it easy to spot problems.

12 Mean time to detection (MTTD): When problems happen, it is important that we identify them quickly. The last thing we want is to have a major partial or complete system outage and not know about it. Having robust application monitoring and good observability tools in place will help us detect issues quickly. Once they are detected, we also must fix them quickly!

13 Mean time to recovery (MTTR): This metric helps us track how long it takes to recover from failures. A key metric for the business is keeping failures to a minimum and being able to recover from them quickly. It is typically measured in hours and may refer to business hours, not calendar hours.

These are the major metrics that any DevOps team should track and monitor to understand how well CI/CD process is established and how it helps to deliver quality application to the users.

Simulating a typical CI/CD Pipeline for a PHP Based application

As part of the ongoing infrastructure development with Ansible started from Project 11, you will be tasked to create a pipeline that simulates continuous integration and delivery. Target end to end CI/CD pipeline is represented by the diagram below. It is important to know that both Tooling and TODO Web Applications are based on an interpreted (scripting) language (PHP). It means, it can be deployed directly onto a server and will work without compiling the code to a machine language.

The problem with that approach is, it would be difficult to package and version the software for different releases. And so, in this project, we will be using a different approach for releases, rather than downloading directly from git, we will be using Ansible uri module.

Set Up

This project is partly a continuation of your Ansible work, so simply add and subtract based on the new setup in this project. It will require a lot of servers to simulate all the different environments from dev/ci all the way to production. This will be quite a lot of servers altogether (But you don't have to create them all at once. Only create servers required for an environment you are working with at the moment. For example, when doing deployments for development, do not create servers for integration, pentest, or production yet).

Try to utilize your AWS free tier as much as you can, you can also register a new account if you have exhausted the current one. Alternatively, you can use Google Cloud (GCP) to rent virtual machines from this cloud service provider - you can get $300 credit here or here (NOTE: Please read instructions carefully to get your credits)

NOTE: This is still NOT a cloud-focus project yet. AWS cloud end to end project begins from project-15. Therefore, do not worry about advanced AWS or GCP configuration. All we need here is virtual machines that can be accessed over SSH.

To minimize the cost of cloud servers, you don not have to create all the servers at once, simply spin up a minimal server set up as you progress through the project implementation and have reached a need for more.

To get started, we will focus on these environments initially.

Ci
Dev
Pentest

Both SIT - For System Integration Testing and UAT - User Acceptance Testing do not require a lot of extra installation or configuration. They are basically the webservers holding our applications. But Pentest - For Penetration testing is where we will conduct security related tests, so some other tools and specific configurations will be needed. In some cases, it will also be used for Performance and Load testing. Otherwise, that can also be a separate environment on its own. It all depends on decisions made by the company and the team running the show.

What we want to achieve, is having Nginx to serve as a reverse proxy for our sites and tools. Each environment setup is represented in the below table and diagrams.

CI-Environment

Other Environments from Lower To Higher

DNS requirements

Make DNS entries to create a subdomain for each environment. Assuming your main domain is darey.io

You should have a subdomains list like this:

.tg {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top}
.tg .tg-7btt{border-color:inherit;font-weight:bold;text-align:center;vertical-align:top}

Server	Domain
Jenkins	https://ci.infradev.darey.io
Sonarqube	https://sonar.infradev.darey.io
Artifactory	https://artifacts.infradev.darey.io
Production Tooling	https://tooling.darey.io
Pre-Prod Tooling	https://tooling.preprod.darey.io
Pentest Tooling	https://tooling.pentest.darey.io
UAT Tooling	https://tooling.uat.darey.io
SIT Tooling	https://tooling.sit.darey.io
Dev Tooling	https://tooling.dev.darey.io
Production TODO-WebApp	https://todo.darey.io
Pre-Prod TODO-WebApp	https://todo.preprod.darey.io
Pentest TODO-WebApp	https://todo.pentest.darey.io
UAT TODO-WebApp	https://todo.uat.darey.io
SIT TODO-WebApp	https://todo.sit.darey.io
Dev TODO-WebApp	https://todo.dev.darey.io

Ansible Inventory should look like this

├── ci
├── dev
├── pentest
├── pre-prod
├── prod
├── sit
└── uat

ci inventory file

[jenkins]
<Jenkins-Private-IP-Address>

[nginx]
<Nginx-Private-IP-Address>

[sonarqube]
<SonarQube-Private-IP-Address>

[artifact_repository]
<Artifact_repository-Private-IP-Address>

dev Inventory file

[tooling]
<Tooling-Web-Server-Private-IP-Address>

[todo]
<Todo-Web-Server-Private-IP-Address>

[nginx]
<Nginx-Private-IP-Address>

[db:vars]
ansible_user=ec2-user
ansible_python_interpreter=/usr/bin/python

[db]
<DB-Server-Private-IP-Address>

pentest inventory file

[pentest:children]
pentest-todo
pentest-tooling

[pentest-todo]
<Pentest-for-Todo-Private-IP-Address>

[pentest-tooling]
<Pentest-for-Tooling-Private-IP-Address>

Observations:

You will notice that in the pentest inventory file, we have introduced a new concept pentest:children This is because, we want to have a group called pentest which covers Ansible execution against both pentest-todo and pentest-tooling simultaneously. But at the same time, we want the flexibility to run specific Ansible tasks against an individual group.
The db group has a slightly different configuration. It uses a RedHat/Centos Linux distro. Others are based on Ubuntu (in this case user is ubuntu). Therefore, the user required for connectivity and path to python interpreter are different. If all your environment is based on Ubuntu, you may not need this kind of set up. Totally up to you how you want to do this. Whatever works for you is absolutely fine in this scenario.

This makes us to introduce another Ansible concept called group_vars. With group vars, we can declare and set variables for each group of servers created in the inventory file.

For example, If there are variables we need to be common between both pentest-todo and pentest-tooling, rather than setting these variables in many places, we can simply use the group_vars for pentest. Since in the inventory file it has been created as pentest:children Ansible recognizes this and simply applies that variable to both children.

Ansible Roles for CI Environment

Add two more roles to ansible:

SonarQube (Scroll down to the Sonarqube section to see instructions on how to set up and configure SonarQube manually)
Artifactory

Why do we need SonarQube?

SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality, it is used to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities. Watch a short description here. There is a lot more hands on work ahead with SonarQube and Jenkins. So, the purpose of SonarQube will be clearer to you very soon.

Why do we need Artifactory?

Artifactory is a product by JFrog that serves as a binary repository manager. The binary repository is a natural extension to the source code repository, in that the outcome of your build process is stored. It can be used for certain other automation, but we will it strictly to manage our build artifacts.

Watch a short description here Focus more on the first 10.08 mins

Configuring Ansible For Jenkins Deployment

In previous projects, you have been launching Ansible commands manually from a CLI. Now, with Jenkins, we will start running Ansible from Jenkins UI.

To do this,

Navigate to Jenkins URL
Install & Open Blue Ocean Jenkins Plugin
Create a new pipeline

Select GitHub

Connect Jenkins with GitHub

![Jenkins-Create-Access-Token-To-Github](./Images/Images%2014/Jen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/79bzke2hjiehukqjzlqq.png)

Copy Access Token

Paste the token and connect

Create a new Pipeline

At this point you may not have a Jenkinsfile in the Ansible repository, so Blue Ocean will attempt to give you some guidance to create one. But we do not need that. We will rather create one ourselves. So, click on Administration to exit the Blue Ocean console.

Here is our newly created pipeline. It takes the name of your GitHub repository.

Let us create our `Jenkinsfile`

Inside the Ansible project, create a new directory deploy and start a new file Jenkinsfile inside the directory.

Add the code snippet below to start building the Jenkinsfile gradually. This pipeline currently has just one stage called Build and the only thing we are doing is using the shell script module to echo Building Stage

pipeline {
    agent any


  stages {
    stage('Build') {
      steps {
        script {
          sh 'echo "Building Stage"'
        }
      }
    }
    }
}

Now go back into the Ansible pipeline in Jenkins, and select configure

Scroll down to Build Configuration section and specify the location of the Jenkinsfile at deploy/Jenkinsfile

Back to the pipeline again, this time click "Build now"

This will trigger a build and you will be able to see the effect of our basic Jenkinsfile configuration by going through the console output of the build.

To really appreciate and feel the difference of Cloud Blue UI, it is recommended to try triggering the build again from Blue Ocean interface.

Click on Blue Ocean

Select your project
Click on the play button against the branch

Notice that this pipeline is a multibranch one. This means, if there were more than one branch in GitHub, Jenkins would have scanned the repository to discover them all and we would have been able to trigger a build for each branch.

Let us see this in action.

Create a new git branch and name it feature/jenkinspipeline-stages
Currently we only have the Build stage. Let us add another stage called Test. Paste the code snippet below and push the new changes to GitHub.

   pipeline {
    agent any

  stages {
    stage('Build') {
      steps {
        script {
          sh 'echo "Building Stage"'
        }
      }
    }

    stage('Test') {
      steps {
        script {
          sh 'echo "Testing Stage"'
        }
      }
    }
    }
}

To make your new branch show up in Jenkins, we need to tell Jenkins to scan the repository.
1. Click on the "Administration" button

Navigate to the Ansible project and click on "Scan repository now"

Refresh the page and both branches will start building automatically. You can go into Blue Ocean and see both branches there too.

In Blue Ocean, you can now see how the Jenkinsfile has caused a new step in the pipeline launch build for the new branch.

A QUICK TASK FOR YOU!

1. Create a pull request to merge the latest code into the `main branch`
2. After merging the `PR`, go back into your terminal and switch into the `main` branch.
3. Pull the latest change.
4. Create a new branch, add more stages into the Jenkins file to simulate below phases. (Just add an `echo` command like we have in `build` and `test` stages)
   1. Package 
   2. Deploy 
   3. Clean up
5. Verify in Blue Ocean that all the stages are working, then merge your feature branch to the main branch
6. Eventually, your main branch should have a successful pipeline like this in blue ocean

Running Ansible Playbook from Jenkins

Now that you have a broad overview of a typical Jenkins pipeline. Let us get the actual Ansible deployment to work by:

Installing Ansible on Jenkins
Installing Ansible plugin in Jenkins UI
Creating Jenkinsfile from scratch. (Delete all you currently have in there and start all over to get Ansible to run successfully)

You can watch a 10 minutes video here to guide you through the entire setup

Note: Ensure that Ansible runs against the Dev environment successfully.

Possible errors to watch out for:

Ensure that the git module in Jenkinsfile is checking out SCM to main branch instead of master (GitHub has discontinued the use of Master due to Black Lives Matter. You can read more here)
Jenkins needs to export the ANSIBLE_CONFIG environment variable. You can put the .ansible.cfg file alongside Jenkinsfile in the deploy directory. This way, anyone can easily identify that everything in there relates to deployment. Then, using the Pipeline Syntax tool in Ansible, generate the syntax to create environment variables to set.

https://wiki.jenkins.io/display/JENKINS/Building+a+software+project

Possible issues to watch out for when you implement this

Remember that ansible.cfg must be exported to environment variable so that Ansible knows where to find Roles. But because you will possibly run Jenkins from different git branches, the location of Ansible roles will change. Therefore, you must handle this dynamically. You can use Linux Stream Editor sed to update the section roles_path each time there is an execution. You may not have this issue if you run only from the main branch.
If you push new changes to Git so that Jenkins failure can be fixed. You might observe that your change may sometimes have no effect. Even though your change is the actual fix required. This can be because Jenkins did not download the latest code from GitHub. Ensure that you start the Jenkinsfile with a clean up step to always delete the previous workspace before running a new one. Sometimes you might need to login to the Jenkins Linux server to verify the files in the workspace to confirm that what you are actually expecting is there. Otherwise, you can spend hours trying to figure out why Jenkins is still failing, when you have pushed up possible changes to fix the error.
Another possible reason for Jenkins failure sometimes, is because you have indicated in the Jenkinsfile to check out the main git branch, and you are running a pipeline from another branch. So, always verify by logging onto the Jenkins box to check the workspace, and run git branch command to confirm that the branch you are expecting is there.

If everything goes well for you, it means, the Dev environment has an up-to-date configuration. But what if we need to deploy to other environments?

Are we going to manually update the Jenkinsfile to point inventory to those environments? such as sit, uat, pentest, etc.
Or do we need a dedicated git branch for each environment, and have the inventory part hard coded there.

Think about those for a minute and try to work out which one sounds more like a better solution.

Manually updating the Jenkinsfile is definitely not an option. And that should be obvious to you at this point. Because we try to automate things as much as possible.

Well, unfortunately, we will not be doing any of the highlighted options. What we will be doing is to parameterise the deployment. So that at the point of execution, the appropriate values are applied.

Parameterizing `Jenkinsfile` For Ansible Deployment

To deploy to other environments, we will need to use parameters.

Update sit inventory with new servers

[tooling]
<SIT-Tooling-Web-Server-Private-IP-Address>

[todo]
<SIT-Todo-Web-Server-Private-IP-Address>

[nginx]
<SIT-Nginx-Private-IP-Address>

[db:vars]
ansible_user=ec2-user
ansible_python_interpreter=/usr/bin/python

[db]
<SIT-DB-Server-Private-IP-Address>

Update Jenkinsfile to introduce parameterization. Below is just one parameter. It has a default value in case if no value is specified at execution. It also has a description so that everyone is aware of its purpose.

pipeline {
    agent any

    parameters {
      string(name: 'inventory', defaultValue: 'dev',  description: 'This is the inventory file for the environment to deploy configuration')
    }
...

In the Ansible execution section, remove the hardcoded inventory/dev and replace with `${inventory}

From now on, each time you hit on execute, it will expect an input.

Notice that the default value loads up, but we can now specify which environment we want to deploy the configuration to. Simply type sit and hit Run

Add another parameter. This time, introduce tagging in Ansible. You can limit the Ansible execution to a specific role or playbook desired. Therefore, add an Ansible tag to run against webserver only. Test this locally first to get the experience. Once you understand this, update Jenkinsfile and run it from Jenkins.

CI/CD Pipeline for TODO application

We already have tooling website as a part of deployment through Ansible. Here we will introduce another PHP application to add to the list of software products we are managing in our infrastructure. The good thing with this particular application is that it has unit tests, and it is an ideal application to show an end-to-end CI/CD pipeline for a particular application.

Our goal here is to deploy the application onto servers directly from Artifactory rather than from git. If you have not updated Ansible with an Artifactory role, simply use this guide to create an Ansible role for Artifactory (ignore the Nginx part). Configure Artifactory on Ubuntu 20.04

Phase 1 - Prepare Jenkins

Fork the repository below into your GitHub account

https://github.com/darey-devops/php-todo.git

On you Jenkins server, install PHP, its dependencies and Composer tool (Feel free to do this manually at first, then update your Ansible accordingly later)

sudo apt install -y zip libapache2-mod-php phploc php-{xml,bcmath,bz2,intl,gd,mbstring,mysql,zip}

Install Jenkins plugins
1. Plot plugin
2. Artifactory plugin

We will use plot plugin to display tests reports, and code coverage information.
The Artifactory plugin will be used to easily upload code artifacts into an Artifactory server.

In Jenkins UI configure Artifactory

Configure the server ID, URL and Credentials, run Test Connection.

Phase 2 - Integrate Artifactory repository with Jenkins

Create a dummy Jenkinsfile in the repository
Using Blue Ocean, create a multibranch Jenkins pipeline
On the database server, create database and user

Create database homestead; CREATE USER 'homestead'@'%' IDENTIFIED BY 'sePret^i'; GRANT ALL PRIVILEGES ON * . * TO 'homestead'@'%';

Update the database connectivity requirements in the file .env.sample
Update Jenkinsfile with proper pipeline configuration

`
pipeline {
agent any

stages {

 stage("Initial cleanup") {
      steps {
        dir("${WORKSPACE}") {
          deleteDir()
        }
      }
    }

stage('Checkout SCM') {
  steps {
        git branch: 'main', url: 'https://github.com/darey-devops/php-todo.git'
  }
}

stage('Prepare Dependencies') {
  steps {
         sh 'mv .env.sample .env'
         sh 'composer install'
         sh 'php artisan migrate'
         sh 'php artisan db:seed'
         sh 'php artisan key:generate'
  }
}

}
}
`
Notice the Prepare Dependencies section

The required file by PHP is .env so we are renaming .env.sample to .env
Composer is used by PHP to install all the dependent libraries used by the application
php artisan uses the .env file to setup the required database objects - (After successful run of this step, login to the database, run show tables and you will see the tables being created for you)

Update the Jenkinsfile to include Unit tests step

stage('Execute Unit Tests') steps { sh './vendor/bin/phpunit' }

Phase 3 - Code Quality Analysis

This is one of the areas where developers, architects and many stakeholders are mostly interested in as far as product development is concerned. As a DevOps engineer, you also have a role to play. Especially when it comes to setting up the tools.

For PHP the most commonly tool used for code quality analysis is phploc. Read the article here for more

The data produced by phploc can be ploted onto graphs in Jenkins.

Add the code analysis step in Jenkinsfile. The output of the data will be saved in build/logs/phploc.csv file.
`
stage('Code Analysis') {
steps {
sh 'phploc app/ --log-csv build/logs/phploc.csv'

}
}
`
Plot the data using plot Jenkins plugin.

This plugin provides generic plotting (or graphing) capabilities in Jenkins. It will plot one or more single values variations across builds in one or more plots. Plots for a particular job (or project) are configured in the job configuration screen, where each field has additional help information. Each plot can have one or more lines (called data series). After each build completes the plots' data series latest values are pulled from the CSV file generated by phploc.

`
stage('Plot Code Coverage Report') {
steps {

        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Lines of Code (LOC),Comment Lines of Code (CLOC),Non-Comment Lines of Code (NCLOC),Logical Lines of Code (LLOC)                          ', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'A - Lines of code', yaxis: 'Lines of Code'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Directories,Files,Namespaces', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'B - Structures Containers', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Average Class Length (LLOC),Average Method Length (LLOC),Average Function Length (LLOC)', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'C - Average Length', yaxis: 'Average Lines of Code'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Cyclomatic Complexity / Lines of Code,Cyclomatic Complexity / Number of Methods ', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'D - Relative Cyclomatic Complexity', yaxis: 'Cyclomatic Complexity by Structure'      
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Classes,Abstract Classes,Concrete Classes', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'E - Types of Classes', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Methods,Non-Static Methods,Static Methods,Public Methods,Non-Public Methods', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'F - Types of Methods', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Constants,Global Constants,Class Constants', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'G - Types of Constants', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Test Classes,Test Methods', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'I - Testing', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Logical Lines of Code (LLOC),Classes Length (LLOC),Functions Length (LLOC),LLOC outside functions or classes ', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'AB - Code Structure by Logical Lines of Code', yaxis: 'Logical Lines of Code'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Functions,Named Functions,Anonymous Functions', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'H - Types of Functions', yaxis: 'Count'
        plot csvFileName: 'plot-396c4a6b-b573-41e5-85d8-73613b2ffffb.csv', csvSeries: [[displayTableFlag: false, exclusionValues: 'Interfaces,Traits,Classes,Methods,Functions,Constants', file: 'build/logs/phploc.csv', inclusionFlag: 'INCLUDE_BY_STRING', url: '']], group: 'phploc', numBuilds: '100', style: 'line', title: 'BB - Structure Objects', yaxis: 'Count'


  }
}

You should now see a Plot menu item on the left menu. Click on it to see the charts. (The analytics may not mean much to you as it is meant to be read by developers. So, you need not worry much about it - this is just to give you an idea of the real-world implementation).

![Jenkins-PHPloc-Plot]](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m3x2oc1ndotz0xhjzglw.png)

Bundle the application code for into an artifact (archived package) upload to Artifactory

stage ('Package Artifact') { steps { sh 'zip -qr php-todo.zip ${WORKSPACE}/*' } }

Publish the resulted artifact into Artifactory

`
stage ('Upload Artifact to Artifactory') {
steps {
script {
def server = Artifactory.server 'artifactory-server'

def uploadSpec = """{
"files": [
{
"pattern": "php-todo.zip",
"target": "/php-todo",
"props": "type=zip;status=ready"

                   }
                ]
             }""" 

             server.upload spec: uploadSpec
           }
        }

    }

Deploy the application to the dev environment by launching Ansible pipeline

stage ('Deploy to Dev Environment') { steps { build job: 'ansible-project/main', parameters: [[$class: 'StringParameterValue', name: 'env', value: 'dev']], propagate: false, wait: true } }

The build job used in this step tells Jenkins to start another job. In this case it is the ansible-project job, and we are targeting the main branch. Hence, we have ansible-project/main. Since the Ansible project requires parameters to be passed in, we have included this by specifying the parameters section. The name of the parameter is env and its value is dev. Meaning, deploy to the Development environment.

But how are we certain that the code being deployed has the quality that meets corporate and customer requirements? Even though we have implemented Unit Tests and Code Coverage Analysis with phpunit and phploc, we still need to implement Quality Gate to ensure that ONLY code with the required code coverage, and other quality standards make it through to the environments.

To achieve this, we need to configure SonarQube - An open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities.

SonarQube Installation

Before we start getting hands on with SonarQube configuration, it is incredibly important to understand a few concepts:

Software Quality - The degree to which a software component, system or process meets specified requirements based on user needs and expectations.
Software Quality Gates - Quality gates are basically acceptance criteria which are usually presented as a set of predefined quality criteria that a software development project must meet in order to proceed from one stage of its lifecycle to the next one.

SonarQube is a tool that can be used to create quality gates for software projects, and the ultimate goal is to be able to ship only quality software code.

Despite that DevOps CI/CD pipeline helps with fast software delivery, it is of the same importance to ensure the quality of such delivery. Hence, we will need SonarQube to set up Quality gates. In this project we will use predefined Quality Gates (also known as The Sonar Way). Software testers and developers would normally work with project leads and architects to create custom quality gates.

Install SonarQube on Ubuntu 20.04 With PostgreSQL as Backend Database

Here is a manual approach to installation. Ensure that you can to automate the same with Ansible.

Below is a step by step guide how to install SonarQube 7.9.3 version. It has a strong prerequisite to have Java installed since the tool is Java-based. MySQL support for SonarQube is deprecated, therefore we will be using PostgreSQL.

We will make some Linux Kernel configuration changes to ensure optimal performance of the tool - we will increase vm.max_map_count, file discriptor and ulimit.

Tune Linux Kernel

This can be achieved by making session changes which does not persist beyond the current session terminal.

sudo sysctl -w vm.max_map_count=262144 sudo sysctl -w fs.file-max=65536 ulimit -n 65536 ulimit -u 4096

To make a permanent change, edit the file /etc/security/limits.conf and append the below

sonarqube - nofile 65536 sonarqube - nproc 4096

Before installing, let us update and upgrade system packages:

sudo apt-get update sudo apt-get upgrade

Install wget and unzip packages

sudo apt-get install wget unzip -y

Install OpenJDK and Java Runtime Environment (JRE) 11

sudo apt-get install openjdk-11-jdk -y sudo apt-get install openjdk-11-jre -y

Set default JDK - To set default JDK or switch to OpenJDK enter below command:

sudo update-alternatives --config java

If you have multiple versions of Java installed, you should see a list like below:

`
Selection Path Priority Status

0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode

1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode

2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode

3 /usr/lib/jvm/java-8-oracle/jre/bin/java 1081 manual mode

Type "1" to switch OpenJDK 11

Verify the set JAVA Version:

java -version

Output
`
java -version

openjdk version "11.0.7" 2020-04-14

OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-3ubuntu1)

OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-3ubuntu1, mixed mode, sharing)
`

Install and Setup PostgreSQL 10 Database for SonarQube

The command below will add PostgreSQL repo to the repo list:

sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/lsb_release -cs-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'

Download PostgreSQL software

wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -

Install PostgreSQL Database Server

sudo apt-get -y install postgresql postgresql-contrib

Start PostgreSQL Database Server

sudo systemctl start postgresql

Enable it to start automatically at boot time

sudo systemctl enable postgresql

Change the password for default postgres user (Pass in the password you intend to use, and remember to save it somewhere)

sudo passwd postgres

Switch to the postgres user

su - postgres

Create a new user by typing

createuser sonar

Switch to the PostgreSQL shell

psql

Set a password for the newly created user for SonarQube database

ALTER USER sonar WITH ENCRYPTED password 'sonar';

Create a new database for PostgreSQL database by running:

CREATE DATABASE sonarqube OWNER sonar;

Grant all privileges to sonar user on sonarqube Database.

grant all privileges on DATABASE sonarqube to sonar;

Exit from the psql shell:

\q

Switch back to the sudo user by running the exit command.

exit

Install SonarQube on Ubuntu 20.04 LTS

Navigate to the tmp directory to temporarily download the installation files

cd /tmp && sudo wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.3.zip

Unzip the archive setup to /opt directory

sudo unzip sonarqube-7.9.3.zip -d /opt

Move extracted setup to /opt/sonarqube directory

sudo mv /opt/sonarqube-7.9.3 /opt/sonarqube

Configure SonarQube

We cannot run SonarQube as a root user, if you run using root user it will stop automatically. The ideal approach will be to create a separate group and a user to run SonarQube

Create a group sonar

sudo groupadd sonar

Now add a user with control over the /opt/sonarqube directory

sudo useradd -c "user to run SonarQube" -d /opt/sonarqube -g sonar sonar sudo chown sonar:sonar /opt/sonarqube -R

Open SonarQube configuration file using your favourite text editor (e.g., nano or vim)

sudo vim /opt/sonarqube/conf/sonar.properties

Find the following lines:

sonar.jdbc.username=

sonar.jdbc.password=

Uncomment them and provide the values of PostgreSQL Database username and password:

--------------------------------------------------------------------------------------------------

DATABASE

IMPORTANT:

- The embedded H2 database is used by default. It is recommended for tests but not for

production use. Supported databases are Oracle, PostgreSQL and Microsoft SQLServer.

- Changes to database connection URL (sonar.jdbc.url) can affect SonarSource licensed products.

User credentials.

Permissions to create tables, indices and triggers must be granted to JDBC user.

The schema must be created first.

sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.jdbc.url=jdbc:postgresql://localhost:5432/sonarqube
`

Edit the sonar script file and set RUN_AS_USER

sudo nano /opt/sonarqube/bin/linux-x86-64/sonar.sh

If specified, the Wrapper will be run as the specified user.

IMPORTANT - Make sure that the user has the required privileges to write

the PID file and wrapper.log files. Failure to be able to write the log

file will cause the Wrapper to exit without any way to write out an error

message.

NOTE - This will set the user which is used to run the Wrapper as well as

the JVM and is not useful in situations where a privileged resource or

port needs to be allocated prior to the user being changed.

RUN_AS_USER=sonar
`

Now, to start SonarQube we need to do following:

Switch to sonar user

sudo su sonar

Move to the script directory

cd /opt/sonarqube/bin/linux-x86-64/

Run the script to start SonarQube

./sonar.sh start

Expected output shall be as:

`
Starting SonarQube...

Started SonarQube
`

Check SonarQube running status:

./sonar.sh status

Sample Output below:

`
$./sonar.sh status

SonarQube is running (176483).
`

To check SonarQube logs, navigate to /opt/sonarqube/logs/sonar.log directory

tail /opt/sonarqube/logs/sonar.log

Output

`
INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube]: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp --add-opens=java.base/java.util=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.|[::1] -cp ./lib/common/:/opt/sonarqube/lib/jdbc/h2/h2-1.3.176.jar org.sonar.ce.app.CeServer /opt/sonarqube/temp/sq-process15059956114837198848properties

INFO app[][o.s.a.SchedulerImpl] Process[ce] is up

INFO app[][o.s.a.SchedulerImpl] SonarQube is up
`

You can see that SonarQube is up and running

Configure SonarQube to run as a `systemd` service

Stop the currently running SonarQube service

cd /opt/sonarqube/bin/linux-x86-64/

Run the script to start SonarQube
./sonar.sh stop

Create a systemd service file for SonarQube to run as System Startup.

sudo nano /etc/systemd/system/sonar.service

Add the configuration below for systemd to determine how to start, stop, check status, or restart the SonarQube service.

`
[Unit]
Description=SonarQube service
After=syslog.target network.target

[Service]
Type=forking

ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop

User=sonar
Group=sonar
Restart=always

LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target
`

Save the file and control the service with systemctl

sudo systemctl start sonar sudo systemctl enable sonar sudo systemctl status sonar

Access SonarQube

To access SonarQube using browser, type server's IP address followed by port 9000

http://server_IP:9000 OR http://localhost:9000

Now, when SonarQube is up and running, it is time to setup our Quality gate in Jenkins.

Configure SonarQube and Jenkins For Quality Gate

In Jenkins, install SonarScanner plugin
Navigate to configure system in Jenkins. Add SonarQube server as shown below: Manage Jenkins > Configure System

Generate authentication token in SonarQube User > My Account > Security > Generate Tokens

Configure Quality Gate Jenkins Webhook in SonarQube - The URL should point to your Jenkins server http://{JENKINS_HOST}/sonarqube-webhook/
Administration > Configuration > Webhooks > Create
Setup SonarQube scanner from Jenkins - Global Tool Configuration
Manage Jenkins > Global Tool Configuration

Update Jenkins Pipeline to include SonarQube scanning and Quality Gate

Below is the snippet for a Quality Gate stage in Jenkinsfile.

`
stage('SonarQube Quality Gate') {
environment {
scannerHome = tool 'SonarQubeScanner'
}
steps {
withSonarQubeEnv('sonarqube') {
sh "${scannerHome}/bin/sonar-scanner"
}

}
}

NOTE: The above step will fail because we have not updated `sonar-scanner.properties

Configure sonar-scanner.properties - From the step above, Jenkins will install the scanner tool on the Linux server. You will need to go into the tools directory on the server to configure the properties file in which SonarQube will require to function during pipeline execution.

cd /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQubeScanner/conf/

Open sonar-scanner.properties file

sudo vi sonar-scanner.properties

Add configuration related to php-todo project

sonar.host.url=http://<SonarQube-Server-IP-address>:9000
sonar.projectKey=php-todo
#----- Default source code encoding
sonar.sourceEncoding=UTF-8
sonar.php.exclusions=**/vendor/**
sonar.php.coverage.reportPaths=build/logs/clover.xml
sonar.php.tests.reportPath=build/logs/junit.xml

HINT: To know what exactly to put inside the sonar-scanner.properties file, SonarQube has a configurations page where you can get some directions.

A brief explanation of what is going on the the stage - set the environment variable for the scannerHome use the same name used when you configured SonarQube Scanner from Jenkins Global Tool Configuration. If you remember, the name was SonarQubeScanner. Then, within the steps use shell to run the scanner from bin directory.

To further examine the configuration of the scanner tool on the Jenkins server - navigate into the tools directory

cd /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQubeScanner/bin

List the content to see the scanner tool sonar-scanner. That is what we are calling in the pipeline script.

Output of ls -latr

ubuntu@ip-172-31-16-176:/var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQubeScanner/bin$ ls -latr
total 24
-rwxr-xr-x 1 jenkins jenkins 2550 Oct  2 12:42 sonar-scanner.bat
-rwxr-xr-x 1 jenkins jenkins  586 Oct  2 12:42 sonar-scanner-debug.bat
-rwxr-xr-x 1 jenkins jenkins  662 Oct  2 12:42 sonar-scanner-debug
-rwxr-xr-x 1 jenkins jenkins 1823 Oct  2 12:42 sonar-scanner
drwxr-xr-x 2 jenkins jenkins 4096 Dec 26 18:42 .

So far you have been given code snippets on each of the stages within the Jenkinsfile. But, you should also be able to generate Jenkins configuration code yourself.

To generate Jenkins code, navigate to the dashboard for the php-todo pipeline and click on the Pipeline Syntax menu item

Dashboard > php-todo > Pipeline Syntax

Click on Steps and select withSonarQubeEnv - This appears in the list because of the previous SonarQube configurations you have done in Jenkins. Otherwise, it would not be there.

Within the generated block, you will use the sh command to run shell on the server. For more advanced usage in other projects, you can add to bookmarks this SonarQube documentation page in your browser.

End-to-End Pipeline Overview

Indeed, this has been one of the longest projects from Project 1, and if everything has worked out for you so far, you should have a view like below:

But we are not completely done yet!

The quality gate we just included has no effect. Why? Well, because if you go to the SonarQube UI, you will realise that we just pushed a poor-quality code onto the development environment.

Navigate to php-todo project in SonarQube

There are bugs, and there is 0.0% code coverage. (code coverage is a percentage of unit tests added by developers to test functions and objects in the code)

If you click on php-todo project for further analysis, you will see that there is 6 hours' worth of technical debt, code smells and security issues in the code.

In the development environment, this is acceptable as developers will need to keep iterating over their code towards perfection. But as a DevOps engineer working on the pipeline, we must ensure that the quality gate step causes the pipeline to fail if the conditions for quality are not met.

Conditionally deploy to higher environments

In the real world, developers will work on feature branch in a repository (e.g., GitHub or GitLab). There are other branches that will be used differently to control how software releases are done. You will see such branches as:

Develop
Master or Main (The * is a place holder for a version number, Jira Ticket name or some description. It can be something like Release-1.0.0)
Feature/*
Release/*
Hotfix/*

etc.

There is a very wide discussion around release strategy, and git branching strategies which in recent years are considered under what is known as GitFlow (Have a read and keep as a bookmark - it is a possible candidate for an interview discussion, so take it seriously!)

Assuming a basic gitflow implementation restricts only the develop branch to deploy code to Integration environment like sit.

Let us update our Jenkinsfile to implement this:

First, we will include a When condition to run Quality Gate whenever the running branch is either develop, hotfix, release, main, or master

when { branch pattern: "^develop*|^hotfix*|^release*|^main*", comparator: "REGEXP"}

Then we add a timeout step to wait for SonarQube to complete analysis and successfully finish the pipeline only when code quality is acceptable.

    timeout(time: 1, unit: 'MINUTES') {
        waitForQualityGate abortPipeline: true
    }

The complete stage will now look like this:

    stage('SonarQube Quality Gate') {
      when { branch pattern: "^develop*|^hotfix*|^release*|^main*", comparator: "REGEXP"}
        environment {
            scannerHome = tool 'SonarQubeScanner'
        }
        steps {
            withSonarQubeEnv('sonarqube') {
                sh "${scannerHome}/bin/sonar-scanner -Dproject.settings=sonar-project.properties"
            }
            timeout(time: 1, unit: 'MINUTES') {
                waitForQualityGate abortPipeline: true
            }
        }
    }

To test, create different branches and push to GitHub. You will realise that only branches other than develop, hotfix, release, main, or master will be able to deploy the code.

If everything goes well, you should be able to see something like this:

Notice that with the current state of the code, it cannot be deployed to Integration environments due to its quality. In the real world, DevOps engineers will push this back to developers to work on the code further, based on SonarQube quality report. Once everything is good with code quality, the pipeline will pass and proceed with sipping the codes further to a higher environment.

Complete the following tasks to finish Project 14

Introduce Jenkins agents/slaves - Add 2 more servers to be used as Jenkins slave. Configure Jenkins to run its pipeline jobs randomly on any available slave nodes.
Configure webhook between Jenkins and GitHub to automatically run the pipeline when there is a code push.
Deploy the application to all the environments
Optional - Experience pentesting in pentest environment by configuring Wireshark there and just explore for information sake only. Watch Wireshark Tutorial here
- Ansible Role for Wireshark:
- https://github.com/ymajik/ansible-role-wireshark (Ubuntu)
- https://github.com/wtanaka/ansible-role-wireshark (RedHat)

Congratulations!

You have just experienced one of the most interesting and complex projects in your Project Based Learning journey so far. The vast experience and knowledge you have acquired here will set the stage for the next 6 projects to come. You should be ready to start applying for DevOps jobs after completing Project 20.

Instructions On How To Submit Your Work For Review And Feedback

To submit your work for review and feedback - follow this instruction.
In addition to your GitHub projects (both, Ansible and PHP application) also prepare and submit the following:

Make a short video on how your pipelines have executed
In the video, showcase your SonarQube UI

Tooling Website deployment automation with Continuous Integration. Introduction to Jenkins

Christian Ameachi — Sat, 24 Feb 2024 22:16:51 +0000

Tooling Website deployment automation with Continuous Integration. Introduction to Jenkins

In previous Project 8 we introduced horizontal scalability concept, which allow us to add new Web Servers to our Tooling Website and you have successfully deployed a set up with 2 Web Servers and also a Load Balancer to distribute traffic between them. If it is just two or three servers - it is not a big deal to configure them manually. Imagine that you would need to repeat the same task over and over again adding dozens or even hundreds of servers.

DevOps is about Agility, and speedy release of software and web solutions. One of the ways to guarantee fast and repeatable deployments is Automation of routine tasks.

In this project we are going to start automating part of our routine tasks with a free and open source automation server - Jenkins. It is one of the mostl popular CI/CD tools, it was created by a former Sun Microsystems developer Kohsuke Kawaguchi and the project originally had a named "Hudson".

Acording to Circle CI, Continuous integration (CI) is a software development strategy that increases the speed of development while ensuring the quality of the code that teams deploy. Developers continually commit code in small increments (at least daily, or even several times a day), which is then automatically built and tested before it is merged with the shared repository.

In our project we are going to utilize Jenkins CI capabilities to make sure that every change made to the source code in GitHub https://github.com/<yourname>/tooling will be automatically be updated to the Tooling Website.

Side Self Study

Read about Continuous Integration, Continuous Delivery and Continuous Deployment.

Task

Enhance the architecture prepared in Project 8 by adding a Jenkins server, configure a job to automatically deploy source codes changes from Git to NFS server.

Here is how your update architecture will look like upon competition of this project:

Tooling Website deployment automation with Continuous Integration. Introduction to Jenkins

DevOps is about Agility, and speedy release of software and web solutions. One of the ways to guarantee fast and repeatable deployments is Automation of routine tasks.

Side Self Study

Read about Continuous Integration, Continuous Delivery and Continuous Deployment.

Task

Enhance the architecture prepared in Project 8 by adding a Jenkins server, configure a job to automatically deploy source codes changes from Git to NFS server.

Here is how your update architecture will look like upon competition of this project:

Instructions On How To Submit Your Work For Review And Feedback

To submit your work for review and feedback - follow this instruction.

Step 1 - Install Jenkins server

Create an AWS EC2 server based on Ubuntu Server 20.04 LTS and name it "Jenkins"
Install JDK (since Jenkins is a Java-based application)

sudo apt update
sudo apt install openjdk-11-jre
#Confirm Java installation
java -version

Install Jenkins

Refer to official Jenkins documentation for more details.

curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install -y jenkins

Make sure Jenkins is up and running

sudo systemctl status jenkins

By default Jenkins server uses TCP port 8080 - open it by creating a new Inbound Rule in your EC2 Security Group

Perform initial Jenkins setup.

From your browser access http://<Jenkins-Server-Public-IP-Address-or-Public-DNS-Name>:8080

You will be prompted to provide a default admin password

Retrieve it from your server:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Then you will be asked which plugings to install - choose suggested plugins.

Once plugins installation is done - create an admin user and you will get your Jenkins server address.

The installation is completed!

Step 2 - Configure Jenkins to retrieve source codes from GitHub using Webhooks

In this part, you will learn how to configure a simple Jenkins job/project (these two terms can be used interchangeably). This job will will be triggered by GitHub webhooks and will execute a 'build' task to retrieve codes from GitHub and store it locally on Jenkins server.

Enable webhooks in your GitHub repository settings

Go to Jenkins web console, click "New Item" and create a "Freestyle project"

To connect your GitHub repository, you will need to provide its URL, you can copy from the repository itself

In configuration of your Jenkins freestyle project choose Git repository, provide there the link to your Tooling GitHub repository and credentials (user/password) so Jenkins could access files in the repository.

Save the configuration and let us try to run the build. For now we can only do it manually.
Click "Build Now" button, if you have configured everything correctly, the build will be successfull and you will see it under #1

You can open the build and check in "Console Output" if it has run successfully.

If so - congratulations! You have just made your very first Jenkins build!

But this build does not produce anything and it runs only when we trigger it manually. Let us fix it.

Click "Configure" your job/project and add these two configurations

Configure triggering the job from GitHub webhook:

Configure "Post-build Actions" to archive all the files - files resulted from a build are called "artifacts".

Now, go ahead and make some change in any file in your GitHub repository (e.g. README.MD file) and push the changes to the master branch.

You will see that a new build has been launched automatically (by webhook) and you can see its results - artifacts, saved on Jenkins server.

You have now configured an automated Jenkins job that receives files from GitHub by webhook trigger (this method is considered as 'push' because the changes are being 'pushed' and files transfer is initiated by GitHub). There are also other methods: trigger one job (downstreadm) from another (upstream), poll GitHub periodically and others.

By default, the artifacts are stored on Jenkins server locally

ls /var/lib/jenkins/jobs/tooling_github/builds/<build_number>/archive/

Step 3 - Configure Jenkins to copy files to NFS server via SSH

Now we have our artifacts saved locally on Jenkins server, the next step is to copy them to our NFS server to /mnt/apps directory.

Jenkins is a highly extendable application and there are 1400+ plugins available. We will need a plugin that is called "Publish Over SSH".

Install "Publish Over SSH" plugin.

On main dashboard select "Manage Jenkins" and choose "Manage Plugins" menu item.

On "Available" tab search for "Publish Over SSH" plugin and install it

Configure the job/project to copy artifacts over to NFS server.

On main dashboard select "Manage Jenkins" and choose "Configure System" menu item.

Scroll down to Publish over SSH plugin configuration section and configure it to be able to connect to your NFS server:

Provide a private key (content of .pem file that you use to connect to NFS server via SSH/Putty)
Arbitrary name
Hostname - can be private IP address of your NFS server
Username - ec2-user (since NFS server is based on EC2 with RHEL 8)
Remote directory - /mnt/apps since our Web Servers use it as a mointing point to retrieve files from the NFS server

Test the configuration and make sure the connection returns Success. Remember, that TCP port 22 on NFS server must be open to receive SSH connections.

Save the configuration, open your Jenkins job/project configuration page and add another one "Post-build Action"

Configure it to send all files produced by the build into our previously define remote directory. In our case we want to copy all files and directories - so we use **.
If you want to apply some particular pattern to define which files to send - use this syntax.

Save this configuration and go ahead, change something in README.MD file in your GitHub Tooling repository.

Webhook will trigger a new job and in the "Console Output" of the job you will find something like this:

SSH: Transferred 25 file(s)
Finished: SUCCESS

To make sure that the files in /mnt/apps have been udated - connect via SSH/Putty to your NFS server and check README.MD file

cat /mnt/apps/README.md

If you see the changes you had previously made in your GitHub - the job works as expected.

Congratulations!

You have just implemented your first Continous Integration solution using Jenkins CI. Watch out for advanced CI configurations in upcoming projects.

Forem: Christian Ameachi

[Boost]

Building a Secure and Observable E-commerce Microservices Platform on AWS EKS

Christian Ameachi ・ Feb 22

Building a Secure and Observable E-commerce Microservices Platform on AWS EKS

Introduction

The Stack

Infrastructure as Code (IaC)

The CI/CD Engine

Zero-Downtime Reliability

Observability: The Full Stack

Conclusion

[Boost]

From Linux Primitives to Docker Swarm: A Deep Dive into Container Networking 🚀

Christian Ameachi ・ Jan 9

From Linux Primitives to Docker Swarm: A Deep Dive into Container Networking 🚀

🏗️ Phase 1: The Hard Way (Linux Primitives)

🐳 Phase 2: Optimization with Docker

🌐 Phase 3: The Distributed Horizon (Multi-Host Networking)

📊 Performance Benchmark: Linux vs. Docker

Very simple and useful cli

Building a Simple Ticket Tracker CLI in Go

Christian Ameachi ・ Nov 22

Building a Simple Ticket Tracker CLI in Go

A lightweight, command-line alternative to complex ticketing systems.

Introduction

The Tech Stack

How It Works

1. The Command Structure

2. Data Persistence (The "Database")

Installation & Usage

Adding a Ticket

Listing Tickets

Filter by date

CSV Storage

Columns :

Future Improvements

Conclusion

Experience Continuous Integration with Jenkins | Ansible | Artifactory | SonarQube | PHP

Experience Continuous Integration with Jenkins | Ansible | Artifactory | SonarQube | PHP

What is Continuous Integration?

Continuous Integration in The Real World

Common Best Practices of CI/CD

Why are we doing everything we are doing? - 13 DevOps Success Metrics

Simulating a typical CI/CD Pipeline for a PHP Based application

Set Up

CI-Environment

Other Environments from Lower To Higher

DNS requirements

Ansible Inventory should look like this

Ansible Roles for CI Environment

Why do we need SonarQube?

Why do we need Artifactory?

Configuring Ansible For Jenkins Deployment

Let us create our Jenkinsfile

A QUICK TASK FOR YOU!

Running Ansible Playbook from Jenkins

Parameterizing Jenkinsfile For Ansible Deployment

CI/CD Pipeline for TODO application

Phase 1 - Prepare Jenkins

Phase 2 - Integrate Artifactory repository with Jenkins

Phase 3 - Code Quality Analysis

SonarQube Installation

Install SonarQube on Ubuntu 20.04 With PostgreSQL as Backend Database

Tune Linux Kernel

Install and Setup PostgreSQL 10 Database for SonarQube

Install SonarQube on Ubuntu 20.04 LTS

Configure SonarQube

sonar.jdbc.username=

sonar.jdbc.password=

--------------------------------------------------------------------------------------------------

DATABASE

IMPORTANT:

- The embedded H2 database is used by default. It is recommended for tests but not for

production use. Supported databases are Oracle, PostgreSQL and Microsoft SQLServer.

- Changes to database connection URL (sonar.jdbc.url) can affect SonarSource licensed products.

User credentials.

Permissions to create tables, indices and triggers must be granted to JDBC user.

The schema must be created first.

If specified, the Wrapper will be run as the specified user.

Let us create our `Jenkinsfile`

Parameterizing `Jenkinsfile` For Ansible Deployment

Configure SonarQube to run as a `systemd` service