Forem: Tung Leo

Wrapping Up 2024: IIS and Apache Tomcat Join the DevOps Basics Journey! 🎉

Tung Leo — Wed, 25 Dec 2024 06:16:59 +0000

Introduction

As we close out an exciting and productive 2024, I’m thrilled to share the latest updates to the DevOps Basics repository. This year has been a remarkable journey, thanks to the incredible support from all of you—our growing community of learners and practitioners. Your feedback, contributions, and encouragement have been instrumental in pushing this project forward. 🙏

To wrap up 2024, I’m excited to introduce two new topics: IIS Web Server and Apache Tomcat Web Server. These additions mark another step in our mission to provide a comprehensive foundation for anyone exploring the DevOps landscape.

2024 Highlights ✨

The repository now has 1.7k stars (and counting!), and it's all because of your amazing support.
We have 34 topics for both Dev and DevOps. Each topic includes documentation, a "getting started" guide, and hands-on examples.
There are 148 forks with 34 contributors.

Release v0.35.0

Here's a quick overview of what’s new in the latest release:

New Topics 🚀

IIS Web Server: Dive into the world of Microsoft’s Internet Information Services, learn how to set it up, and explore hosting capabilities for websites and web applications.
Apache Tomcat: A lightweight yet powerful Java-based web server, ideal for hosting Java web applications and implementing Java Servlet and JSP technologies.

More Improvements 📈

VirtualBox Basics: A starting point for mastering virtualization.
Dynatrace Basics: Understand monitoring and observability tools for modern applications.
Bug Fixes and Enhancements: Improved documentation and user experience across various topics.

Check out the full changelog for all updates.

Looking Ahead to 2025 🛠️

As we turn the page to 2025, I’m excited to keep building and improving the DevOps Basics repository. Plans include:

Adding more hands-on examples to existing topics.
Expanding into new tools and technologies that align with the evolving DevOps landscape.
Refining content to ensure it's beginner-friendly yet valuable for experienced engineers.

Thank You 💖

To everyone who has starred, forked, contributed, or simply explored this repository—thank you! Your support means the world to me, and it inspires me to continue growing this resource.

If you haven’t already, please consider starring the repository ⭐️ on GitHub to help us reach 2,000 stars and beyond! Every star is a testament to the value of shared knowledge and a community-driven approach.

Here’s to another year of learning, growth, and success. See you in 2025! 🎉

Star devops-basics ⭐️ on GitHub

If you have suggestions for future topics or improvements, feel free to open an issue or reach out. Together, let’s make DevOps Basics an even better resource for everyone. 🚀

Introducing TheDevOpsHub: Your Central Hub for DevOps Learning! 🚀

Tung Leo — Sat, 19 Oct 2024 15:21:43 +0000

Welcome to TheDevOpsHub, a new initiative designed to provide a comprehensive learning experience for anyone passionate about DevOps. Whether you're a beginner or an advanced practitioner, TheDevOpsHub is your one-stop resource for documentation, hands-on demos, and projects that span the entire DevOps landscape—from cloud computing to CI/CD pipelines, containerization, and infrastrucure as code (IaC).

What is TheDevOpsHub?

TheDevOpsHub is a free online platform that curates a wide array of DevOps resources, organized into easy-to-navigate categories. It offers:

In-depth documentation: covering popular DevOps tools, workflows, and best practices.
Hands-on demos and projects: to help you apply what you’ve learned in real-world scenarios.
Community engagement: currently maintained by me (Tung Leo) and supported by a growing community of DevOps enthusiasts.
GitHub organization: https://github.com/thedevopshub
Official website: https://thedevopshub.org

Featured Projects

1. The Trio DevOps Projects

We’ve divided our core projects into three categories for structured learning:

devops-basics (1.6k stars ⭐): Practical starting points for understanding key DevOps concepts and tools.
devops-practice: Hands-on exercises to reinforce your skills through real-world scenarios.
devops-project: Advanced projects designed to elevate your DevOps skills to the next level.

2. Cloud Learning

Our cloud-focused repositories include resources for Azure and AWS:

Azure:
- AzureHub: Learning resources and documentation for Azure services.
- AZ-104: A complete guide for preparing for the AZ-104 certification.
AWS:
- AWSHub: A collection of AWS service resources.

3. Linux and OS

LinuxHub: Resources for Linux best practices, tips, and tricks.

4. CI/CD Tools

JenkinsHub: All things Jenkins—from setup to automation pipelines.

5. Infrastructure as Code (IaC)

Terraform:
- TerraformHub: Extensive practical resources for mastering Terraform.
- terraform-template: Best-practice templates for multi-environment, modular deployments.
Ansible:
- AnsibleHub: Ansible learning resources and templates.
- ansible-template: Ansible template repository with CI, linting, containerization, and more...

6. Containerization & Kubernetes

microservices-deployment: A complete microservices deployment setup on Kubernetes, including monitoring and logging features.
k8sHub: Comprehensive Kubernetes examples and practices.

7. Monitoring

MonitoringHub: Prometheus, Grafana, and more for end-to-end system monitoring.

Explore More Resources...

Looking for daily-used command-line tools or DevOps books? Check out our cheatsheets and book collections:

cmd: Bookmark essential command-line tools.
Books: Our curated list of DevOps books to deepen your knowledge.

Stay Connected and Up-to-Date

This is just the beginning! I will be actively maintaining TheDevOpsHub, adding fresh content, and releasing new projects and demos to keep it growing. Expect more learning resources and hands-on examples to be added regularly. So, stay tuned and follow our GitHub organization to stay updated! ⭐

If you'd like to support my work and help sustain TheDevOpsHub for everyone’s benefit, you can contribute at TheDevOpsHub Sponsors ❤️. Your support allows me to keep creating high-quality content and resources for the DevOps community!

Feel free to reach out via email at info@thedevopshub.org.

Start your DevOps journey today by exploring TheDevOpsHub and level up your skills with hands-on projects and comprehensive resources!

DevOps Basics reaches 200 stars: A milestone celebration 🎉

Tung Leo — Sun, 15 Sep 2024 09:38:13 +0000

I am thrilled to announce that my GitHub repository, devops-basics, has just reached an exciting milestone of 200 stars! 🎉 This achievement is a testament to the growing interest in DevOps practices and the value the community finds in this resource.

Impressive Metrics 📈

I'm proud to share some impressive numbers that showcase the community's engagement with this project:

200+ Stars: A clear indication of the value and popularity of the repository.
56 Forks: Demonstrating how many users are actively engaging with and building upon the content.
1100+ Commits: Reflecting mythe community dedication to continually improving and expanding the resource.
30+ Topics: Covering a comprehensive range of DevOps subjects, from containerization. CI/CD tools to cloud services and beyond.
34 Contributors:Thank you all for your contributions! 💖

These metrics highlight not only the project's growth but also the active development, extensive coverage, and community interest in DevOps practices.

What is devops-basics?

The devops-basics repository is a comprehensive resource I created to share my information and experience in DevOps with everyone. It's designed to help both newcomers and experienced professionals enhance their DevOps skills. The repository covers a wide range of topics including Docker, Kubernetes, Terraform, Ansible, cloud services, system architecture, monitoring, and much more. With over 30+ important DevOps subjects, each topic comes with an overview, official documentation links, cheatsheets, and additional resources to facilitate learning.

Key Features 🚀

Wide range of topics: Explore 30+ important DevOps subjects
Useful resources: Each topic includes overviews, documentation links, cheatsheets, and extra learning materials
Hands-On practice: Basic examples for each topic allow you to apply what you've learned
Advanced examples: Further elevate your skills with more complex scenarios

Thank You to the Supporters 💖

I want to extend my heartfelt gratitude to everyone who has shown interest in and supported this project. Your stars, forks, and contributions have been instrumental in reaching this milestone. This achievement wouldn't have been possible without the DevOps community's engagement and enthusiasm.

Looking Ahead 👓

Reaching 200 stars is just the beginning! I am committed to maintaining and frequently updating the content in the devops-basics repository. My goal is to continue providing valuable resources and practical examples to help DevOps enthusiasts at all levels improve their skills and knowledge.

Get Involved 💎

Your feedback and contributions are always welcome! If you have suggestions for new topics, improvements to existing content, or want to contribute your own examples, please don't hesitate to open an issue or submit a pull request on GitHub.

Star the Repository ⭐️

If you find the devops-basics repository helpful and haven't starred it yet, I'd greatly appreciate your support!

Star devops-basics ⭐️ on GitHub

Thank you once again for being part of this journey. Let's continue learning and growing together in the exciting world of DevOps! Happy coding! 🚀

Introducing DevOps Toolkit CLI 🐳

Tung Leo — Sun, 01 Sep 2024 03:26:28 +0000

Background

If you've been following the DevOps Toolkit project, you know how it simplifies the management of multiple DevOps tools in a containerized environment. We've continuously improved the toolkit, from its initial release to the recent addition of Configuration Reusability. Today, I'm excited to introduce our latest feature: the DevOps Toolkit CLI.

The DevOps Toolkit CLI

The DevOps Toolkit CLI is a powerful new tool designed to enhance your experience with the DevOps Toolkit. It provides a streamlined interface for managing your DevOps Toolkit containers, making it easier than ever to initialize, run, update, and clean up your DevOps environments.

Key Features of DevOps Toolkit CLI

Multi-Container Management: Initialize and manage multiple DevOps Toolkit containers, each with its own configuration.
Easy Initialization: Quickly set up new DevOps Toolkit containers with a single command.
Efficient Updates: Update your containers to the latest DevOps Toolkit image with ease.
Streamlined Execution: Run commands in your containers without having to manually attach to them.
Simple Cleanup: Remove individual containers or clean up all DevOps Toolkit containers at once.

Getting Started with DevOps Toolkit CLI

Prerequisites

Ensure you have Docker installed on your system. If you're new to Docker, check out this Docker guide to get started.

Installation

curl -o devops-toolkit-cli https://raw.githubusercontent.com/tungbq/devops-toolkit/main/devops-toolkit-cli
chmod +x devops-toolkit-cli
sudo mv devops-toolkit-cli /usr/local/bin/

Basic Usage

Here are some example commands to get you started with the DevOps Toolkit CLI:

Navigate to your workspace folder:

# For example:
cd /home/user/working/coding/

Initialize a new container:

devops-toolkit-cli init my_toolkit_01

Run a command in a container:

devops-toolkit-cli run my_toolkit_01 python3 --version

Update a container to the latest DevOps Toolkit image:

devops-toolkit-cli update my_toolkit_01

Clean up a specific container:

devops-toolkit-cli cleanup my_toolkit_01

Clean up all DevOps Toolkit containers:

devops-toolkit-cli cleanup --all

List all initialized containers:

devops-toolkit-cli list

For more commands instruction, run devops-toolkit-cli help:

➜ devops-toolkit-cli help

Usage: /usr/local/bin/devops-toolkit-cli [command] <container_name> [options]
Commands:
  init <container_name> [version]: Initialize a new container and configuration directory. If version is omitted, initializes with the latest image.
  run <container_name> [command]: Start a new container and run a command
  exec <container_name> [command]: Execute a command in an existing container
  cleanup --all: Remove all DevOps Toolkit containers
  cleanup <container_name>: Remove the specified container
  update <container_name> [version]: Update the container image. If version is omitted, updates to latest.
  version <container_name>: Display version information for the specified container
  list-versions: List available versions of the DevOps toolkit
  health <container_name>: Perform a health check on the specified running container
  logs <container_name>: View the logs of the specified running container
  shell <container_name>: Get a shell inside the specified running container
  list: List all initialized containers

Conclusion

The DevOps Toolkit CLI is a significant step forward in our mission to simplify DevOps workflows. By providing an efficient way to manage your DevOps Toolkit containers, we're making it easier than ever to leverage the power of our pre-configured DevOps environment.

We encourage you to try out the DevOps Toolkit CLI and share your feedback. Your input is invaluable in helping us improve and expand the capabilities of this tool.

Happy coding, and may your DevOps journey be smoother than ever! 💖

Star devops-toolkit ⭐️ on GitHub

Introducing the cmd: Essential commands for your daily work 🔖

Tung Leo — Wed, 14 Aug 2024 17:40:05 +0000

Background

As a software engineer, are you tired of constantly searching for that one CLI command you can't quite remember? Look no further! I'm excited to introduce my new CLI Commands Reference project (cmd), a comprehensive collection of essential command-line commands for your daily tasks.

Project Overview

This newly created project aims to be your go-to resource for quickly looking up commands across a wide range of technologies.
The cmd repository collects essential command-line commands for a software engineer's daily tasks 📑

Website: tungbq.github.io/cmd (auto-deploy on source code changed)
Source code: tungbq/cmd ⭐
Contributing guideline: here 📖

As this is a growing project and may not be the perfect version as of now. So if you have any tools or commands you'd like to see added, please feel free to comment or create a pull request. Your contributions are what will make this resource truly valuable for the entire community.

Now lets check the content that we had so far in the first release v0.1.0 🚀

Linux Commands
SSH Commands
JQ Commands
Git Commands
Docker Commands
Kubernetes Commands
Helm Commands
Ansible Commands
Terraform Commands
PostgreSQL Commands
Python Commands
Azure CLI Commands
...and more upcoming content, stay tuneeeeed!

Linux Commands

# Clear terminal output
clear

# List files in the current directory with details
ls -la

# Check system stats
df -h
cat /etc/os-release
cat /proc/meminfo
nproc

# Working with linux service
systemctl status target_service
systemctl start target_service
systemctl stop target_service
journalctl -u target_service

# Change file mode
chmod +x some_file.sh

# Change directory owner
chown 0600 /path/to/directory

# Create new directory
mkdir -p /path/to/directory

# Change directory
cd /path/to/directory

# Copy a file or directory
cp source destination

# Move or rename a file or directory
mv old-name new-name

# Remove a file
rm file-name

# Show disk usage of directories and files
du -h --max-depth=1

# Search for a pattern in files
grep -r "pattern" /path/to/search

# Display the current directory's path
pwd

# History
history

# CURL, add '-k' to ignore certificate verification
curl https://abc.example.com

# wget - Download a file from a URL
wget http://example.com/file.zip

# wget - Download a file and save it with a different name
wget -O newfile.zip http://example.com/file.zip

# Check network
netstat -anpt

# Check system metric (CPU/RAM/...)
top

# Tar
tar -cvf sampleArchive.tar /home/sampleArchive

# Extract
tar -xvf sampleArchive.tar
tar -xvf sampleArchive.tar.gz -C /home/ExtractedFiles/

SSH Commands

# Generate SSH keys
ssh-keygen -t rsa

# SSH to a server using username/password
ssh user@1.2.3.4

# SSH to a server using key
ssh -i user_key.pem user@1.2.3.4

# Use locally available keys to authorise logins on a remote machine
ssh-copy-id user@1.2.3.4

JQ Commands

# Pretty-print JSON data
jq '.'

# Extract a specific field from a JSON object
jq '.fieldName' file.json

# Filter JSON data by a specific condition
jq 'select(.fieldName == "value")' file.json

# Parse JSON from an API response
curl -s http://api.example.com/data | jq '.'

# Flatten a nested JSON structure
jq '.[].nestedField' file.json

# Format JSON output into a more readable structure
jq '.' file.json | less

Git Commands

# Clone repo via HTTPS
git clone https://github.com/REPO_OWNER/repo.git

# Clone repo via SSH
git clone git@github.com:REPO_OWNER/repo.git

# Create a new branch
git checkout -b new-branch

# Pull the latest changes from the remote repository
git pull origin main

# Check the status of your working directory
git status

# Add all changes to the staging area
git add .

# Commit changes with a message
git commit -m "Your commit message"

# View log
git log

# Push to remote feature branch
git push origin new-branch

# Reset staging area to match the lastest origin commit
git reset origin/main

Docker Commands

# List all running containers
docker ps

# List all containers (including stopped ones)
docker ps -a

# Build an image from a Dockerfile
docker build -t your-image-name .

# Run a container from an image
docker run --name container-name your-image-name

# Stop a running container
docker stop container-name

# Remove a stopped container
docker rm container-name

# Remove an image
docker rmi image-name

# Show container logs
docker logs container-name

# List all Docker images
docker images

# Enter a running container's shell
docker exec -it container-name /bin/bash

# Cleanup docker (add `-f` flag to force cleanup)
docker system prune

## Docker Compose
# Start up containers as defined in the Docker Compose file
docker compose up

# Start up containers in detached mode
docker compose up -d

# Stop running containers
docker compose down

Kubernetes Commands

# Export KUBECONFIG to access the cluster
export KUBECONFIG=/path/to/kubeconfig.conf

# Check nodes
kubectl get nodes

# Get all pods in the target namespace
kubectl get pods -n your_namespace

# Get all services in the current namespace
kubectl get svc -n your_namespace

# Get all resources in all namespace
kubectl get all -A

# Describe a pod
kubectl describe pod pod-name

# Apply a configuration to a resource by filename or stdin
kubectl apply -f config-file.yaml

# Delete resources by filenames, stdin, resources, and names
kubectl delete -f config-file.yaml

# Get logs from a pod
kubectl logs pod-name

# Execute a command in a container
kubectl exec -it pod-name -- /bin/bash

# Forward a port from a pod to the local machine
kubectl port-forward pod-name 8080:80

Helm Commands

# Add a Helm repository
helm repo add repo-name https://example.com/repo

# Update all Helm repositories
helm repo update

# Search for charts in a Helm repository
helm search repo repo-name

# Install a Helm chart
helm install release-name repo-name/chart-name

# List all Helm releases across all namespaces
helm list --all-namespaces

# Uninstall a Helm release
helm uninstall release-name

# Upgrade an existing Helm release
helm upgrade release-name repo-name/chart-name

# Get the status of a Helm release
helm status release-name

# Show the history of a Helm release
helm history release-name

Ansible Commands

# Run an Ansible playbook
ansible-playbook playbook.yml -i inventory.ini

# Run a single Ansible ad-hoc command
ansible all -m ping

# List all available Ansible roles
ansible-galaxy list

# Install a role from Ansible Galaxy
ansible-galaxy install role-name

# Test an Ansible inventory file
ansible-inventory --list -i inventory.ini

# Run an Ansible playbook with increased verbosity
ansible-playbook playbook.yml -vvv

Terraform Commands

# Initialize a Terraform configuration directory
terraform init

# Plan the changes required by the Terraform configuration
terraform plan

# Apply the changes required by the Terraform configuration
terraform apply

# Create a Terraform execution plan and save it to a file (recommended)
terraform plan -out="tfplan.out"

# Apply the changes from saved plan
terraform apply "tfplan.out"

# Apply the changes automatically (use with your own risk, suitable for automation tasks)
terraform apply --auto-aprove

# Destroy all the resources managed by Terraform
terraform destroy

# Validate the Terraform configuration files
terraform validate

# Format Terraform configuration files
terraform fmt

# Show the current state of Terraform-managed infrastructure
terraform show

# Refresh the state file with the real infrastructure
terraform refresh

# List the available Terraform providers
terraform providers

PostgreSQL Commands

# Connect to a PostgreSQL database
psql -h hostname -U username -d database_name

# List all databases
psql -c "\l"

# List all tables in the current database
psql -c "\dt"

# Show the structure of a specific table
psql -c "\d table_name"

# Create a new database
createdb new_database_name

# Drop an existing database
dropdb database_name

# Create a new user
createuser new_user_name

# Drop an existing user
dropuser user_name

# Grant all privileges on a database to a user
psql -c "GRANT ALL PRIVILEGES ON DATABASE database_name TO user_name"

# Revoke all privileges on a database from a user
psql -c "REVOKE ALL PRIVILEGES ON DATABASE database_name FROM user_name"

# Backup a PostgreSQL database to a file
pg_dump database_name > backup_file.sql

# Restore a PostgreSQL database from a backup file
psql database_name < backup_file.sql

Python Commands

# Run a python script
python python_script.py

# Install a package using pip
pip install package-name

# Install a specific version of a package
pip install package-name==1.2.3

# Uninstall a package
pip uninstall package-name

# List all installed packages
pip list

# Freeze installed packages into a requirements file
pip freeze > requirements.txt

# Install packages from a requirements file
pip install -r requirements.txt

Azure CLI Commands

# Login to Azure
az login

# Login with device code
az login --use-device-code

# Get the details of a subscription
az account show

# List all subscriptions associated with your account
az account list --output table

# Set a specific subscription as default
az account set --subscription "SUBSCRIPTION_ID"

# List all resource groups
az group list --output table

...and I am sure there are more upcoming content, stay tuneeeeed!

Conclusion

This CLI Commands Reference project is a living document, and I'm committed to keeping it updated and expanding it with your help. I hope this resource will become an useful resource for your daily work.

Final words, thank you for reading this post. If you have suggestions for new commands, improvements to existing ones, or even new categories to add, please don't hesitate to get involved. Let's work together to create the most comprehensive and up-to-date CLI reference for software engineers everywhere, your contributions can make a significant impact!

Don't forget to star the repository or bookmark the GitHub page tungbq.github.io/cmd if you found it useful, and happy coding ❤️!

Star tungbq/cmd ⭐️ on GitHub

DevOps Basics: 3 new topics to boost your skills ✨

Tung Leo — Tue, 30 Jul 2024 17:05:25 +0000

In this post I will keep it short, I'm excited to inform that the DevOps Basics repository on GitHub has just expanded its offerings with 3 exciting new topics (via version v0.30.0). This update continues the project's mission to provide a comprehensive resource for DevOps enthusiasts and professionals alike.

1. HAProxy: Load Balancing Made Easy

HAProxy, a reliable, high-performance TCP/HTTP load balancer, is now part of the DevOps Basics curriculum. This addition will help you understand how to efficiently distribute network traffic across multiple servers, ensuring high availability and improved performance for your applications. See: devops-basics/topics/haproxy

2. Agile and Scrum: Embracing Modern Development Methodologies

The repository now includes a section on Agile and Scrum methodologies. This topic is crucial for understanding the iterative, collaborative approach that underpins many DevOps practices. Learn how these methodologies can streamline your development process and improve team productivity. See: devops-basics/topics/agile

3.Packer: Automating Machine Image Creation

Hashicorp's Packer tool has been added to the lineup, complete with hands-on examples. Packer allows you to create identical machine images for multiple platforms from a single source configuration. This addition will teach you how to automate the creation of consistent, reproducible machine images for your infrastructure. See: devops-basics/topics/packer

These new topics, along with continuous updates to existing content, make DevOps Basics an even more valuable resource for anyone looking to enhance their DevOps skills. Whether you're a beginner or an experienced practitioner, there's always something new to learn in this ever-evolving field.

Check out the DevOps Basics repository on GitHub to explore these new topics and much more. Also let me know if you have any feedback or topic would like to add to the devops basics repository. Thank you and happy learning! 🔥

Star devops-basics ⭐️ on GitHub

Automating GitLab Runner Setup on AWS EC2 Using Terraform 🦊☁️

Tung Leo — Thu, 25 Jul 2024 17:57:42 +0000

Introduction 👋

As DevOps engineers, our daily tasks often involve setting up and managing infrastructure to support CI/CD pipelines. Automating these processes can significantly enhance our efficiency and consistency. In this blog post, I'll walk you through provisioning and configuring a GitLab Runner on an AWS EC2 instance using Terraform. This setup will allow you to integrate GitLab CI/CD with AWS infrastructure seamlessly.

Prerequisites

Before we start, make sure you have the following:

A GitLab account: https://gitlab.com/
Create new group > new project on GitLab if needed, e.g: thedevopshub/hello-gitlab.git
An active AWS account: https://signin.aws.amazon.com/signup?request_type=register
Terraform installed on your machine
AWS CLI installed and configured on your machine

You can refer to my previous Terraform post here to learn more about Terraform

Steps Overview

Prepare the environment
Configure Terraform
Provision the GitLab Runner on AWS
Verify GitLab CI/CD pipeline runs on new EC2 runner
Clean up resources

NOTE: All the code for this blog post are available at: https://github.com/TheDevOpsHub/TerraformHub/tree/main/AWS/aws-ec2-gitlab-runner

Step 1: Prepare the Environment

Create an AWS Key Pair:

Follow the AWS documentation to create a key pair: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html

Generate a GitLab Runner Registration Token:

Navigate to your GitLab project or group, then go to Settings > CI/CD > Runners, and register a new runner to get the token. Register GitLab Runner

Clone the Repository:

Clone the repository containing the Terraform configuration or create a new directory for your Terraform files.

git clone git@github.com:TheDevOpsHub/TerraformHub.git
# or creating your file manually: `mkdir -p TerraformHub/AWS/aws-ec2-gitlab-runner`
cd TerraformHub/AWS/aws-ec2-gitlab-runner

Step 2: Configure Terraform

Create a main.tf file with the following content to define the AWS resources:

resource "aws_security_group" "gitlab_runner" {
  name        = "gitlab-runner-sg"
  description = "Security group for GitLab Runner"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "gitlab_runner" {
  # Check all available us-east-1 AMIs at: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#AMICatalog
  ami             = "ami-03972092c42e8c0ca" # Amazon Linux 2 AMI
  instance_type   = var.instance_type
  key_name        = var.key_name
  security_groups = [aws_security_group.gitlab_runner.name]

  user_data = templatefile("install_runner.tpl", {
    gitlab_runner_registration_token = var.gitlab_runner_registration_token
  })

  tags = {
    Name = "AWS EC2 GitLab Runner"
  }
}

Create an install_runner.tpl file for the user data script:

#!/bin/bash
# Install necessary dependencies
# set -x enables a mode of the shell where all executed commands are printed to the terminal
set -x
echo "Hello from EC2 user data script"

yum update -y
yum install -y curl git

# Install GitLab Runner
curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
chmod +x /usr/local/bin/gitlab-runner


useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Start the GitLab Runner service
/usr/local/bin/gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
/usr/local/bin/gitlab-runner start

# Register the GitLab Runner
/usr/local/bin/gitlab-runner register --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "${gitlab_runner_registration_token}" \
  --executor "shell" \
  --description "AWS GitLab Runner" \
  --tag-list "aws,linux" \
  --run-untagged="true" \
  --locked="false"

systemctl status -l gitlab-runner.service

Create a variables.tf file to define variables:

variable "aws_region" {
  description = "The AWS region to create resources in."
  type        = string
  default     = "us-east-1"
}

variable "instance_type" {
  description = "The type of instance to create."
  type        = string
  default     = "t2.micro"
}

variable "key_name" {
  description = "The name of the key pair to use for the instance."
  type        = string
  default     = "ec2-ssh-key" # change to yours
}

variable "gitlab_runner_registration_token" {
  description = "The GitLab Runner registration token."
  type        = string
  sensitive   = true
}

Create a terraform.tfvars file:

aws_region = "us-east-1"
instance_type = "t2.micro"
key_name = "your-key-name"
gitlab_runner_registration_token = "your-registration-token"

Create an output.tf file to display outputs:

output "instance_id" {
  description = "The ID of the instance."
  value       = aws_instance.gitlab_runner.id
}

output "instance_public_ip" {
  description = "The public IP of the instance."
  value       = aws_instance.gitlab_runner.public_ip
}

Step 3: Provision the GitLab Runner on AWS

Initialize Terraform:

Navigate to your project directory and initialize Terraform.

terraform init

Plan the Configuration:

Create a plan to see the actions Terraform will take.

terraform plan -out "runner.tfplan"

Apply the Configuration:

Apply the Terraform plan to provision the infrastructure.

terraform apply "runner.tfplan"

Verify the Deployment:

After Terraform completes, verify that the GitLab Runner is running on the EC2 instance.

# Connect to your instance
ssh -i "your-key-name.pem" ec2-user@your-instance-public-ip

# Check GitLab Runner status
systemctl status gitlab-runner.service

On GitLab, navigate to your project > Setting > CICD > Expand the runner you would see the new active runner available.

Step 4: Verify GitLab CI/CD pipeline runs on new EC2 runner

Add the .gitlab-ci.yml file to the root directory of main branch:

# .gitlab-ci.yml
stages:
  - verify-runner

verify-runner:
  stage: verify-runner
  script:
    - echo "Hello, World!"
    - cat /etc/os-release
    - hostname -f
    - date
  tags:
    - aws

Commit the change and our pipeline will be run successfully on your newly registered EC2 runner:

Pipeline flow:

Logs:

Step 5: Clean Up Resources

Once you no longer need the runner, you can destroy the resources to avoid unnecessary costs:

terraform destroy

Conclusion

By following these steps, you can automate the setup of a GitLab Runner on AWS EC2 using Terraform. This approach not only saves time but also ensures consistency and repeatability in your CI/CD pipeline infrastructure. If you have any questions or suggestions, feel free to leave a comment below.

Want to learn more about DevOps? Visit https://github.com/TheDevOpsHub to explore various repositories from document/hands-on/practical guide

Thank you for reading and Happy DevOps-ing! 🚀

The Terraform Getting Started Guide 🚀

Tung Leo — Fri, 12 Jul 2024 16:51:41 +0000

As a DevOps engineers, our daily routine often revolves around deploying and managing infrastructure. Mastering the features and functionalities Terraform offers is one of the best investments you can make in yourself as a DevOps.

In this post we will go through the Terraform introduction, concepts, then get hands on by building, changing, destroying the AWS EC2 instance.
If you think I've overlooked something or should delve deeper into a particular topic, please leave a comment below. I'll update this post accordingly. Thanks in advance!

Now let’s get started 🔥🔥🔥

What is Terraform?

Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision a datacenter infrastructure using a declarative configuration language known as HashiCorp Configuration Language (HCL), or optionally JSON.

Key Concepts

Understanding the core concepts of Terraform is crucial to effectively managing infrastructure. Here are the foundational elements:

Providers: Providers are responsible for understanding API interactions and exposing resources. AWS, Azure, and Google Cloud are all examples of providers.
Resources: Resources are the most important element in the Terraform language. Each resource block describes one or more infrastructure objects, such as virtual networks, compute instances, or higher-level components such as DNS records.
Modules: Modules are containers for multiple resources that are used together. They are the primary way to package and reuse resource configurations.
State: Terraform state is used to map real-world resources to your configuration, keep track of metadata, and improve performance for large infrastructures. The state is stored in a .tfstate file.
Provisioners: Provisioners are used to execute scripts on a local or remote machine as part of the resource creation or destruction.
Data Sources: Data sources allow Terraform to use information defined outside of Terraform, defined by another separate Terraform configuration, or modified by functions.

Installing Terraform

Check this document: https://developer.hashicorp.com/terraform/install to install terraform on your machine.

Terraform with AWS Setup

In this guide we will demonstrate the integration between Terraform and AWS infrastructure, to setup AWS with Terraform you will need:

The Terraform CLI (1.2.0+) installed.
The AWS CLI installed (https://aws.amazon.com/cli/).
AWS account and associated credentials that allow you to create resources (https://aws.amazon.com/console/).

To use your IAM credentials to authenticate the Terraform AWS provider, to set the AWS credentials permanently, use aws configure:

aws configure

## Then input your AWS information:
# AWS Access Key ID [****************53GT]:
# AWS Secret Access Key [****************IGNx]:
# Default region name [us-east-1]:
# Default output format [None]:

## Check your credential by running:
aws configure list

## Output
# ➜  ~ aws configure list
#       Name                    Value             Type    Location
#       ----                    -----             ----    --------
#    profile                <not set>             None    None
# access_key     ****************53GT shared-credentials-file
# secret_key     ****************IGNx shared-credentials-file
#     region                us-east-1      config-file    ~/.aws/config

Your First Terraform Configuration

Create a new directory for your project and navigate into it:

mkdir terraform-aws-demo
cd terraform-aws-demo

Create a file named main.tf and open it in your favorite text editor. This file will contain your Terraform configuration.

Here’s a simple configuration to create an EC2 Ubuntu 22.04 instance on AWS:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "example" {
  ami             = "ami-0e001c9271cf7f3b9" # Ubuntu 22.04
  instance_type = "t2.micro"

  tags = {
    Name = "Terraform Demo"
  }
}

Check all available AMI in us-east-1 region at: https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#AMICatalog

NOTE: You could find the terraform code in this blog post here

Initializing Terraform

Before Terraform can perform any operations, it needs to be initialized. This is done by running:

terraform init

## Sample output:
# Initializing the backend...
# Initializing provider plugins...
# - Finding latest version of hashicorp/aws...
# - Installing hashicorp/aws v5.58.0...
# - Installed hashicorp/aws v5.58.0 (signed by HashiCorp)
# ...
# Terraform has been successfully initialized!

This command downloads the necessary provider plugins and prepares your environment.

Creating the Infrastructure

To see what Terraform will do before actually applying changes, run:

terraform plan

If everything looks good, apply the changes:

terraform apply

## Sample output:
# Enter a value: yes
# aws_instance.example: Creating...
# aws_instance.example: Still creating... [10s elapsed]
# aws_instance.example: Still creating... [20s elapsed]
# aws_instance.example: Still creating... [30s elapsed]
# aws_instance.example: Creation complete after 38s [id=i-099dc08ae004bb7f7]

# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Terraform will prompt you to confirm. Type yes to proceed. Terraform will now provision the resources defined in your configuration.

Now visit AWS EC2 Console you can see your Terraform Demo instance up and running:

Inspecting the State

Terraform's state is stored in a file named terraform.tfstate. You can inspect the current state by running:

terraform show

## Output
# aws_instance.example:
# resource "aws_instance" "example" {
#     ami                                  = "ami-0e001c9271cf7f3b9"
#     arn                                  = "arn:aws:ec2:us-east-1:992382371456:instance/i-099dc08ae004bb7f7"
#     associate_public_ip_address          = true
#     availability_zone                    = "us-east-1b"
#     cpu_core_count                       = 1
#     ...
#     host_id                              = null
#     iam_instance_profile                 = null
#     id                                   = "i-099dc08ae004bb7f7"
#     instance_initiated_shutdown_behavior = "stop"
#     instance_lifecycle                   = null
#     instance_state                       = "running"
#     instance_type                        = "t2.micro"
# ...other information

This command displays all the infrastructure managed by Terraform.

Modifying Infrastructure

To make changes to your infrastructure, modify the main.tf file. For example, you can modify the tags associated with the EC2 instance:

resource "aws_instance" "example" {
  ami           = "ami-0e001c9271cf7f3b9" # Ubuntu 22.04
  instance_type = "t2.micro"

  tags = {
    Name = "Updated Terraform Demo" # Update instance tag name
    Environment = "Development" # Adding new tag
  }
}

Initializing Changes

After modifying your configuration, initialize Terraform again to update its state and refresh dependencies:

terraform init

Applying Changes

After initializing, you can apply your changes to update the infrastructure:

terraform apply

## Sample output
# aws_instance.example: Refreshing state... [id=i-099dc08ae004bb7f7]

# Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
#   ~ update in-place

# Terraform will perform the following actions:
#   # aws_instance.example will be updated in-place
#   ~ resource "aws_instance" "example" {
#         id                                   = "i-099dc08ae004bb7f7"
#       ~ tags                                 = {
#           + "Environment" = "Development"
#           ~ "Name"        = "Terraform Demo" -> "Updated Terraform Demo"
#         }
#       ~ tags_all                             = {
#           + "Environment" = "Development"
#           ~ "Name"        = "Terraform Demo" -> "Updated Terraform Demo"
#         }
#         # (38 unchanged attributes hidden)
#         # (8 unchanged blocks hidden)
#     }

Confirm by typing yes when prompted. Terraform will then update the tags for the existing EC2 instance to reflect your changes.

Visit AWS EC2 Console you can see your instance name is changed to Updated Terraform Demo:

Using Variables and Outputs

Variables in Terraform allow you to parameterize your configurations. They can make your configurations more dynamic and reusable. Here's an example of using variables:

Create a variables.tf file:

variable "instance_name" {
  description = "Name of the EC2 instance"
  default     = "Variable Terraform Demo"
}
variable "environment" {
  description = "Name of the environment"
  default     = "development"
}

Modify main.tf to use the variable:

resource "aws_instance" "example" {
  ami           = "ami-0e001c9271cf7f3b9" # Ubuntu 22.04
  instance_type = "t2.micro"

  tags = {
    Name = var.instance_name
    Environment = var.environment
  }
}

In this example, instance_name is a variable that defaults to "Variable Terraform Demo". You can override this value when running terraform apply.

Outputs in Terraform allow you to extract and display information about your infrastructure. Add an outputs.tf file:

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.example.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.example.public_ip
}

Apply your configuration by runnint terraform apply:

terraform apply
## Sample output
# aws_instance.example: Modifying... [id=i-099dc08ae004bb7f7]
# aws_instance.example: Modifications complete after 7s [id=i-099dc08ae004bb7f7]

# Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

# Outputs:

# instance_id = "i-099dc08ae004bb7f7"
# instance_public_ip = "44.202.163.210"

Now you can view the output by running:

# Get Instance ID
terraform output instance_id
## i-099dc08ae004bb7f7

# Get Public IP
terraform output instance_public_ip
## "x.x.x.x"

Destroying Infrastructure

To tear down the infrastructure managed by Terraform, you can run:

terraform destroy

## Output:
# ...
# Do you really want to destroy all resources?
#   Terraform will destroy all your managed infrastructure, as shown above.
#   There is no undo. Only 'yes' will be accepted to confirm.

#   Enter a value: yes

# aws_instance.example: Destroying... [id=i-099dc08ae004bb7f7]
# aws_instance.example: Still destroying... [id=i-099dc08ae004bb7f7, 10s elapsed]
# aws_instance.example: Still destroying... [id=i-099dc08ae004bb7f7, 20s elapsed]
# aws_instance.example: Still destroying... [id=i-099dc08ae004bb7f7, 30s elapsed]
# aws_instance.example: Still destroying... [id=i-099dc08ae004bb7f7, 40s elapsed]
# aws_instance.example: Destruction complete after 44s

# Destroy complete! Resources: 1 destroyed.

Confirm by typing yes when prompted. Terraform will then proceed to destroy all resources defined in your configuration.

Best Practices

Here are some best practices to follow when working with Terraform:

Use Version Control: Always keep your Terraform configuration files in version control (e.g., Git).
Use Remote State: Store your state file remotely to collaborate with others and avoid potential issues with local state files.
Modularize: Use modules to organize your configuration into reusable components.
Backup State Files: Regularly back up your state files to prevent data loss.
Use Variables and Outputs: Use variables to make your configuration flexible and outputs to expose information about your infrastructure.

Common Commands Cheat Sheet

terraform init - Initialize a Terraform configuration
terraform plan - Show changes required by the current configuration
terraform apply - Apply the changes required by the current configuration
terraform destroy - Destroy the Terraform-managed infrastructure
terraform fmt - Reformat your configuration in the standard style
terraform validate - Validate the configuration files in a directory
terraform state - Advanced state management
terraform output - Show output values from a state file

Useful Resources

Here are some tools and resources to help you along your Terraform journey:

terraform/docs
Awesome Terraform
Terraform Hub
Book: Terraform: Up and Running

Summary

In this post, we introduced the fundamentals of Terraform, covering key concepts like providers, resources, and modules. We demonstrated installing Terraform, creating a basic EC2 instance on AWS, and performing essential operations like initializing, applying, modifying, and destroying infrastructure. Additionally, we highlighted best practices and basic usage of variables and outputs to enhance your Terraform configurations.

I hope this help you for the Terraform journey. Thank you for reading and happy coding! 💖

Introducing the new feature of DevOps Toolkit: Configuration Reusability 🚀

Tung Leo — Tue, 09 Jul 2024 15:11:18 +0000

Background

In the world of DevOps, managing multiple tools on your computer can be quite a challenge. I know this struggle firsthand. Setting up each tool, ensuring they work together, and keeping them updated is a time-consuming and often frustrating process.

That's why I created the DevOps Toolkit to solve these problems. It's built on top of the Docker platform. I wanted to make it easier for developers and operations teams to get started with DevOps without the headaches of tool compatibility, setup, maintenance, and keeping everything up to date.
You could find my previous post for this tool here

The DevOps Toolkit 🧰

GitHub repository: tungbq/devops-toolkit
Image on Dockerhub: tungbq/devops-toolkit:latest
Description: DevOps toolkit is a container image for an all-in-one DevOps environment with popular tools like Ansible, Terraform, kubectl, helm, AWS CLI, Azure CLI, Git, Python, and more...

Announcing Configuration Reusability

In our latest release, we're excited to introduce a powerful new feature: Configuration Reusability. This feature allows you to mount a configuration folder from the host to the container, enabling the reuse of configurations within the container, such as AWS and Azure login sessions. This enhancement simplifies the workflow, ensuring that you don't have to log in repeatedly and can maintain a consistent environment across multiple runs.

Key Updates in This Release 🚀

Core Updates:
- Updated tool versions
New Features:
- Enabled toolkit container configuration reusable
Improvements:
- Installed SSH client
- Ansible documentation improvement and minor image name change
- Updated deploy-docker-image-release.yml to replace deprecated set-output
- Updated instructions
- Adjusted note in quick start section

You can checkout all devops-toolkit releases and new features here.

Quick Start Guide 📖

Before you begin, ensure that you have Docker installed. It's also helpful to have a basic understanding of Docker concepts. If you are new to Docker, don't worry, you can refer to this docker document to get started.

Step 1: Pull the Official Image from Docker Hub

DockerHub image: tungbq/devops-toolkit

docker pull tungbq/devops-toolkit:latest

Step 2: Start and Explore the Toolkit Container

To start using the toolkit, run the following command:

mkdir -p ~/.devops-toolkit-config
docker run --network host -it --rm -v ~/.devops-toolkit-config:/config tungbq/devops-toolkit:latest

This command mounts your host's .devops-toolkit-config directory to the container, allowing the reuse of configurations.

Step 3: Work on the Toolkit Container

Once inside the container, you can try login to AWS and/or Azure with aws or az CLI:

root@docker-desktop:~# aws configure
root@docker-desktop:~# az login --use-device-code

Exit the container and move to the next step

Step 4: Verify the Configuration Reusability

Run the new container, you can find your previous configurations without login or configure again:

docker run --network host -it --rm -v ~/.devops-toolkit-config:/config tungbq/devops-toolkit:latest
root@docker-desktop:~# aws configure list
root@docker-desktop:~# az account show

Similar feature will happen with other tools like Ansible/Terraform/...

Conclusion

In summary, the DevOps Toolkit simplifies the complexities of managing multiple DevOps tools and keeping them updated, in this new Configuration Reusability feature it will help us more. If you're interested, give it a try, share your feedback, and let's continue improving together. Happy coding! 💖

Star devops-toolkit ⭐️ on GitHub

The Azure Hub ☁️ Your Azure learning resources 📚

Tung Leo — Sat, 15 Jun 2024 09:58:47 +0000

Greetings 👋

Hi there! Tung Leo here. Previously, I've published The AWS Hub ☁️ Your new AWS learning resources post to help everyone with the AWS learning repository.

This time, we'll dive into another cloud provider - the Azure cloud, and I'm excited to introduce you to a new comprehensive repository packed with Azure learning resources and documentation. This is AzureHub ⭐

Introduction ☁️

Whether you're a new member of the Azure community or a developer/cloud engineer seeking a centralized document hub for all things Azure, the AzureHub repository will assist you. Consider this repository as your bookmark for all essential Azure resources, including Azure Certification materials, documentation for Azure services, and Azure Architecture insights. 🔥

In the following sections, I'll walk you through key content within the AzureHub repository.

Getting started 🚀

What is Azure? ➡️ Watch here
Azure Documentation: 📖 Explore here
Azure Portal: Explore here

Azure Services Learning Resources 📘

This section provides links to detailed documentation, introduction videos, and FAQs for popular Azure services
For the full Azure services learning resources, visit: Azure_Services.md

Azure Certification Resources 💯

Preparing for the AWS Certification Exam? Here are my top personal recommendations for learning resources:

1. Free Exam Preparation

Certification exam preparation by Azure: here
Study guide: Study guide for Exam

2. Guides/Cheat Sheets

Tutorials Dojo - Azure Cheat Sheets: https://tutorialsdojo.com/microsoft-azure-cheat-sheets/

3. Azure Certifications Sub-Reddit

r/AzureCertification brings in your discussions, questions , opinions, news and comments around Azure certifications areas like prep tips, clarifications, lessons learned.

4. Specific Azure Certification resources

	ID	Certification Name	Learning path
1	Azure 900	Microsoft Certified: Azure Fundamentals	📖 az-900.md
2	Azure 104	Microsoft Certified: Azure Administrator Associate	📖 az-104.md

And more upcoming certification resources ⏩ ...

Azure Training 📚

https://learn.microsoft.com/en-us/training/azure/

Azure Architecture Center

Azure Architecture: 📖 Explore here

Learn Azure on Youtube ▶️

Some popular Youtube channels to learn Azure

Microsoft Azure Official: https://www.youtube.com/@MicrosoftAzure
Adam Marczak - Azure for Everyone: https://www.youtube.com/@AdamMarczakYT
John Savill's Technical Training: https://www.youtube.com/@NTFAQGuy
A Guide To Cloud: https://www.youtube.com/@AGuideToCloud

Github Resources

Microsoft Learning: https://github.com/MicrosoftLearning
johnthebrit/CertificationMaterials: https://github.com/johnthebrit/CertificationMaterials

Contributing

See: CONTRIBUTING.md
If you find this repository helpful, kindly consider showing your appreciation by giving it a star ⭐ Thanks! 💖
Looking for the issue to work on? Check the list of our open issues good first issue
Feel free to open a new issue or let us know if you want to request more content about Azure

What's next?

We will keep updating this repository with the latest resources and changes. If you would like to contribute more useful resources, please open a PR, and we will review it to add to the list. Thanks 💖

Feedback 🗣️

Got any feedback, suggestions, or issues with the AzureHub repository? Feel free to reach out to me via GitHub Issues or leave a comment in this blog post 🔗

Star AzureHub ⭐️ on GitHub

Thank you and happy learning! 🎉

Jenkins up and running on Kubernetes 🚀

Tung Leo — Tue, 04 Jun 2024 16:32:02 +0000

Introduction 👋

In this hands-on, we'll cover:

Deploy Jenkins controller on k8s cluster
Configure k8s cluster as Jenkins agents
Create and run a sample pipeline on a k8s Pod Jenkins agent
Watch the Pod life cycle for a pipeline run

Environment ☁️

This hands-on is for a PoC or Pilot environment, to explore the Jenkins and Kubernetes features

Prerequisites 🔓

Before you start, ensure you have:

A running Kubernetes cluster (I used kind for my k8s local environment).
kubectl configured to interact with your cluster.
Basic knowledge of Kubernetes and Jenkins.

Documentation Reference 📖

Deploy Jenkins on Kubernetes and run your pipeline 🔥

Let's start deploying and using Jenkins on Kubernetes by following below steps:

1. Prepare K8s Manifest YAML files

Before start, we need to prepare the K8s YAML files.

NOTE: All the YAML files to deploy Jenkins controller on Kubernetes are available at: K8sHub (hands-on/jenkins-on-k8s/yamls). If you want to use the hands-on example and all-in-one script from my repo (desrible later in the next section), you do not need to create these files manually, just refer them as captured version of the ones in hands-on repository. Otherwise, create these files in your PC with following content and name:

volume.yaml: To create the persitent volumne for our Jenkins instance on k8s (Replace the demo-jenkins-cluster-control-plane by your node name)

  ## volume.yaml
  kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: local-storage
  provisioner: kubernetes.io/no-provisioner
  volumeBindingMode: WaitForFirstConsumer
  ---
  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: jenkins-pv-volume
    labels:
      type: local
  spec:
    storageClassName: local-storage
    claimRef:
      name: jenkins-pv-claim
      namespace: devops-tools
    capacity:
      storage: 10Gi
    accessModes:
      - ReadWriteOnce
    local:
      ## Replace by your desired path
      path: /mnt/jenkins
    nodeAffinity:
      required:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                ## Replace by your node name
                values:
                  - demo-jenkins-cluster-control-plane
  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: jenkins-pv-claim
    namespace: devops-tools
  spec:
    storageClassName: local-storage
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3Gi

service_account.yaml: To create jenkins-admin service account for the Deployment usage

  ## service_account.yaml
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: jenkins-admin
  rules:
    - apiGroups: ['']
      resources: ['*']
      verbs: ['*']
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: jenkins-admin
    namespace: devops-tools
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: jenkins-admin
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: jenkins-admin
  subjects:
    - kind: ServiceAccount
      name: jenkins-admin
      namespace: devops-tools

deployment.yaml: To deploy latest jenkins/jenkins:lts Jenkins version with jenkins-admin service account and us jenkins-pv-claim persistent volume

  ## deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: jenkins
    namespace: devops-tools
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: jenkins-server
    template:
      metadata:
        labels:
          app: jenkins-server
      spec:
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
        serviceAccountName: jenkins-admin
        containers:
          - name: jenkins
            image: jenkins/jenkins:lts
            resources:
              limits:
                memory: '2Gi'
                cpu: '1000m'
              requests:
                memory: '500Mi'
                cpu: '500m'
            ports:
              - name: httpport
                containerPort: 8080
              - name: jnlpport
                containerPort: 50000
            livenessProbe:
              httpGet:
                path: '/login'
                port: 8080
              initialDelaySeconds: 90
              periodSeconds: 10
              timeoutSeconds: 5
              failureThreshold: 5
            readinessProbe:
              httpGet:
                path: '/login'
                port: 8080
              initialDelaySeconds: 60
              periodSeconds: 10
              timeoutSeconds: 5
              failureThreshold: 3
            volumeMounts:
              - name: jenkins-data
                mountPath: /var/jenkins_home
        volumes:
          - name: jenkins-data
            persistentVolumeClaim:
              claimName: jenkins-pv-claim

services.yaml: To expose the jenkins-server as a k8s service

  ## services.yaml
  apiVersion: v1
  kind: Service
  metadata:
    name: jenkins-service
    namespace: devops-tools
    annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path: /
      prometheus.io/port: '8080'
  spec:
    selector:
      app: jenkins-server
    ports:
      - name: httpport
        port: 8080
        targetPort: 8080
      - name: jnlpport
        port: 50000
        targetPort: 50000

2. Deploy Jenkins Controller

You can deploy it manually or using my all-in-one deploy script, choose one of the following options:

Option 1. Deploy Jenkins with prepare scipt

Clone the repo contains the hands-on example. Then, run the deployment script:

## Checkout the source code for this hands-on
git clone https://github.com/tungbq/K8sHub.git
cd K8sHub/hands-on/jenkins-on-k8s

## NOTE: In `hands-on/jenkins-on-k8s/yamls/deployment.yaml` replace the `demo-jenkins-cluster-control-plane` value by your node name

## Create new `devops-tools` namespace
kubectl create namespace devops-tools

## Now run the deploy script
./deploy.sh

Option 2. Deploy Jenkins manually

## Ensure you created 4 k8s manifest YAML files as describle in previous section
kubectl create namespace devops-tools
kubectl apply -f sevice_account.yaml
kubectl apply -f volume.yaml
kubectl apply -f deployment.yaml
kubectl apply -f services.yaml

3. Access Jenkins Controller

3.1. Port-Forwarding

Open a new terminal and set up port forwarding to access Jenkins:

## Replace 8087 with an available port on your PC
kubectl port-forward service/jenkins-service -n devops-tools 8087:8080

3.2. Get Initial Password

Jenkins requires an initial admin password for first-time access. Retrieve it with the following commands:

## Get the Jenkins pod name
kubectl get pods --namespace=devops-tools

## Retrieve the initialAdminPassword
kubectl exec -it <pod_name> cat /var/jenkins_home/secrets/initialAdminPassword -n devops-tools
## Sample output: d72493ce44fb48bc8833da94b40cdd68

3.3. Access Jenkins

Open your browser and navigate to http://localhost:8087. Log in using the initial password obtained earlier, install the suggested plugins, and create an admin user.

4. Configure Jenkins Agents on Kubernetes

With Jenkins up and running, the next step is to configure Kubernetes as Jenkins agents.

4.1. Install Kubernetes Plugin

Navigate to Dashboard > Manage Jenkins > Plugins (http://localhost:8087/manage/pluginManager/available), search for the "Kubernetes" plugin, and install it. Restart Jenkins after installation.

4.2. Configure the Kubernetes Plugin

Navigate to Dashboard > Manage Jenkins > Clouds (http://localhost:8087/manage/cloud/). Select "New cloud" and input the cloud name (e.g., k8s-agents), then select "Create."

Configure the following fields:

Kubernetes URL: Leave empty
Kubernetes server certificate key: Leave empty
Kubernetes Namespace: devops-tools
Credentials: Leave as None
Jenkins URL: http://jenkins-service.devops-tools.svc.cluster.local:8080

Select "Test connection" to verify the connection. If you see Connected to Kubernetes vX.Y.Z, the configuration is successful.

5. Run a Sample Job on a Kubernetes Pod

Now that the Jenkins controller and k8s agents are configured, let's run a sample job on a Pod.

5.1. Create a New Pipeline

Go to the Jenkins homepage, select "New item," choose "Pipeline," input a name (e.g., Demo-Run-Shell-Inside-K8s-Pod), and create.

5.2. Define the Pipeline Script

In the pipeline configuration section, input the following script:

podTemplate(containers: [
    containerTemplate(
        name: 'jnlp',
        image: 'jenkins/inbound-agent:latest'
        )
  ]) {

    node(POD_LABEL) {
        stage('Shell') {
            container('jnlp') {
                stage('Shell Execution') {
                    sh '''
                    echo "Hello! Running from shell"
                    '''
                }
            }
        }

    }
}

5.3. Build the Pipeline

Select "Build Now" to trigger the pipeline.

Jenkins will create a new Pod based on your template and run the pipeline inside the Pod.

5.4. Monitor Pod Lifecycle

To see the Pods created and terminated with each build, use the following command:

## Check pods in devops-tools namespace
kubectl get pods -n devops-tools -w

Cleanup

To clean up your environment, delete the namespace along with its resources:

## Terminate the namespace
kubectl delete namespace devops-tools

Alternatively, run the cleanup script:

cd hands-on/jenkins-on-k8s
./cleanup.sh

Troubleshooting 🔨

Common Issues and Solutions:

Persistent volume binding issues: Delete the PV with kubectl delete pv jenkins-pv-volume and redeploy.
Connection issues to the Jenkins service: Verify the service port and check the agent pod logs using kubectl logs -f <your_pod_name> -n devops-tools.
Losing connection to Jenkins page: Re-run kubectl port-forward service/jenkins-service -n devops-tools 8087:8080 and ensure the Jenkins pod is running.

Conclusion ✒️

By following this guide, you should have a functional Jenkins setup on Kubernetes, allowing you to run pipelines within k8s Pods. Experiment with custom pipelines and explore more advanced Jenkins features on Kubernetes. Happy DevOps-ing!

I'm building the K8sHub to help everyone learn and practice Kubernetes from the basics to pratical hands-on, If you're interested in this project please cónider giving it a star ⭐️. Any star you can give me will help me grow it even more ❤️ Thank you!

Star K8sHub repo ⭐️ on GitHub

GitHub: Create your own repository landscape ⭐️

Tung Leo — Sun, 28 Apr 2024 03:20:25 +0000

Overview 👋

Do you want to create a central repository to showcase all your highlighted repositories with descriptions and stars? Something like this:

I've created the repository landscape template tungbq/repos that can help you build your own repository landscape.
This blog post will describe the features and guide you on how to use this repository template to create your own repository landscape.

Key Features 🚀

The tungbq/repos repository template provides some features that help you create your repo landscape content automatically:

Supports defining the list of repositories to be shown in the landscape, file repository_list.txt.
Automated script to generate content from the repository_list.txt.
A GitHub Action CI pipeline to generate new repository landscape content:
- Triggers the automated generation script.
- Creates a pull request to propose the new content.
The repository landscape format provides:
- Indexing with repo name title
- Repo URL
- Repo description (archived via GitHub API call)
- A GitHub stars badge (leverage this tool https://shields.io/)

Quick start 📖

This is optional; you can skip this section and jump into Configure and use the template 📘 to create your own repo.
If you want to generate the content locally, update the repository_list.txt and run this script:

git clone https://github.com/tungbq/repos.git
cd repos
./generate_content.sh repository_list.txt

Result will be similar to:

➜  repos git:(main) ✗ ./generate_content.sh repository_list.txt
Working on repo: tungbq/devops-basics, with index: 1
Working on repo: tungbq/AWSHub, with index: 2
Working on repo: tungbq/devops-toolkit, with index: 3
Working on repo: tungbq/devops-project, with index: 4
Working on repo: tungbq/aws-lab-with-terraform, with index: 5
Working on repo: tungbq/awesome-workflow, with index: 6
Working on repo: tungbq/k8sHub, with index: 7
Working on repo: tungbq/Azure-DevOps-Pipeline, with index: 8
Working on repo: tungbq/find-github-issue, with index: 9
Working on repo: tungbq/challenges, with index: 10
Working on repo: tungbq/devops-dockerfiles, with index: 11
Working on repo: tungbq/terraform-sample-project, with index: 12
Working on repo: tungbq/repos, with index: 13

Now check the README.md and you would find your repository landscape content

Configure and use the template 📘

Below are the steps to help you create and configure your own landscape repository.
In short we only need to do 4 major steps: Create new repo from template > Set up GITHUB_TOKEN > Update repository list > Merge the automated PR.
Let's dive into the detailed as below:

1. Create a new repo from the template

Visit the template repository: https://github.com/tungbq/repos.
In the top right corner, select Use this template > Create a new repository:

Input your repository name to create and add a description if needed, then select Create repository.

Wait for a few seconds, and your repository will be created.

2. Configure `GITHUB_TOKEN` for the new repo

There is a CI workflow to automatically generate content and open a new Pull Request for your repository landscape, so setting up GITHUB_TOKEN permission is required.

Goto: https://github.com/YOUR_USERNAME/YOUR_REPO/settings/actions (Repo > Setting > Action > General), in the Workflow permission section, enable following options:

Read and write permissions
Allow GitHub Actions to create and approve pull requests

Now we are ready to trigger the CI workflow to update the repo landscape.
Now we are ready to trigger the CI workflow to update the repo landscape.

3. Update your repository list

Add your repository list to the file repository_list.txt in your repository and merge it into the main branch, for example:

tungbq/devops-basics
tungbq/AWSHub
tungbq/devops-toolkit
tungbq/devops-project
tungbq/aws-lab-with-terraform
tungbq/awesome-workflow

4. Trigger the CI pipeline

Automatically trigger on merge event to main

Once the repository_list.txt is merged into the main branch, the CI pipeline Update content will be triggered automatically.

The CI pipeline will read the repository list and generate your new README.md content.
Then, it will check and create a Pull Request to propose the new repository landscape content.

Manual trigger (Optional)

You can also trigger that CI at any moment you want.
Go to Actions and select Update content.
Select "Run workflow" to trigger the CI:

5. Review and merge the PR

Once the PR is raised automatically, you just need to review and merge, and that's it! You will have your own landscape content in README.md.

Congratulations 🎉, you've successfully created your own repo landscape! Now check the README.md for the final result.

Advance CI configuration (Optional)

In the previous section, we triggered the CI pipeline to run on a merge event to main branch or manual event. You could improve that by changing the trigger event to run on a timer basis. For example, once a week or once a month. This would help us regularly check and update the landscape, keeping it up to date.
To do so, add your desired cron trigger to repos/.github/workflows/generate_content.yaml

on:
  schedule:
    - cron: '0 0 * * 6' # Run every Saturday at midnight
  workflow_dispatch:
  # other events as needed

Check https://crontab.guru/ for the CRON syntax

If you find this repository template helpful, feel free to use it, kindly consider showing your appreciation by giving it a star ⭐️. Any star you can give me will help me grow it even more ❤️

Star repos ⭐️ on GitHub

Thank you and happy coding! 🔥

Forem: Tung Leo

Wrapping Up 2024: IIS and Apache Tomcat Join the DevOps Basics Journey! 🎉

Introduction

2024 Highlights ✨

Release v0.35.0

Looking Ahead to 2025 🛠️

Thank You 💖

Introducing TheDevOpsHub: Your Central Hub for DevOps Learning! 🚀

What is TheDevOpsHub?

Featured Projects

1. The Trio DevOps Projects

2. Cloud Learning

3. Linux and OS

4. CI/CD Tools

5. Infrastructure as Code (IaC)

6. Containerization & Kubernetes

7. Monitoring

Explore More Resources...

Stay Connected and Up-to-Date

DevOps Basics reaches 200 stars: A milestone celebration 🎉

Impressive Metrics 📈

What is devops-basics?

Key Features 🚀

Thank You to the Supporters 💖

Looking Ahead 👓

Get Involved 💎

Star the Repository ⭐️

Introducing DevOps Toolkit CLI 🐳

Background

The DevOps Toolkit CLI

Key Features of DevOps Toolkit CLI

Getting Started with DevOps Toolkit CLI

Prerequisites

Installation

Basic Usage

Conclusion

Introducing the cmd: Essential commands for your daily work 🔖

Background

Project Overview

Table of Contents

Linux Commands

SSH Commands

JQ Commands

Git Commands

Docker Commands

Kubernetes Commands

Helm Commands

Ansible Commands

Terraform Commands

PostgreSQL Commands

Python Commands

Azure CLI Commands

Conclusion

DevOps Basics: 3 new topics to boost your skills ✨

1. HAProxy: Load Balancing Made Easy

2. Agile and Scrum: Embracing Modern Development Methodologies

3.Packer: Automating Machine Image Creation

Automating GitLab Runner Setup on AWS EC2 Using Terraform 🦊☁️

Introduction 👋

Prerequisites

Steps Overview

Step 1: Prepare the Environment

Create an AWS Key Pair:

Generate a GitLab Runner Registration Token:

Clone the Repository:

Step 2: Configure Terraform

Step 3: Provision the GitLab Runner on AWS

Initialize Terraform:

Plan the Configuration:

Apply the Configuration:

Verify the Deployment:

Step 4: Verify GitLab CI/CD pipeline runs on new EC2 runner

Step 5: Clean Up Resources

Conclusion

The Terraform Getting Started Guide 🚀

What is Terraform?

Key Concepts

Installing Terraform

Terraform with AWS Setup

Your First Terraform Configuration