Forem: Daniel Favour

Automating User Management on Linux using Bash Script

Daniel Favour — Tue, 02 Jul 2024 21:10:30 +0000

Efficient user management is important for maintaining security and productivity. Manual management of users and groups can be time-consuming, especially in larger organizations where administrators need to handle multiple accounts and permissions. Automating these tasks not only saves time but also reduces the risk of human error.

This guide discusses a practical approach to automating user management on a Linux machine using a Bash script. You will learn how to create users, assign groups, generate secure passwords for users, and log actions using a single bash script.

Overview of the Bash Script Functionality

Below is an overview of the tasks the Bash script will automate for efficient user management:

Read User Data: The script will read a text file containing employee usernames and their corresponding group names.
Create Users and Groups: It will create users and groups as specified in the text file.
Set Up Home Directories: The script will set up home directories for each user with appropriate permissions and ownership.
Generate Secure Passwords: It will generate random, secure passwords for the users.
Log Actions: All actions performed by the script will be logged to the /var/log/user_management.log directory.
Store Passwords Securely: Generated passwords will be securely stored in the /var/secure/user_passwords.txt directory.
Error Handling: The script will include error handling to manage scenarios such as existing users and groups.

Prerequisites

To get started with this tutorial, you must have the following:

A Linux machine with administrative privileges.
Basic knowledge of Linux commands.
A text editor of choice (vim, nano, etc)

Setting Up the User Data File

The first step is to create a text file containing the username for each employee and the groups to be assigned to each of them.

In your terminal, create a user_password.txt file:

touch user_passwords.txt

Paste the below content into the file

john;qa
jane;dev,manager
robert;marketing
emily;design,research
michael;devops
olivia;design,research
william;support
sophia;content,marketing
daniel;devops,sre
ava;dev,qa

The above is a list of usernames for the employees and their respective group(s).

Writing the Bash script

To start creating the Bash script, follow these steps in your terminal:

Create the Script file

Open your terminal and run the following command to create an empty file named create_users.sh:

touch create_users.sh

Use your preferred text editor to open the create_users.sh file and begin writing the script:

nano create_users.sh

(If you are using a different editor, replace nano with its command)

Add the Shebang Line

At the top of the create_users.sh file, include the shebang.

#!/bin/bash

This line specifies the interpreter that will be used to execute the script. In this case,#!/bin/bash indicates that the script should be run using the Bash shell.

Check Root Privileges

Creating users and groups typically requires administrative privileges because it involves modifying system files and configurations. After the shebang line, add the below configuration to ensure that the Bash script is executed with root privileges:

if [[ $(id -u) -ne 0 ]]; then
    echo "This script must be run as root."
    exit 1
fi

This checks if the script is running with root privileges. If the current user ID ($(id -u)) does not equal (-ne) 0, then the condition is true (indicating the script is not running as root), and the code within the then ... fi block will execute accordingly. In this case, it will output "This script must be run as root." to the terminal, and then the script will exit with a status of 1. This exit status is a signal to the operating system and any other processes that the script encountered an error and did not complete successfully.

Check that the Input File is Passed as an Argument

In Bash scripting, "arguments" are the values or parameters provided to a script or command when it is invoked from the command line. For example, if you have a Bash script named process_file.sh and you want to read an input file, data.txt provided as an argument, in the terminal, you will execute it with:

./process_file.sh data.txt

Inside your Bash script (process_file.sh), you can access this argument using special variables like $1, $2, etc. $1 specifically refers to the first argument passed (data.txt in this case). Once the script captures the argument ($1), it can use it in various ways. For instance, it might open and read the file specified (data.txt), process its contents, or perform any other operation that the script is designed to do.

In this case, the create_users.sh script needs to read the user data file, user_passwords.txt containing the usernames and groups of the employees so it can perform certain actions.

Paste the below configuration in your script:

if [[ $# -ne 1 ]]; then
    echo "Usage: $0 <input-file>"
    exit 1
fi

if [[ ... ]]; then ... fi: This is a conditional statement in Bash. The code inside the then ... fi block will execute only if the condition within the double square brackets [[ ... ]] evaluates to true.
if [[ $# -ne 1 ]]; then: This checks if the number of arguments ($#) passed to the script is not equal (-ne) to 1.
echo "Usage: $0 <input-file>": If the condition is true (meaning the wrong number of arguments were provided), this line prints a helpful message to the terminal explaining how the script should be used.
- Usage: A standard keyword indicating the start of usage instructions.
- $0: This is a special variable that holds the name of the script itself. It is automatically replaced with the actual name of the script when it runs (for example, "create_users.sh").
- <input-file>: This placeholder communicates to the user that they need to provide the name of the input file (for example, "user_passwords.txt") as the argument when running the script.
exit 1: This terminates the script with an exit status of 1. An exit status of 1 signals that the script encountered an error and did not complete successfully.

Assign Variables

The next step is to assign variables for essential paths and files.

INPUT_FILE=$1
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"

INPUT_FILE=$1: This line assigns the first command-line argument (the user_passwords.txt file) to the variable INPUT_FILE. This makes it easier to reference the filename throughout the script.
LOG_FILE="/var/log/user_management.log": This sets the variable LOG_FILE to the path where the script will write its log messages. This log will help track the actions performed by the script.
PASSWORD_FILE="/var/secure/user_passwords.txt": This sets the variable PASSWORD_FILE to the path where the generated passwords for the users will be stored securely.

Log Messages

Logging messages is a common practice in scripting and software development as it records what happens at each step in the script.

Add the below configuration to log messages in the $LOG_FILE directory:

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOG_FILE
}

log_message(): This function is a reusable piece of code designed to create formatted log entries and write them to a specified log file.
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1": date '+%Y-%m-%d %H:%M:%S': This generates the current date and time in the format "YYYY-MM-DD HH:MM:SS". $1 represents the message you pass to the function as the first argument. This command combines the timestamp and the message, separated by a hyphen (-), creating the formatted log entry.
tee -a $LOG_FILE: The tee command reads from standard input (in this case, the output of the echo command) and writes it to both standard output (the terminal) and to one or more files. The -a option tells tee to append to the file ($LOG_FILE) instead of overwriting it. This ensures that previous log entries are preserved.

Ensure the `/var/secure` Directory Exists

Before proceeding with user creation, it's imperative to establish a secure environment for storing the generated passwords. The passwords will be stored in the /var/secure directory, so it is necessary to check if this directory exists and configure it with appropriate permissions to limit access to authorized users only.

if [[ ! -d "/var/secure" ]]; then
    mkdir -p /var/secure
    chown root:root /var/secure
    chmod 600 /var/secure
fi

if [[ ! -d "/var/secure" ]]; then: This condition checks if the /var/secure directory exists. The -d flag checks if the path is a directory, and the ! negates the result, meaning the code within the then block will only execute if the directory does not exist.
mkdir -p /var/secure: This line creates the /var/secure directory if it does not exist. The -p option ensures that any necessary parent directories are also created.
chown root:root /var/secure: This changes the owner of the /var/secure directory to the root user and the root group. This is a security best practice, as sensitive data like passwords should only be accessible to the system administrator.
chmod 600 /var/secure: This changes the permissions of the /var/secure directory so that the owner (root) has full read and write access to the directory, no users in the root group (other than root itself) can access the directory's contents in any way and no other users on the system can access the directory's contents.
fi: It is used to close an if statement and indicates the end of the block of code that should be executed conditionally based on the evaluation of the if statement.

Generate the User Passwords

User passwords should be unique and secure. /dev/urandom ensures your passwords are both random and secure. It is a special file in Unix-like systems that provides a constant stream of high-quality random data, making it difficult to predict or replicate the generated passwords.

Paste the below configuration in the script:

generate_password() {
    tr -dc 'A-Za-z0-9!@#$%^&*()_+=-[]{}|;:<>,.?/~' </dev/urandom | head -c 16
}

generate_password() {: This defines the start of a function named generate_password.
tr -dc: This command is used to delete all characters from the input that are not in the specified set. tr stands for "translate" and is used to delete or replace characters, the -d option specifies that characters should be deleted, and the -c option complements the set of characters. This means that instead of deleting the characters specified in the set, it will delete all characters that are not in the set.
'A-Za-z0-9!@#$%^&*()_+=-[]{}|;:<>,.?/~': This is the set of characters allowed in the password, including uppercase letters (A-Z), lowercase letters (a-z), digits (0-9), and various special characters.
</dev/urandom |: /dev/urandom is a special file in Unix-like operating systems that provides random data. < is used to redirect the contents of /dev/urandom as input to the command on the left, and the file contents are passed through the pipe | to the next command.
head -c 16: The head command displays the first few lines of the file content that passes through the pipe, and the -c 16 option modifies head to output only the first 16 bytes of its input.

Process the Input File

The user_password.txt file that was passed in can now be used to carry out user management tasks. It will read the usernames and groups from the input file, create the users and their personal group, add them to their respective groups, set up their home directory, and generate and store their passwords. To execute these tasks efficiently, it's beneficial to keep them within a while loop so that each line of user data is processed sequentially, ensuring systematic user management operations.

To create the while loop, use:

# Read the input file line by line
while IFS=';' read -r username groups; do

while ... do: This starts a loop that continues to read lines from the input until there are no more lines to read.
IFS=';': This sets the internal field separator (IFS) to a semicolon. It tells the read command to use semicolons as the delimiter for splitting input lines into fields.
read -r: Reads the input line into the variables username and groups.
tr -d '[:space:]' removes all whitespace characters from the username and groups.

In this loop, there will be several iterations that will be carried out:

Iteration 1: Trim any leading/trailing whitespace

    # Trim any leading/trailing whitespace
    username=$(echo "$username" | tr -d '[:space:]')
    groups=$(echo "$groups" | tr -d '[:space:]')

username=$(echo "$username" | tr -d '[:space:]'): This line removes all whitespace characters from the beginning and end of the username value.
groups=$(echo "$groups" | tr -d '[:space:]'): This line does the exact same thing as the first line, but for the groups variable. It removes any leading or trailing whitespace from the list of groups associated with the user.

Logs the username and associated groups read from the input file to be certain the script is reading the user_password.txt file correctly:

    # Debug: Log the username and groups read from the file
    log_message "Read line: username='$username', groups='$groups'"

This is useful for troubleshooting.

Iteration 2: Check if usernames or groups are empty

    if [[ -z "$username" || -z "$groups" ]]; then
        log_message "Error: Username or groups missing in line: $username"
        continue
    fi

[[ -z "$username" || -z "$groups" ]]: [[ ... ]] is a conditional expression in Bash for testing. -z "$username" checks if the variable username is empty (has zero length). -z "$groups" checks if the variable groups is empty (has zero length). || is the logical OR operator, which means the condition is true if either $username or $groups (or both) are empty.
log_message "Error: Username or groups missing in line: $username": If either $username or $groups is empty, this command writes an error message to the $LOG_FILE.
continue: If the condition is true (i.e., either $username or $groups is empty), continue skips the rest of the current iteration of the loop. The script then moves on to the next iteration to process the next line from the input file.

Iteration 3: Check if the user already exists, otherwise create the user's personal group

    # Check if the user already exists
    if id "$username" &>/dev/null; then
        log_message "User $username already exists, skipping."
    else
        # Create the user's personal group
        if ! getent group "$username" >/dev/null; then
            groupadd "$username"
            log_message "Created group: $username"
        fi

if id "$username" &>/dev/null; then: This if statement checks if the user $username exists by querying the user database. &>/dev/null redirects both stdout (standard output) and stderr (standard error) to /dev/null, discarding any output. If the user exists, the condition is true (id command succeeds), and the script proceeds inside the if block.
log_message "User $username already exists, skipping.": If the user already exists (id command succeeds), this message is logged and the user creation process is skipped for this user.
else: If the user does not exist (the id command fails), the script proceeds with user and group creation.
if ! getent group "$username" >/dev/null; then: This command checks if a group with the username already exists and proceeds only if it doesn't.
groupadd "$username": Creates a new group with the same name as the username. This group will serve as the user's primary group, providing some basic permissions and ownership settings.
log_message "Created group: $username": Calls the log_message function to record the action of creating the group. The message passed to the function includes the name of the group that was created.

Iteration 4: Create the users with their personal group
To create users with their personal groups, add:

        # Create the user with the personal group
        useradd -m -g "$username" "$username"
        log_message "Created user: $username"

useradd -m -g "$username" "$username": This creates a new user account with the specified username, creates their home directory, and assigns the user to their personal group.
log_message "Created user: $username": Calls the logging function and logs the username of the newly created account.

Iteration 5: Assign passwords to users
For the users to have access, they should be assigned their own passwords each.

        # Generate a random password and set it for the user
        password=$(generate_password)
        echo "$username:$password" | chpasswd
        log_message "Set password for user: $username"

password=$(generate_password): Calls the generate_password function to generate a password, and stores the output within a variable named password.
echo "$username:$password" | chpasswd: This takes the generated password, formats it correctly, and passes it to the chpasswd command to set the password for the user.
log_message "Set password for user: $username": The function logs the action taken, indicating that the password has been set for the user.

Iteration 6: Add the user to additional groups

        IFS=',' read -ra group_array <<< "$groups"
        for group in "${group_array[@]}"; do
            if ! getent group "$group" &>/dev/null; then
                groupadd "$group"
                log_message "Created group: $group"
            fi
            usermod -aG "$group" "$username"
            log_message "Added user $username to group: $group"
        done

IFS=',' read -ra group_array <<< "$groups": IFS=',' sets the Internal Field Separator (IFS) to comma (,). This means that when the read command reads $groups, it will split it into multiple parts using comma as the delimiter. read -ra group_array <<< "$groups" reads the content of $groups into an array group_array, splitting it based on the comma delimiter (','), -r prevents backslashes from being interpreted as escape characters and -a group_array assigns the result to the array variable group_array.
for group in "${group_array[@]}"; do: Iterates over each element (group) in the group_array array.
if ! getent group "$group" &>/dev/null; then: getent group "$group" checks if the group $group exists in the system, ! negates the result, meaning if the group does not exist (! getent ...), the condition becomes true. &>/dev/null redirects both stdout and stderr to /dev/null, discarding any output. If the group does not exist, it proceeds with group creation.
groupadd "$group": Creates the group $group if it does not already exist.
log_message "Created group: $group": Logs a message indicating that the group $group was successfully created.
usermod -aG "$group" "$username": Adds the user $username to the group $group. -aG "$group": -a appends the user to the group without removing them from other groups -G specifies a list of supplementary groups.
log_message "Added user $username to group: $group": Logs a message indicating that the user $username was successfully added to the group $group.

Iteration 7: Create and set the home directory permissions
Users should have access to their individual home directories to perform certain actions. Add the below configuration in the script:

mkdir -p "/home/$username"
chown -R "$username:$username" "/home/$username"
chmod 755 "/home/$username"

mkdir -p "/home/$username": This creates the home directory for each new user, where $username is the username of the user being processed.
chown -R "$username:$username" "/home/$username": This changes the ownership of the user's home directory and all files and subdirectories within it.
chmod 755 "/home/$username: This sets the permissions for the newly created user's home directory. 755 means the owner ($username) has read, write, and execute permissions (rwx). Users in the same group as the owner and other users have read and execute permissions (r-x).

Iteration 8: Store the username and password securely

        # Store the username and password securely
        echo "$username,$password" >> $PASSWORD_FILE
        chmod 600 "$PASSWORD_FILE"
        log_message "Password for $username stored in $PASSWORD_FILE."
    fi

echo "$username:$password" >> "$PASSWORD_FILE": This line stores the username and its corresponding generated password in a file designated for storing passwords securely.
chmod 600 "$PASSWORD_FILE": This command restricts access to the password file to ensure it remains secure and confidential.
log_message "Password for $username stored in $PASSWORD_FILE.": Logs a message indicating that the password for $username has been stored in the password file ($PASSWORD_FILE).
fi: The fi keyword marks the end of the entire if...else conditional block, which started at iteration 3. It marks the end of the code block to execute if the condition is true.

End the Loop

End the loop using:

done < "$1"

The expression done < "$1" signifies the end of the while loop and instructs it to read input from the file specified by the INPUT_FILE=$1 variable.

The complete script is available in this GitHub repository.

Run the Script

To execute the Bash script without calling Bash in your terminal, make the script executable:

chmod +x create_users.sh

Once the script is executable, run it from the directory where it resides:

./create_users.sh user_passwords.txt

Verify the Script Executed Tasks Successfully

Several checks should be carried out to ensure that the script executed all the user management tasks successfully.

To verify user existence, run:

id <username>

To verify that the user password file was successfully created in the /var/secure/ directory, run:

cat /var/secure/user_passwords.txt

To verify group existence, run:

getent group <username>

To verify the log file as created and all actions are logged correctly without any errors or unexpected behaviors, run:

cat /var/log/user_management.log

To verify that each user has a personal group with the same name as their username, run:

getent passwd <username>
getent group <username>

Conclusion

This article highlights the automation of user management tasks using Bash scripts. It explores how scripting enhances efficiency in creating users, managing groups, setting permissions, and securing passwords on Linux systems.

Setting up a LAMP Stack application

Daniel Favour — Sun, 31 Mar 2024 03:44:09 +0000

The LAMP stack is a powerful combination of open-source technologies, enabling developers to build dynamic and interactive websites and web applications.

This article is a comprehensive guide to containerizing a LAMP (Linux, Apache, MySQL, PHP) stack application. It covers everything from setting up the application to running the containers using Docker commands and Docker Compose, along with monitoring the containers. To improve readability, it has been split up into different parts to form a series. This first part focuses on setting up the application on your local machine.

Application Architecture

Please check the below embed to view the image in higher resolution:

user-images.githubusercontent.com

In the above architecture diagram, users initiate an HTTP request by accessing the application through the browser using either "localhost" or the server's IP address. The server, with Apache installed, responds by serving the "form.html" file to users, prompting them to fill in their details, including their name, email, and description.

Upon completing the form, users submit the data back to the server. Apache then forwards the submitted data to a PHP script responsible for storing this information in the MySQL database. If the data is successfully stored, MySQL communicates this success to the PHP script, which responds with an HTML message displayed in the user's browser. On the other hand, if there is an issue while saving the data, the PHP script returns an error message to the user's browser, notifying them of the encountered problem.

This robust architecture ensures a seamless flow of data between users, Apache, PHP, and MySQL, providing a smooth user experience and reliable data management.

Prerequisites

Before we get started, ensure that you have the following in place:

Docker or Docker Desktop installed
Docker Compose (If you are using Docker Desktop, it comes with it already)
An IDE, VSCode recommended

Application Setup

For setup and testing purposes, the application will be deployed and tested on a Linux Ubuntu OS, utilizing VirtualBox and Vagrant within a macOS environment.

Project Structure

At the beginning of the build, the environment will resemble the structure below but overtime as we create more folders and files, the structure will change.

.
├── form_submit.php
├── form.html
├── install.sh
├── setup.sh
└── vagrantfile

Setup the Linux Environment

To setup the Linux Environment on mac, we will be utilizing Vagrant and Virtual Box. A script has been provided to automatically provision them.

NB: This step is applicable to mac users only. Linux users can proceed to the installation section.

Create an install.sh file in the root of the project and copy the below contents into it.

#/bin/sh

# Check if VirtualBox is installed
if ! command -v VBoxManage &> /dev/null; then
    echo "VirtualBox not found. Installing VirtualBox..."
    brew install --cask virtualbox
else
    echo "VirtualBox is already installed."
fi


# Check if Vagrant is installed
if ! command -v vagrant &> /dev/null; then
    echo "Vagrant not found. Installing Vagrant..."
    brew install --cask vagrant
    brew install --cask vagrant-manager
else
    echo "Vagrant is already installed."
fi

Give the script executable permission and run it with the following command:

chmod +x install.sh
./install.sh

It will automatically check if Vagrant and Virtual Box are already installed on your system, if they are installed outputs a message to let you know. However, if they are not found, the script will proceed to install both Vagrant and VirtualBox for you seamlessly.

Also in the root of the project folder, create a vagrantfile file and paste the below contents into it:

NUM_CONTROLLER_NODE = 1
IP_NTW = "192.168.56."
CONTROLLER_IP_START = 2
NODE_IP_START = 3

Vagrant.configure("2") do |config|
    config.vm.box = "ubuntu/bionic64"

    i = 0

    (1..NUM_CONTROLLER_NODE).each do [i] 
        config.vm.define "dockertask" do |node|
            node.vm.provider "virtualbox" do |vb|
                vb.name = "dockertask"
                vb.memory = 2048
                vb.cpus = 2
            end

            node.vm.hostname = "dockertask"
            node.vm.network "private_network", ip: IP_NTW + "#{CONTROLLER_IP_START + i}"
            node.vm.network "forwarded_port", guest: 22, host: "#{2710 + i}"
        end
    end
end

This vagrantfile configuration helps in automatically setting up the Linux VM, utilizing an ubuntu/bionic64 image for the setup. The VM has been given a name dockertask which you are free to modify. To provision the VM, run the following command

vagrant up --provision

Open the Virtual Box application to see the setup VM

NB: To seamlessly copy files from your local (macOs) to the Linux VM, you can use the below command:

vagrant scp </absolute-path-of-the-project-directory-to-the project-folder/> <name-of-vm>:/home/vagrant

It should look like this:

vagrant scp /Users/favour/Desktop/Modules/module-2/ dockertask:/home/vagrant

Note that the command should always be ran every time a change has been made on your local so that the Linux VM has the updated version.

Installation

With the VM now provisioned, the next step involves installing Apache, MySQL, and PHP, creating the essential LAMP stack foundation for the application. A script has also been provided to automate the installation process.

Create a setup.sh file at the root of the project folder, copy and paste the below contents into it:

#/bin/sh

# update the VM
sudo apt-get update -y

# Install apache web server and start it
sudo apt-get install apache2 -y
sudo service apache2 start

# Install mysql and go through the setup. Please remember your root password
sudo apt-get install mysql-server -y
sudo /usr/bin/mysql_secure_installation

# Install PHP
sudo apt-get install php -y
sudo apt-get install php-mysql -y
sudo apt-get install libapache2-mod-php -y
sudo apache2ctl -M
sudo a2dismod mpm_event
sudo a2enmod mpm_prefork
sudo a2enmod php7.2
sudo /etc/init.d/apache2 restart

What each command does:

sudo apt-get update -y: This updates the package lists and ensures that the package information is up-to-date before proceeding with any installations.
sudo apt-get install apache2 -y: This installs the Apache web server on the system. The -y flag allows the installation to proceed automatically without asking for user confirmation.
sudo service apache2 start: This command starts the Apache web server, so it becomes active and can serve web pages.
sudo apt-get install mysql-server -y: This installs the MySQL database server. The -y flag allows the installation to proceed automatically without asking for user confirmation.
sudo /usr/bin/mysql_secure_installation: This script guides you through a series of steps to set up MySQL securely. It prompts you to configure the root password, remove anonymous users, disable remote root login, and more.
sudo apt-get install php -y: This installs PHP, a server-side scripting language, on the system.
sudo apt-get install php-mysql -y: This installs the PHP MySQL extension, which allows PHP to communicate with a MySQL database. This extension is required if your PHP application needs to interact with a MySQL database.
sudo apt-get install libapache2-mod-php -y: This installs the PHP module for Apache web server (libapache2-mod-php) along with its dependencies. The -y flag allows the installation to proceed automatically without asking for user confirmation.
sudo apache2ctl -M: This command lists all the loaded Apache modules. It is used to check if the PHP module (mod_php) is successfully loaded after installation.
sudo a2dismod mpm_event: This disables the Apache event module (mpm_event) to switch to the prefork module, which is required for running PHP with Apache.
sudo a2enmod mpm_prefork: This enables the Apache prefork module (mpm_prefork), which is necessary to work with PHP.
sudo a2enmod php7.2: This enables the PHP module (mod_php) in Apache for PHP version 7.2. Replace 7.2 with the appropriate version if you are using a different PHP version.
sudo /etc/init.d/apache2 restart: This restarts the Apache web server to apply the changes made by enabling the PHP module and switching to the prefork module.

To execute the script, run the below command:

chmod +x setup.sh
./setup.sh

A prompt will come up asking you to pick a password level for mysql, it should look like this :

Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: y

It is best to give a password for security purposes, so input "y", then you will get the following prompt asking you to pick a password level:

LOW    Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary

Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 2

For securing the MySQL server, you have the option to select the password complexity level of your preference. If you choose the "LOW" length, which is set to "0", you could use a password like dockertask to meet the requirements.

It's important to note that selecting a more complex password, especially for production environments, is strongly recommended to enhance security. However, for local development or testing purposes, a simple password like dockertask can be used.

Enabling password validation during the MySQL setup will prompt the system to evaluate the strength of the root password you provided. The server will then present you with the password strength assessment. If you are satisfied with the current password, you can proceed by entering "Y" for "yes" at the prompt. This step ensures that you have reviewed the password strength and are content with the chosen password before proceeding with the setup.

It should resemble the below output:

Estimated strength of the password: 100 
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y

After completing the MySQL setup, the script will proceed with the remaining instructions. To ensure that PHP has been successfully installed, you can confirm it by running the following command:

php -v

Executing this command will display the installed PHP version along with relevant details, affirming that PHP is now operational and ready for use in your environment.

Now, open your web browser and enter the IP address of the server, which should be 192.168.56.2 or localhost depending on the environment you are working from. If everything is set up correctly, you should see the Apache web server running, and it will display the default Apache landing page or any other content that you might have configured. This confirms that Apache is up and running and successfully serving web pages.

To verify that you can access the MySQL console with the updated root user password, execute the following command:

sudo mysql -p

NB: The -p flag prompts you to enter the password you set during the MySQL setup. After providing the correct password, you should be able to log in to the MySQL console, where you can manage and interact with the MySQL database.

To exit the MySQL console, simply type:

exit

This will take you out of the MySQL console and return you to the regular command prompt.

Setup the Application

In the root of the project folder, create a form.html file:

touch form.html

Paste the below contents into it

<head>
    <title>
    Test Page
    </title>
    </head>
    <body>
    <form action="http://localhost/form_submit.php" class="alt" method="POST">
    <div class="row uniform">
    <div class="name">
    <input name="name" id="" placeholder="Name" type="text">
    </div>
    <div class="email">
    <input name="email" placeholder="Email" type="email">
    </div>
    <div class="message">
    <textarea name="message" placeholder="Message" rows="4"></textarea>
    </div>
    </div>
    <br/>
    <input class="alt" value="Submit" name="submit" type="submit">
    </form>
    </body>

In the same directory, create a form_submit.php file:

touch form_submit.php

Paste the below contents into it:'

<?php
$host = getenv('DB_HOST');
$db_name = getenv('MYSQL_DATABASE');
$username = getenv('DB_USER');
$password = getenv('MYSQL_PASSWORD');
$option = array(
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
  );

# Catch errors
try{
    $connection = new PDO("mysql:host=" . $host . ";dbname=" . $db_name, $username, $password, $option);
    $connection->exec("set names utf8");
} catch(PDOException $exception){
    echo "Connection error: " . $exception->getMessage();
}


function saveData($name, $email, $message){
    global $connection;
    $query = "INSERT INTO test(name, email, message) VALUES( :name, :email, :message)";

    try {
        $callToDb = $connection->prepare( $query );
        $name=htmlspecialchars(strip_tags($name));
        $email=htmlspecialchars(strip_tags($email));
        $message=htmlspecialchars(strip_tags($message));
        $callToDb->bindParam(":name",$name);
        $callToDb->bindParam(":email",$email);
        $callToDb->bindParam(":message",$message);

        if($callToDb->execute()){
            return '<h3 style="text-align:center;">Your information has been submitted to the database successfully!</h3>';
        }   else {
            return '<h3 style="text-align:center;">Failed to save data.</h3>';
        }
    } catch (PDOException $exception) {
        return '<h3 style="text-align:center;">Error: ' . $exception->getMessage() . '</h3>';
    }
}


if( isset($_POST['submit'])){
    $name = htmlentities($_POST['name']);
    $email = htmlentities($_POST['email']);
    $message = htmlentities($_POST['message']);

    //then you can use them in a PHP function. 
    $result = saveData($name, $email, $message);
    echo $result;
} else{
    echo '<h3 style="text-align:center;">A very detailed error message ( ͡° ͜ʖ ͡°)</h3>';
}
?>

Create a .env file which will contain environment variables for the PHP script to use in connecting to the database:

touch .env

Paste the below content into the file:

DB_HOST=mysql
MYSQL_DATABASE=dev_to
DB_USER=root
MYSQL_PASSWORD=dockertask

Creating a Virtual Host

Creating a Virtual Host in Apache provides numerous benefits, such as hosting multiple websites on a single server while maintaining separate configurations for each site. It conserves IP addresses, streamlines website management, and enhances security by isolating websites from one another.

While it is not mandatory to create a virtual host, doing so is highly beneficial, especially when dealing with multiple websites that need to be served using Apache. We will be creating a virtual host for learning purposes as this process helps you gain familiarity with Apache's directory configurations, enabling you to manage websites more effectively in the future.

In case you choose not to create a virtual host, you can proceed to work with the default directory located at /var/www/html. This directory serves as the default location for hosting websites on the Apache web server.

For better organization, we will create a directory called dockertask to house the project files, making it easier to manage and serve them.

To create the directory for the dockertask, run the following command:

sudo mkdir /var/www/dockertask

To ensure proper ownership and permissions, assign the directory dockertask to your current system user by running the below command:

sudo chown -R $USER:$USER /var/www/projectlamp
sudo chmod -R 755 /var/www/dockertask

This will allow you to work with the project files without encountering any permission issues.

Create and open a new configuration file, dockertask.conf, in Apache’s sites-available directory

sudo nano /etc/apache2/sites-available/dockertask.conf

This will create a new blank file. Paste in the following bare-bones configuration, then do a ctrl+x and type y for yes to save it:

<VirtualHost 192.168.56.2:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/dockertask
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

If you are working directly from a Linux OS, use the belo configuration:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/dockertask
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Set the ServerName directive globally, add the ServerName at the top or bottom of the file

sudo nano /etc/apache2/apache2.conf

On the first line of the apache2.conf file, add the below:

ServerName 192.168.56.2

Leave the ServerName as localhost if you are working directly with a Linux OS.

Setting ServerName globally in Apache means specifying a default server name that will be used for any virtual host that does not explicitly define its own ServerName.

In Apache HTTP Server, the ServerName directive is used to define the hostname and port number of the virtual host. A virtual host allows you to run multiple websites on the same physical server, and each virtual host can have its own ServerName that corresponds to a unique domain or hostname.

Use the ls command to confirm the dockertask.conf file exists in apache's sites-available directory

sudo ls /etc/apache2/sites-available

The output should resemble the below:

000-default.conf  default-ssl.conf  dockertask.conf

Enable the newly created virtual host:

sudo a2ensite dockertask

NB: We need to disable the default website that comes installed with Apache. This is required if you’re not using a custom domain name, because in this case Apache’s default configuration would overwrite your virtual host.

To disable Apache’s default website use a2dissite command , run the following command:

sudo a2dissite 000-default

To make sure the configuration file doesn’t contain syntax errors, run:

sudo apache2ctl configtest

The output should resemble the below:

Syntax OK

Finally, reload Apache so the changes take effect:

sudo systemctl reload apache2

The new website is now active, but the web root /var/www/dockertask is still empty.

Now we move the required folders, form.html and form_submit.php, to this directory from the home directory where the files exist

To move the files, run the below command:

sudo mv form.html /var/www/dockertask
sudo mv form_submit.php /var/www/dockertask

By default, Apache prioritises the index.html file over index.php, making it the landing page for the application. After maintenance, simply renaming or removing the index.html from the document root restores the regular application page.

To change this behavior, we need to edit the /etc/apache2/mods-enabled/dir.conf file and change the order in which the index.php file is listed within the DirectoryIndex directive:

sudo nano /etc/apache2/mods-enabled/dir.conf

Change the existing configuration to look like the below:

<IfModule mod_dir.c>
        #Change this:
        #DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
        #To this:
        DirectoryIndex form.html form_submit.php index.cgi index.pl index.xhtml index.htm
</IfModule>

After saving and closing the file, reload Apache for the changes to take effect:

sudo systemctl reload apache2

Setup Mysql database

Log into mysql to set up a database

sudo mysql -u root -p

Create a new MySQL user:

In the MySQL shell, run the following query to create a new user:

CREATE USER 'vagrant'@'192.168.56.2' IDENTIFIED BY 'Strongpassword@123';

NB: You can choose a different user and password for the above. Also remember to change the server address depending on the environment.

Grant privileges to the new user:

Next, grant the necessary privileges to the new user and flush the privileges to ensure the changes take effect immediately:

GRANT ALL PRIVILEGES ON dev_to.* TO 'vagrant'@'192.168.56.2';
FLUSH PRIVILEGES;

NB: Troubleshooting

If you run into access denied error for root user after providing your password, troubleshoot with this. It is a common issue with ubuntu/linux systems

Create a database using the below SQL commands:

create database dev_to;
use dev_to;
create table test(
    id int NOT NULL AUTO_INCREMENT,
    name varchar(255),
    email varchar(255),
    message text,
    PRIMARY KEY (id)
);

Confirm the database was created;

describe test;

You should see an output resembling the below

mysql> describe test;
+---------+--------------+------+-----+---------+----------------+
| Field   | Type         | Null | Key | Default | Extra          |
+---------+--------------+------+-----+---------+----------------+
| id      | int(11)      | NO   | PRI | NULL    | auto_increment |
| name    | varchar(255) | YES  |     | NULL    |                |
| email   | varchar(255) | YES  |     | NULL    |                |
| message | text         | YES  |     | NULL    |                |
+---------+--------------+------+-----+---------+----------------+

Exit the MySQL shell:

exit

Fix binding settings for mysql

Skip this step if you are working from a Linux OS with your ServerName as localhost.

By default, MySQL is bound to the local host (127.0.0.1). However, since the Vagrant server has its own unique IP address setup, we must configure MySQL to bind to this specific IP address.

Open the file for mysql conf

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

Search for bind-address and change it from 0.0.0.0 to 192.168.56.2, save and exit.
Restart the mysql server

sudo service mysql restart

Now, any information filled into the html form will automatically be saved in this database.

To run checks, log into the mysql database

sudo mysql -u root -p

Check for existing databases and identify the one we created earlier, dev_to. Once located, we can proceed to examine the tables residing under it, including the test table.

SHOW DATABASES;
USE dev_to;
SHOW TABLES;
describe test;
SELECT * FROM test;

Conclusion

In this article, we explored the process of setting up a fully functional LAMP stack application and learned how the components communicate with each other. In the next article, we will explore how to containerize the application using Docker.

Credits to Adnan Alam

Container Monitoring: Ensuring Application Performance and Health

Daniel Favour — Sun, 06 Aug 2023 15:24:52 +0000

In the previous article, we successfully containerized the LAMP stack application and conducted health checks on the containers. In this article, we will focus effectively monitoring these containers.

Container Monitoring is essential for maintaining the stability and efficiency of containerized applications. Monitoring containers allows you to gain insights into the resource usage, performance, and health of individual containers running within pods. It helps you identify potential issues with specific containers, track resource consumption, and troubleshoot application problems at a granular level.

To effectively monitor the containers, we will utilize tools like cAdvisor, Prometheus, MySQLd Exporter, and Grafana. We will look into them individually as we progress.

Environment Setup

For the setup, we could install them directly on the host OS. However, given that the article is container-focused, we will deploy them within containers. This approach simplifies the process and cleanup after the tasks have been completed.

Architecture Diagram

This is an overview of what the setup will look like and how each component communicates with the other:

Please check the below embed to view in higher resolution

user-images.githubusercontent.com

Project Structure

The folder structure for this article will look like the below:



.
├── .env
├── docker-compose.yml
├── install.sh
├── mysql
│   ├── db.sql
│   └── makefile
├── monitoring
│   ├── alertmanager
│   │   └── alert.yml
│   ├── docker-compose.yml
│   ├── grafana
│   │   └── grafana_db
│   ├── prometheus
│   │   ├── prometheus.yml
│   │   └── rules.yml
│   ├── .env
│   └── .my.cnf
├── php
│   ├── .env
│   ├── Dockerfile
│   ├── form.html
│   ├── form_submit.php
│   └── makefile
├── setup.sh
└── vagrantfile

The only thing new here is the monitoring directory, we already created the rest directories and files in the previous article.

cAdvisor

cAdvisor (Container Advisor) is an open-source monitoring and performance analysis tool specifically designed for containerized environments. It provides real-time insights into container resource usage and performance characteristics on a host system by collecting metrics like CPU, memory, network stats, and file system use at regular intervals for monitoring and analysis.

Our objective here is to set up cAdvisor to monitor and extract metrics from the containers. Subsequently, Prometheus will be configured to scrape these metrics, enabling them to be collected and processed. Prometheus utilizes the metrics it scrapes from cAdvisor to enable comprehensive monitoring and management of the containers. These metrics are stored as time series data, forming the foundation for analysis, visualization, and querying.

For cAdvisor to scrape metrics from containers, it needs access to the Docker data directory and Docker socket (which is essentially the Docker API) in order to effectively scrape metrics from containers and provide monitoring information. Since we will be running the containers with Docker compose, in our Docker Compose file, we will mount the relevant paths with container information inside the cAdvisor container.

If you choose not to use cAdvisor to collect metrics, you can use the Docker Daemon or Engine. If you're using Docker Desktop, go to your Docker Desktop settings, select "Docker Engine," and add the following configuration:



"metrics-addr" : "127.0.0.1:9323",

and change "experimental" from false to true. The final configuration should look like this:



{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true,
  "features": {
    "buildkit": true
  },
  "metrics-addr": "127.0.0.1:9323"
}

or like this



{
  "metrics-addr" : "127.0.0.1:9323",
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true,
  "features": {
    "buildkit": true
  }

Click on the Apply & Restart button.

NB: If you have docker engine installed on your system, go to /etc/docker/docker-daemon directory and do the same thing.

With this, the Docker Engine exposes the metrics at http://localhost:9323 for Prometheus to scrape.

To see the metrics from the Docker Daemon/Engine, visit http://localhost:9323/metrics:

Prometheus

Prometheus is an open-source systems monitoring and alerting tool. In the context of cAdvisor, Prometheus scrapes (pulls) the metrics collected by cAdvisor from the Docker Daemon about containers and stores them in its database as time series data. This allows the metrics to be easily queried, analysed, and visualised for monitoring and understanding container behaviour.

In the monitoring directory, create a prometheus folder and a prometheus.yml file inside it:



mkdir prometheus
cd prometheus
touch prometheus.yml

Paste the below contents into the prometheus.ymlfile: ```jsx

global:
scrape_interval: 5s
external_labels:
monitor: 'docker-container-monitor'

rule_files:

rules.yml

alerting:
alertmanagers:

static_configs:
- targets:
  - alertmanager:9093

scrape_configs:

job_name: 'prometheus'
static_configs:
- targets: ['prometheus:9090']
job_name: 'cadvisor'
scrape_interval: 5s
static_configs:
- targets: ['cadvisor:8080']
job_name: 'mysqld_exporter'
scrape_interval: 5s
static_configs:
- targets: ['mysqld_exporter:9104']
job_name: 'node_exporter'
scrape_interval: 5s
static_configs:
- targets: ['node-exporter:9100']

Let's break down what each section does:

- **`global:`** This section contains global configuration settings for Prometheus. The `scrape_interval` specifies the time interval at which Prometheus should scrape metrics (in this case, every 5 seconds). The `external_labels` section is used to add additional labels to all collected metrics. In this example, the label "monitor" with the value "docker-container-monitor" is added to all metrics.

- **`rule_files:`** This specifies the paths to one or more rule files. In this case, the file named "rules.yml" contains alerting rules that Prometheus will evaluate against the collected metrics. We will be looking at this soon.

- **`alerting:`** This section configures how Prometheus sends alerts. It specifies the target addresses of Alertmanager instances. We will also be looking at this soon.

- **`scrape_configs:`** This is the heart of the configuration, defining various jobs for scraping metrics. Each `job_name` corresponds to a specific type of metric source. Under each `job_name`, the `static_configs` section lists the targets (endpoints) from which Prometheus should scrape metrics.

- The first **`job_name`** is "prometheus," which targets the Prometheus instance itself. This self-monitoring capability ensures that Prometheus remains reliable and continues to provide accurate data for other monitoring and alerting tasks.

- The second **`job_name`** is "cadvisor," targeting `cAdvisor` for container-level metrics with a scrape interval of 5 seconds.

- The third **`job_name`** is "mysqld_exporter," targeting the MySQL Exporter for database-related metrics with a scrape interval of 5 seconds.

- The fourth **`job_name`** is "node_exporter," targeting the Node Exporter for host machine operating system and hardware-related metrics with a scrape interval of 5 seconds.

If you went the Docker Daemon path as shown previously rather than using cAdvisor, the below configuration will need to be added inside the `prometheus.yml` file so that Prometheus has the endpoint where it needs to scrape the metrics:
```jsx


  - job_name: docker
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['host.docker.internal:9323']

where;

job_name: docker: This is the name you're giving to the job. It's a label that helps you identify this particular job in Prometheus.
scrape_interval: 5s: This specifies the interval at which Prometheus will scrape (collect) metrics from the specified target(s). In this case, it's set to 5 seconds.
metrics_path: /metrics: This is the path at which Prometheus will request metrics from the target(s). In many applications and services, a common convention is to expose metrics at the "/metrics" endpoint.
static_configs: This section defines a list of static targets that Prometheus will scrape. Each target is specified as a dictionary with its own configuration options.
targets: ['host.docker.internal:9323']: This is the target you're configuring Prometheus to scrape metrics from. In this case, the target is host.docker.internal (which is a special DNS name used to refer to the host machine from within Docker containers) and the port 9323.

Environment Variables

In the monitoring directory, create a .env file which will contain environment variables for the Grafana container:



touch .env

Paste the below contents into the . env file: ```

GF_SECURITY_ADMIN_USER=
GF_SECURITY_ADMIN_PASSWORD=

When you run Grafana for the first time, the default username and password is `admin` but for security reasons, we need to override that and set our own username and password. Be sure to give and username and password in the above.

Now we will create a configuration file that the MySQLd Exporter will use in accessing the MySQL server.

- In the same monitoring directory, create a `.my.cnf` file:
```jsx


touch .my.cnf

Paste the below into it: ```

[client]
user = root
password = Strongpassword@123
host = mysql

If you recall, we used these variables when creating the MySQL server container. If you changed your variables, ensure to replace the above configuration file with the appropriate ones.

## Alerting and Notification: 
We will configure alerts to notify us in case of unusual behaviour or performance decline within the containers. Alerts can be established either through Prometheus or within Grafana. For now, our emphasis will be on configuring alerts in Prometheus. 
Additionally, there are various methods to configure alerts, including email, Slack, Teams, etc. You can learn more about them [here](https://prometheus.io/docs/alerting/latest/configuration/).
For the scope of this article, we will configure alerting for Slack.

### Setup Slack Channel
To setup alerting for Slack, we will need a Slack channel in a workspace that we have admin access to. You can learn how to create a new Slack workspace [here](https://slack.com/help/articles/206845317-Create-a-Slack-workspace).

If you already have a workspace, you can proceed to creating a new channel in the workspace.

- Create a channel in the workspace and give it a name like `alertmanager` or any name of your choice. Click on the channel name at the top.

![New Channel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ki74x8ltdbi30g6w01wt.png)

- You will be presented with the below screen. Select `Integrations` and under Apps, click on the `Add an App` button.
![Add an App](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6e8bahzphprir1rfhrrf.png)

- Search for `Incoming Webhooks` and click the `install` button.

![Incoming Webhooks](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sbmio0ettudkpkyc3nd8.png)

- You will be redirected to your browser, click the `Add to Slack` button.
![Add to Slack](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/692kzk2v1ucnu3w44c0c.png)

- Search for the channel you want the Incoming Webhook app to be added to and select it.

![Add Webhook to channel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r6shvi2xvshr1kzkdao4.png)

- You will be presented with the settings for it, scroll to the bottom and click the `Save Settings` button.

- Scroll down again and copy the Webhook URL already generated for you, you can also choose to customize the icon for it.

![Webhook URL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9s9yiqhn4zfkfzcr104w.png)

- Return back to Slack and you should see a message showing that an integration has been added to the channel.

![Webhook Added to Channel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/46s4o2x7n1tohnuwy2go.png)

Now, the Slack channel has been prepped to receive alerts.

### Alertmanager
In production environments, downtimes occur frequently or less, and ensuring that you know about it before users do is important, hence why we will be setting up alerting for our containers using Alertmanager. 

[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is an open-source component that works in conjunction with Prometheus for managing and sending alerts in a more sophisticated and organized manner. It is responsible for handling alerts generated by Prometheus and other sources, routing them to appropriate receivers, and deduplicating or grouping them as needed. It ensures that alerts are delivered reliably to the right individuals or systems for timely response.

We will be configuring Alertmanager to send alerts generated by Prometheus to the Slack channel we just created.

- Inside the monitoring directory, create a folder called `alertmanager` and create file called `alertmanager.yml` inside it:
```jsx


mkdir alertmanager
cd alertmanger
touch alertmanager.yml

Paste the below configuration into the alertmanager.yml file:



route:
  group_by: [cluster]
  receiver: alert-test
  group_interval: 30s
  repeat_interval: 30s
  routes:
    - match:
        severity: slack
        receiver: alert-test

receivers:
- name: alert-test
  slack_configs:
  - api_url: <insert the Webhook URL copied earlier>
    channel: '#alertmanager'
    icon_url: https://avatars3.githubusercontent.com/u/3380462
    send_resolved: true
    text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

Break down of the configuration:

Route Configuration:

group_by: [cluster]: This indicates that alerts should be grouped based on the "cluster" label. Alerts with the same cluster label value will be grouped together.
receiver: alert-test: This specifies the default receiver for alerts. When an alert is triggered, it will be sent to the "alert-test" receiver.
group_interval: 30s: This sets the interval for grouping alerts with the same label. In this case, alerts with the same cluster label value will be grouped every 30 seconds.
repeat_interval: 30s: This sets the interval at which alerts are repeated. If an alert is still active, it will be sent again every 30 seconds.
routes: This section defines routes based on specific conditions.
match: This indicates that a route should be created when specific conditions are met. severity: slack: This specifies that the route applies to alerts with the "slack" severity label.
receiver: alert-test: This route uses the "alert-test" receiver to send alerts matching the specified conditions.

Receiver Configuration:

name: alert-test: This defines the name of the receiver, which is used to identify it in the route and alerts.
slack_configs: This section configures the Slack notification settings for this receiver.
api_url: This is the Slack incoming webhook URL used to send notifications to Slack. Remember to paste the Webhook URL previously copied from the Webhook integration for the alertmanager slack channel.
channel: This specifies the Slack channel where notifications will be sent, which in this case is the alertmanager channel.
icon_url: This URL points to an icon image for the Slack notification.
send_resolved: true: This setting ensures that resolved alerts (alerts that return to a healthy state) are also sent as notifications.
text: This is the content of the notification message. It uses template variables to include the alert's summary and description.

Return back to the Prometheus directory

Create a rules.yml file in the Prometheus directory and paste in the below configuration:

These rules define alerts that monitor the health and performance of containers and MySQL databases.

For the containers, the alerts include detecting when a Cadvisor container is down, when a container has been killed, is absent, experiences high memory usage, high throttle rate, low CPU utilization, or low memory usage.

For the the MySQL database, the alerts encompass scenarios where MySQL is down, there are too many connections (>80%), high threads running (>60%), slow queries, InnoDB log waits, or when MySQL has been restarted recently.

Docker Compose

For container orchestration, automation and simplicity, we will use a Docker Compose file to run all the containers.

In the root of the project folder, create a new directory called monitoring and in that directory create a docker compose file: ```jsx

mkdir monitoring
cd monitoring
touch docker-compose.yml

- Paste the below contents into the `docker-compose.yml` file:


  


  
  
  Run the Containers


To run the containers defined in the Docker Compose file, run the below command from inside the monitoring directory:



docker compose up -d


</code></pre></div>
<p>This command automates the process of checking for the images specified in the file. If these images are already present on the system, the command proceeds to create and launch the corresponding containers. However, if the images are not available, it fetches them before initiating container creation. The command then ensures that these containers are executed in the background, freeing up the terminal for other tasks.</p>
<h2>
  <a name="monitoring" href="#monitoring">
  </a>
  Monitoring
</h2>

<p>Now that the containers are up and running, we can monitor them and get insights.</p>
<h3>
  <a name="alerts-in-slack" href="#alerts-in-slack">
  </a>
  Alerts in Slack
</h3>

<p>Immediately the containers are brought up with Docker Compose, you should receive an alert in your Slack channel.</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4e8gmkch88lyqzwy3f9j.png" alt="Initial Alert"></p>

<p>The alert mentioned above originates from the MySQLd Exporter and serves to notify you when the MySQL server is either inactive or not functioning. In the previous article, we ran the MySQL container in conjunction with the PHP and phpMyAdmin containers. If the containers are still running, the alert won't trigger. In a situation that all three containers were stopped, the MySQLd Exporter identifies the absence of the MySQL container which runs the MySQL server and, noticing that it's non-operational, sends an alert.</p>

<p>The alert will continue to be sent at intervals to the channel until the issue has been resolved depending on the duration set. </p>

<p>If you take a close look at the rule alert</p>
<div class="highlight"><pre class="highlight jsx"><code>

 <span class="o">-</span> <span class="nx">alert</span><span class="p">:</span> <span class="nx">ContainerKilled</span>
  <span class="nx">expr</span><span class="p">:</span> <span class="dl">'</span><span class="s1">time() - container_last_seen &gt; 60</span><span class="dl">'</span>
  <span class="k">for</span><span class="p">:</span> <span class="mi">0</span><span class="nx">m</span>
  <span class="nx">labels</span><span class="p">:</span>
    <span class="nx">severity</span><span class="p">:</span> <span class="nx">warning</span>
  <span class="nx">annotations</span><span class="p">:</span>
    <span class="nx">summary</span><span class="p">:</span> <span class="nx">Container</span> <span class="nf">killed </span><span class="p">(</span><span class="nx">instance</span> <span class="p">{{</span> <span class="nx">$labels</span><span class="p">.</span><span class="nx">instance</span> <span class="p">}})</span>
    <span class="nx">description</span><span class="p">:</span> <span class="dl">"</span><span class="s2">A container has disappeared</span><span class="se">\n</span><span class="s2">  VALUE = {{ $value }}</span><span class="se">\n</span><span class="s2">  LABELS = {{ $labels }}</span><span class="dl">"</span>


</code></pre></div>
<p>The <code>for: 0m</code> indicates that the alert condition is evaluated at the current moment (time()), and if it's true, the alert will be immediately fired. In other words, the alert doesn't require the condition to persist for a specific duration before being triggered; it will fire as soon as the condition is met.</p>

<p>In this specific case, the condition <code>time() - container_last_seen &gt; 60</code> checks if the time elapsed since the last time the container was seen is greater than 60 seconds. If this condition becomes true at any given moment, the ContainerKilled alert will be fired immediately.</p>

<p>Now, when we bring the MySQL container back up, we will get an alert stating that the issue has been resolved:<br>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5z2ojpxzi008uuzvyh73.png" alt="Resolved"></p>
<h3>
  <a name="cadvisor" href="#cadvisor">
  </a>
  cAdvisor
</h3>

<ul>
<li>To access the cAdvisor container, visit <code>http:localhost:8080</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lobzsj0139xgxwl6yuei.png" alt="cadvisor"></p>

<ul>
<li>To see the metrics it is gathering about your containers from the Docker Daemon, click on <code>Docker Containers</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bmy6eku9xiw3d0eoylm2.png" alt="cadvisor docker container"></p>
<h3>
  <a name="prometheus" href="#prometheus">
  </a>
  Prometheus
</h3>

<ul>
<li>To access the Prometheus GUI, visit <code>http://localhost:9090</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p4a2z7l9k36qg48j0zkq.png" alt="Prometheus"></p>

<ul>
<li>Click on <code>Status</code> as shown in the above image, and select <code>Targets</code> to see the endpoints or data sources that Prometheus is scraping (retrieving) metrics from:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nyxqtvmb87a34552gy2c.png" alt="Targets"></p>

<p>This shows that all the endpoints are active and Prometheus can reach them</p>

<ul>
<li>To see the rules we defined in the Prometheus directly, click on <code>Status</code> again and select <code>Rules</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0u9ydi1p82wssh850oiu.png" alt="Rules"></p>

<p>All the rules defined are shown here.</p>

<ul>
<li>To see the Alerts, Click on <code>Alerts</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y3amhrglmceof8pyzvzt.png" alt="Alerts"></p>

<p>From the above we can see all the alerts from the <code>rules.yml</code> file we configured earlier. We can also see the "inactive," "pending," and "firing" status bars, which are used to describe the states of alerts and how they are managed within the alerting system.</p>

<ul>
<li><p><strong><code>Inactive:</code></strong> An alert is in the "inactive" state when the conditions specified in its alerting rule are not currently met. In other words, the metric or metrics being monitored have not crossed the defined threshold or condition to trigger the alert. Inactive alerts do not generate any notifications or actions.</p></li>
<li><p><strong><code>Pending:</code></strong> When an alert transitions from "inactive" to "firing," it goes through a "pending" state for a duration defined in the alerting rule. The "pending" state helps prevent alerts from rapidly toggling between "inactive" and "firing" due to minor fluctuations in metric values. During the "pending" state, no additional notifications are sent, and the alert is not escalated. If the condition persists for the defined duration, the alert transitions to the "firing" state.</p></li>
<li><p><strong><code>Firing:</code></strong> An alert is in the "firing" state when the conditions specified in its alerting rule are met and sustained over the duration of the "firing" state. This is the state where the actual alert notifications are generated and sent to configured receivers, such as email addresses or external services. The alert will remain in the "firing" state as long as the condition persists. Once the condition is resolved and the metric values no longer trigger the alert, it will transition back to the "inactive" state.</p></li>
</ul>

<p>From the above image, we have 17 alerts in a Pending state due to <code>ContainerLowCpuUtilization</code> and <code>ContainerLowMemoryUsage</code>.</p>

<p>If we look into the <code>ContainerLowCpuUtilization</code>, we can identify the containers with low CPU utilization:</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hii11drp8wwav77wruzw.png" alt="low CPU utilization"></p>

<p>This above image reveals the container names exhibiting minimal CPU usage. The alert criteria was set to identify containers with CPU utilization below 20% over a span of one week. As the duration has not yet reached a full week, all containers are currently indicated as having CPU usage below the 20% threshold.</p>
<h3>
  <a name="node-exporter" href="#node-exporter">
  </a>
  Node exporter
</h3>

<p>A Node Exporter is a utility used in conjunction with the Prometheus monitoring system to collect and expose metrics from an operating system's hardware and system resources. It allows you to monitor various aspects of the host machine's performance, resource utilization, and health.</p>

<p>To see host OS level metrics, access the node exporter at <code>http://localhost:9100</code> and Click on the <code>Metrics</code> link displayed:</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ikznlff3iafqajjs5pel.png" alt="Node Exporter"></p>

<p><strong>NB:</strong> Our focus is not on the host OS metrics but on containers, we are only using the node exporter here to get better insights on monitoring and scraping with Prometheus.</p>
<h3>
  <a name="alertmanager" href="#alertmanager">
  </a>
  Alertmanager
</h3>

<ul>
<li>The alert manager can be accessed at <code>http://localhost:9093</code>:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3bfdvzkmssfucwx30mqq.png" alt="Alertmanager"></p>

<p>All alerts rules that moved from the <code>inactive</code> state to the <code>firing</code> state will appear here and the Alertmanager will send a alert to the Slack channel depending on the duration specified for that alert rule.</p>
<h3>
  <a name="mysqld-exporter" href="#mysqld-exporter">
  </a>
  MySQLd Exporter
</h3>

<ul>
<li>The <a href="https://github.com/prometheus/mysqld_exporter">MySQL Exporter</a> is a tool used in conjunction with the Prometheus monitoring system to collect and expose metrics from MySQL database servers. It allows you to monitor various aspects of your MySQL database's performance and health, enabling you to detect potential issues, track trends, and set up alerts based on the collected metrics.</li>
</ul>

<p>The MySQLd Exporter can be accessed at <code>http://localhost:9104</code>:</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qmp8djz23ep3vkknbey4.png" alt="MySQLd Exporter"></p>
<h2>
  <a name="visualisation" href="#visualisation">
  </a>
  Visualisation
</h2>

<p>To gain deeper insights and enhance our container monitoring, we will be leveraging Grafana, a visualisation tool.</p>
<h3>
  <a name="grafana" href="#grafana">
  </a>
  Grafana
</h3>

<p><a href="http://grafana.com/oss/grafana/">Grafana</a> is an open-source platform used for monitoring, visualisation, and analysis of time-series data. It enables users to create interactive and customisable dashboards that display metrics and data from various sources, making it easier to understand and interpret complex data trends and patterns.  </p>

<p>We will be integrating the data and metrics generated from Prometheus into Grafana so that we can visualise it better.</p>

<p>The Grafana container has already been created and started, in your browser, visit <code>http:localhost:3000</code> to login to Grafana.</p>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fpr500dw0rf9nwb2sjh1.png" alt="Grafana Login"></p>

<ul>
<li>Enter the admin user and password that was passed into the container as environment variables and you should be able to login.</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l82lmbdnz9iwl166nz9u.png" alt="Grafana Logged In"></p>

<ul>
<li>Click on "Add your first data source" and select "Prometheus"</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/30bheoutop4loigyh5c8.png" alt="Add data source"></p>

<ul>
<li>Give it a name or you can stick with the default name, fill in the Prometheus URL which will be <code>http://prometheus:9090</code>. To be sure of the URL, you can check back the targets in Prometheus and the endpoint for Prometheus, that will be the URL to fill in</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/si4rsrdoc44tpjez9kqf.png" alt="Prometheus"></p>

<ul>
<li>Scroll down to the bottom and click on the "Save and Test" button, you should get a successfully queries message, if not, check the URL again for mistakes.</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kd2262w3dcq09bkuha51.png" alt="Endpoint"></p>

<ul>
<li>Now go back to "Home" by clicking on the hamburger icon, then click on "Dashboards"</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jqds7w6ejuhfz8p7y01o.png" alt="Home"></p>

<ul>
<li><p>Click on the "New" button and select "Import"<br>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nssfms497kdnjfnlnbxc.png" alt="Dashboard"></p></li>
<li><p>The first dashboard we will be importing will be for the MySQLd exporter, it's an open source dashboard which can you get more details about <a href="https://grafana.com/grafana/dashboards/14057-mysql/">here</a>. From the link, copy the ID number which is <code>14057</code> and paste into the <code>import</code> box and click on the <code>load</code> button:</p></li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ek5ubydcy7958ze8y8is.png" alt="Import"></p>

<ul>
<li>The MySQLd dashboard will be loaded with metrics from the MySQL database for visualisation:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/orgs4itvnw7gik6d7gzi.png" alt="dashboard"></p>

<ul>
<li>Because the containers haven't been up for way too long, you might not see a lot of metrics displayed. We will can change the time range to <code>2m</code> to see all that has happened under 2 minutes:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u4hddsk18j26961ncyyv.png" alt="time range"></p>

<ul>
<li>After the time range has been reduced to 2 minutes, you should be able to see some metrics displayed in graphs, etc:</li>
</ul>

<p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94pt41btscvqg5f6fmlt.png" alt="Metrics"></p>

<p>With that set, you can visualise your database metrics with the prebuilt dashboard just set up.</p>
<h2>
  <a name="conclusion" href="#conclusion">
  </a>
  Conclusion
</h2>

<p>Throughout this comprehensive three-part article series, we covered a comprehensive setup of a LAMP stack application, containerization of the application, and effectively monitoring of the application.</p>

Containerizing a LAMP Stack Application

Daniel Favour — Sun, 06 Aug 2023 15:24:23 +0000

In the previous article, we learned how to set up a LAMP stack application. In this article, we will be looking at containerizing the LAMP stack application.

Containerization of an application involves packaging it along with its dependencies, configurations, and runtime environment into a Docker container. This encapsulation ensures that the application can run consistently across different environments and eliminates potential compatibility issues.

For this process, our primary focus will be on the PHP script. Initially, we'll create a Dockerfile for the PHP application, allowing us to build the Docker image and subsequently run the container. To establish a database connection, we'll attach a MySQL container. Additionally, we'll introduce an extra container, phpMyAdmin, which will serve as a user-friendly GUI for accessing and managing the database.

Container Architecture diagram

Please check the below embed to view the image in higher resolution:

user-images.githubusercontent.com

The diagram above illustrates the container architecture utilizing Docker Compose. The applications are executed with Docker Compose and are encapsulated within a Docker bridge network, facilitating smooth communication among them. The host OS provides a Docker Engine, making Docker functionality available for seamless container management and orchestration.

Container Communication Diagram

Please check the below embed to view the image in higher resolution:

user-images.githubusercontent.com

Approach to Running the Containers

We will be employing 2 different methods to run the containers, without Docker Compose and with Docker Compose.

In the "without Docker Compose" approach, we will rely on Makefiles to automate the container building and running processes. The Makefiles will handle tasks like pulling the required Docker images, creating and configuring containers, setting up networking, and other necessary actions. This method allows us to manage the containerization workflow efficiently, automating key steps with concise and easily maintainable scripts.

On the other hand, the "with Docker Compose" approach involves utilizing Docker Compose, a powerful tool for defining and managing multi-container applications. With Docker Compose, we can specify the services, networks, volumes, and configurations required for our application within a single YAML file. This streamlines the entire process, making it easier to deploy and manage multiple containers in a cohesive manner.

Project Structure

The project structure should be identical to the below:

.
├── .env
├── docker-compose.yml
├── install.sh
├── mysql
│   ├── db.sql
│   └── makefile
├── php
│   ├── .env
│   ├── Dockerfile
│   ├── form.html
│   ├── form_submit.php
│   └── makefile
├── setup.sh
└── vagrantfile

Without Docker Compose (Makefile Approach)

A Makefile is a file used with the Unix utility "make" to automate the build process of software projects. It contains instructions, typically written in shell commands, and is named "makefile" or "Makefile" depending on the system. Each rule in the Makefile has a target, dependencies, and commands. When you run the "make" command, it reads the Makefile and executes the specified commands to build the software according to the defined rules.

In the context of containerizing our application, a Makefile serves as a valuable automation tool. It streamlines the entire containerization process by automating the necessary tasks. The Makefile is designed to install essential tools, lint both the Dockerfile and PHP application code, build the Docker image, and run the container image.

The primary objective behind using a Makefile is to achieve seamless automation, making the development workflow more efficient and consistent.

Create PHP directory

In the root of the project folder, create a php directory and move the form_submit.php and form.html files into it:

mkdir php
mv form_submit.php /php
mv form.html /php

Environment Variables

We will create environment variables in the php folder. These variables will be used for the form_submit.php.

Create a .env file:

touch .env

Paste the below into the file:

MYSQL_PASSWORD=Strongpassword@123
DB_HOST=mysql
MYSQL_DATABASE=dev_to
DB_USER=root

Remember to change the values and use values of your choice but be consistent with it across the environment. These variables are what php will use to communicate with the MySQL server for data storage.

Write the Dockerfile

To containerize the PHP application, we need to create a Dockerfile. In the php directory, create a new file called Dockerfile without any file extension. Dockerfiles don't require extensions.

Now, paste the following content into the Dockerfile:

FROM php:7.4-apache

WORKDIR /var/www/html

RUN docker-php-ext-install mysqli pdo pdo_mysql && docker-php-ext-enable mysqli

COPY form.html /var/www/html/index.html
COPY form_submit.php /var/www/html

RUN chown -R www-data:www-data /var/www/html \
    && a2enmod rewrite

EXPOSE 80

Breakdown of what each line means:

FROM php:7.4-apache: This line sets the base image for our container. It uses the official PHP image with Apache server, version 7.4. This base image already includes PHP and Apache, making it convenient for hosting PHP applications.
WORKDIR /var/www/html: The WORKDIR /var/www/html instruction sets the working directory inside the container to /var/www/html. It provides a context for file operations, and subsequent commands like COPY or RUN will be executed relative to this directory. In this case, it is set to the common location for web application files in Apache servers, and the following COPY commands copy the PHP application files into this directory inside the container for use by the Apache web server.
RUN docker-php-ext-install mysqli pdo pdo_mysql && docker-php-ext-enable mysqli: This line uses the RUN instruction to execute commands during the Docker image build process. Here, we are installing PHP extensions mysqli, pdo, and pdo_mysql required for database connections. The docker-php-ext-enable command is used to enable the mysqli extension.
COPY form.html /var/www/html/index.html: The COPY instruction copies the form.html file from the host (your local machine) to the container's /var/www/html directory. In this case, it is renamed to index.html, serving as the default HTML page when accessing the root URL.
COPY form_submit.php /var/www/html: Similarly, this line copies the form_submit.php file from the host to the container's /var/www/html directory. This PHP file handles the form submissions from the form.html page.
RUN chown -R www-data:www-data /var/www/html && a2enmod rewrite: Here, we use the RUN instruction to set the ownership of the /var/www/html directory to the www-data user and group. Apache typically runs under the www-data user, so this ensures proper permissions for serving the website content.

The second part of this line uses the a2enmod command to enable the Apache module rewrite. The rewrite module is needed to allow URL rewriting, enabling cleaner URLs and better routing for the PHP application.

EXPOSE 80: The EXPOSE instruction is a metadata declaration that indicates which network ports the container will listen on during runtime. In this case, we specify that the container will listen on port 80. However, this does not actually publish the port to the host machine; it only serves as documentation for the user.

Create the Environment Variables File

In the php folder directory, create a file called .env, it will be used to store environment variables for the containers:

touch .env

Copy the below contents into it, ensure to fill it up with your desired values but leave the DB_Host as mysql and the DB_USER as root:

MYSQL_PASSWORD=
DB_HOST=mysql
MYSQL_DATABASE=
DB_USER=root

Write the Makefile

To create the Makefile file, run the following command:

touch Makefile

Paste the below configuration into the makefile:

# Makefile for PHP Dockerfile and PHP Code

.SILENT:
install:
    # Check if Homebrew is installed otherwise install it
    @if ! command -v brew &> /dev/null; then \
        echo "Installing Homebrew..."; \
        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"; \
        echo "Homebrew installed!"; \
    else \
        echo "Homebrew is already installed."; \
    fi

    # Check if Hadolint is installed otherwise install it
    @if ! command -v hadolint &> /dev/null; then \
        echo "Installing Hadolint..."; \
        brew install hadolint; \
        echo "Hadolint installed!"; \
    else \
        echo "Hadolint is already installed."; \
    fi

    # Check if wget is installed otherwise install it
    @if ! command -v wget &> /dev/null; then \
        echo "Installing wget..."; \
        brew install wget; \
        echo "wget installed!"; \
    else \
        echo "wget is already installed."; \
    fi

    # Check if php is installed otherwise install it
    @if ! command -v php &> /dev/null; then \
        echo "Installing PHP..."; \
        brew install php; \
        echo "PHP installed!"; \
    else \
        echo "PHP is already installed."; \
    fi

    # Install Xcode Developer Tools but check if they are already installed first
    @if ! xcode-select -p > /dev/null 2>&1; then \
        echo "Installing Xcode Developer Tools..."; \
        xcode-select --install; \
    else \
        echo "Xcode Developer Tools are already installed."; \
    fi

.SILENT:
lint_dockerfile:
    # Lint the Dockerfile using hadolint
    # See local hadolint install instructions: https://github.com/hadolint/hadolint
    hadolint Dockerfile

    # Print a successful message
    @echo "Dockerfile linting completed successfully. No errors found, Dockerfile follows best practices."



.SILENT:    
lint_php:
    set -x  # Enable verbose output
    # Check if PHPCS and PHPCBF linter files are downloaded, otherwise download them
    @if [ ! -f "phpcs.phar" ]; then \
        echo "Downloading phpcs.phar..."; \
        wget -q https://squizlabs.github.io/PHP_CodeSniffer/phpcs.phar; \
        echo "phpcs.phar downloaded!"; \
    else \
        echo "phpcs.phar is already downloaded."; \
    fi

    # For PHPCBF
    @if [ ! -f "phpcbf.phar" ]; then \
        echo "Downloading phpcbf.phar..."; \
        wget -q https://squizlabs.github.io/PHP_CodeSniffer/phpcbf.phar; \
        echo "phpcbf.phar downloaded!"; \
    else \
        echo "phpcbf.phar is already downloaded."; \
    fi

    # Make the downloaded PHAR files executable
    chmod +x phpcs.phar
    chmod +x phpcbf.phar

    # Lint the php code and check for any errors
    php phpcs.phar --standard=PSR12 form_submit.php || true

    # Continue with other targets by recursively invoking `make`
    $(MAKE) build run


build:
    # Build the docker image using the Dockerfile in this directory and create a network
    docker build -t php-test .

    # Create docker network but check if the network exists first
    if ! docker network inspect test_network > /dev/null 2>&1; then \
        docker network create test_network; \
    fi


run:
    # for php
    docker run -d --name php --network test -p 80:80 --env-file .env php-test:latest

    # for phpmyadmin
    docker run -d --name phpmyadmin --network test -p 8000:80 -e PMA_ARBITRARY=1 -e PMA_HOST=mysql phpmyadmin


# Target to run the entire setup process
all: install lint_dockerfile lint_php build run

Let's breakdown the configurations:

.SILENT: In a Makefile, the .SILENT special target is used to suppress the normal echoing of commands that are executed during the build process. When you include .SILENT in your Makefile, it tells Make to operate in silent mode for the rules that follow it.

By default, when you run make, it displays each command it executes in the terminal. This can be helpful for understanding what's happening during the build process, but it can also lead to a lot of noise, especially for simple and repetitive commands.

install: This target automates the installation of essential tools and dependencies, including Homebrew, Hadolint, wget, php, and Xcode Developer Tools. The @ symbol before each command suppresses the output of the command, providing a cleaner output during the installation process.

For each tool, the Makefile checks if it is already installed using command -v followed by the tool's name. If the tool is not found, it proceeds with the installation by fetching the necessary installation script or using Homebrew to install the tool. This automation ensures a smooth and efficient setup of the development environment, allowing developers to focus on the project without the hassle of manual tool installations.

For example, when you run the command make install, it should return the below output if the tools are already installed, if otherwise, it will proceed to install them:

User-demo:php daniel$ make install
Homebrew is already installed.
Hadolint is already installed.
wget is already installed.
PHP is already installed.
Xcode Developer Tools are already installed.

lint_dockerfile: This target employs Hadolint, a Dockerfile linter, to enforce best practices for writing Dockerfiles. When the make lint_dockerfile command is executed, it automatically lints the Dockerfile in the current directory, ensuring it adheres to industry-standard guidelines.

The purpose of this target is to provide developers with a quick and automated way to validate their Dockerfiles. If the Dockerfile is free from issues or errors, Hadolint does not produce any output, potentially leading to confusion. To address this, the target includes a specified echo message to indicate that the Dockerfile has passed the linting process successfully.
In case the Dockerfile contains issues or errors, the target will display the relevant error output in the terminal as specified by Hadolint, providing developers with valuable feedback to rectify any non-compliant code.

lint_php: This target is responsible for linting the PHP code using PHP_CodeSniffer (PHPCS). This target serves as a linter for PHP code, ensuring it adheres to PHP coding standards, particularly the PSR-12 standard. To provide more insight into the execution process, the set -x command enables verbose output, displaying the actual commands executed in the terminal during the target's execution.

The target begins by checking if the phpcs.phar and phpcbf.phar files exist in the current directory. If not, it proceeds to download them using the wget command from their official URLs. The chmod +x command is then used to make the downloaded PHAR files (phpcs.phar and phpcbf.phar) executable, allowing them to be run as commands.

Next, the linter executes the phpcs.phar command with the --standard=PSR12 option, analyzing the form_submit.php file for any violations of the PSR-12 standard. If there are errors or warnings, they will be displayed in the terminal. The || true at the end ensures that the Makefile execution continues even if this command encounters failures. This is done to prevent the linter's failure from halting the entire Makefile execution, allowing the focus to remain on the Docker-related tasks. If we have interest in fixing the code errors, we will use the phpcbf.phar to fix them.

build: This target automates the process of building a Docker image based on the Dockerfile located in the current working directory. The image is tagged with the name php-test. Additionally, it checks for the existence of a Docker network named test_network, and if it doesn't exist, the target creates it.
run: This target is responsible for starting two Docker containers: one for PHP and another for phpMyAdmin

For the php container, the command uses docker run to start a new container, -d flag runs the container in detached mode, meaning it runs in the background. The --name php option assigns the name "php" to the container. The --network test option connects the container to the Docker network named test. The -p 80:80 option maps port 80 from the host to port 80 in the container, allowing access to the web server running inside the container. The --env-file .env option specifies the location of an environment file (.env), which is in the current directory, that contains environment variables for the container. php-test specifies the Docker image to use for the container.

For phpmyadmin container, the command uses docker run to start another container, this time for phpMyAdmin. The -d flag runs the container in detached mode. The --name phpmyadmin-test option assigns the name phpmyadmin-test to the container. The --network test option connects the container to the same Docker network test as the PHP container. The -p 8000:80 option maps port 8000 from the host to port 80 in the container, allowing access to phpMyAdmin's web interface.
The -e PMA_ARBITRARY=1 and -e PMA_HOST=mysql options are environment variables for phpMyAdmin container to configure its behaviour. phpmyadmin specifies the Docker image to use for the phpMyAdmin container. The image is pulled from the Docker registry.

all: This target acts as a meta-target, referencing all the targets defined in the Makefile. When you execute the command make all, it will sequentially execute each target in the order they are defined in the Makefile.
To execute the Makefile and automate the build process, run the following command in the terminal:

make all

The Makefile will automatically execute all the defined targets in the specified order, as described previously. This one-liner command saves you from manually performing individual tasks and ensures that the complete workflow, including installation, linting, Docker image building, and running containers, is handled seamlessly.

After executing all the targets in the Makefile using make all, you can check your browser, http://localhost:8000, to access the phpMyAdmin container. However, please note that the PHP and phpMyAdmin containers are not fully functional at this point since they still need to be connected to a MySQL database for full functionality.

Create the MySQL database

In the root of the project folder, create a folder called mysql, cd into the directory and create two files: db.sql and makefile

mkdir mysql
cd mysql/
touch db.sql
touch makefile

Paste the below contents into the db.sql file:

CREATE TABLE IF NOT EXISTS test (
    id INT NOT NULL AUTO_INCREMENT,
    name VARCHAR(255),
    email VARCHAR(255),
    message TEXT,
    PRIMARY KEY (id)
);

The above is an SQL script that creates a table named test with specific column definitions.

CREATE TABLE IF NOT EXISTS: This statement creates the test table only if it doesn't already exist in the database. This prevents any potential errors from re-creating an existing table.

The test table has four columns:

id: An integer column with the NOT NULL constraint, meaning it cannot contain null values. It is also defined as AUTO_INCREMENT, which automatically generates a unique value for each new row.
name: A variable-length character column with a maximum length of 255 characters.
email: Another variable-length character column with a maximum length of 255 characters.
message: A column of the TEXT data type, which can store large amounts of text.
Paste the below configuration into the makefile file:

# Makefile for mysql

pull:
    docker pull mysql:latest

run:
    docker run -d --name mysql --network test -v ./data:/var/lib/mysql -v ./db.sql:/docker-entrypoint-initdb.d/db.sql -e MYSQL_USER=favour -e MYSQL_PASSWORD=mypassword -e MYSQL_ROOT_PASSWORD=Strongpassword@123 -e MYSQL_DATABASE=dev_to -p 3306:3306 mysql:latest

# Target to run the entire setup process
all: pull run

This Makefile provides a convenient way to set up a MySQL container using Docker and automates the process with the following targets:

pull: The pull target is used to pull the latest MySQL Docker image (mysql:latest) from the Docker registry. This ensures that the latest version of the MySQL image is available locally for running the container.
run: The run target is responsible for starting the MySQL container. It uses docker run to create and run a new container named mysql-test. The container is connected to a network called test (--network test) and mounts a local SQL file (db.sql) to the container's /docker-entrypoint-initdb.d/db.sql path. This allows the SQL file to be executed during container initialization, populating the database with any initial data.

Additionally, environment variables are provided for configuration:

-e MYSQL_ROOT_PASSWORD=Strongpassword@123: Sets the root user's password to Strongpassword@123.
-e MYSQL_DATABASE=dev_to: Creates a database named dev_to.
all: The all target acts as a meta-target, referencing both pull and run.

NB: Keep it in mind that important credentials shouldn't be passed directly on the command line, we are passing them directly here just to show the process.

To create the database, run the makefile with the following command from inside the mysql directory:

make all

When you execute make all, it will execute both targets in the specified order. This allows you to pull the latest MySQL image and then run the MySQL container with the necessary configurations in a single command.

Now on your browser, visit localhost to see the form.html file being served by apache. When you fill the form, if the input you gave is successful saved in the database, you will be redirected to localhost:80 where php is running. You will get a php message saying it was successful, if it wasn't, php will still print out an error message.

With Docker-Compose

After using Makefiles to run the containers as well as other processes, we will use Docker Compose which is an effective container orchestration tool to run the containers.

Environment Variables

We need to first create environment variables for the MySQL container to use since it is not good practice to pass sensitive information directly.

In the root of the folder, create a .env file:

touch .env

Paste the below into it

MYSQL_PASSWORD=mypassword
MYSQL_ROOT_PASSWORD=Strongpassword@123
DB_HOST=mysql
MYSQL_DATABASE=dev_to
DB_USER=favour

Replace with the appropriate attributes.

Docker Compose

Now that the environment variables have been set, we can proceed to writing the Docker Compose file.

In the root of the project folder, create a file called docker-compose.yml:

touch docker-compose.yml

Paste the below contents into the file:

version: '3'
services:
  php:
    env_file:
      - .env
    build:
      context: ./php
      args:
        - --no-cache
      dockerfile: Dockerfile
    ports:
      - "80:80"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/"]
      interval: 10s
      timeout: 5s
      retries: 3
    container_name: php
    networks:
      - backend

  mysql:
    env_file:
      - .env
    image: mysql:latest
    restart: always
    ports:
      - "3306:3306"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      timeout: "5s"
      interval: "10s"
      start_period: "3s"
      retries: 5
    environment:
      MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
      MYSQL_DATABASE: "${MYSQL_DATABASE}"
      MYSQL_USER: "${DB_USER}"
      MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
    container_name: mysql
    volumes:
      - ./mysql/db.sql:/docker-entrypoint-initdb.d/db.sql
      - ./mysql/mysql_data:/var/lib/mysql
    networks:
      - backend

  phpmyadmin:
    image: phpmyadmin:latest
    container_name: phpmyadmin
    ports:
      - 8000:80 # Expose phpMyAdmin on port 8000
    restart: always
    environment:
      PMA_ARBITRARY: 1 # Use the value '1' for arbitrary hostname resolution
      PMA_HOST: "${DB_HOST}" # Use the container name of the mysql service as the host
    depends_on:
      - mysql     
    networks:
      - backend

networks:
  backend:
    driver: bridge

The above docker-compose file defines a multi-container environment using Docker Compose, allowing you to run and manage multiple services (containers) together as part of a single application.

Breaking down the files to bits:

Version: The file specifies the version of Docker Compose syntax being used. In this case, it's using version 3.
Services: This section defines three services (containers): php, mysql, and phpmyadmin.
php service: The php service is built using the Dockerfile located in the ./php directory. It uses environment variables defined in the .env file. The service is accessible on port 80 and has a health check to test the health of the container by making an HTTP request to http://localhost/.
mysql service: The mysql service uses the official MySQL image (mysql:latest) from Docker Hub. It reads environment variables from the .env file to set up MySQL's root password, database, user, and password. The container is accessible on port 3306, and it has a health check to verify the health of the MySQL server by pinging it.
phpmyadmin service: The phpmyadmin service uses the official phpMyAdmin image (phpmyadmin:latest) from Docker Hub. It exposes phpMyAdmin on port 8000 and depends on the mysql service. Environment variables are set to configure phpMyAdmin to connect to the MySQL container.
Volumes: The mysql service mounts two volumes: ./mysql/db.sql to initialize the database with an SQL file and ./mysql/mysql_data to persist MySQL data.
Networks: The backend network is created to allow communication between the services (php, mysql, phpmyadmin).
To run the containers, in the root directory where the docker compose file exists, run the following command:

docker compose up -d

This command automatically builds the Docker image and runs the containers.

== we didn't persist data for makefile command

To bring down the containers:

docker compose down

Container Health Checks

Going by standard and best practices, containers should have health checks to ensure they are running properly and responding to requests as expected. Health checks have already been defined in the Docker Compose file, ensuring that the containers are periodically monitored for their health status. When you run the command docker ps -a, you should see the containers listed with their health status displayed under the STATUS column.

CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS                    PORTS                               NAMES
eb6e3ddc3e53   phpmyadmin:latest     "/docker-entrypoint.…"   17 seconds ago   Up 15 seconds (healthy)   0.0.0.0:8000->80/tcp                phpmyadmin
1f1abdcbd82d   mysql:latest          "docker-entrypoint.s…"   17 seconds ago   Up 15 seconds (healthy)   0.0.0.0:3306->3306/tcp, 33060/tcp   mysql
2b91dfbbb74d   module-2-php          "docker-php-entrypoi…"   17 seconds ago   Up 15 seconds (healthy)   0.0.0.0:80->80/tcp                  php

At initial start of the containers, the STATUS column will show (health: starting) since the container is in the process of starting, and the health check is currently being evaluated. The status of a container can also be unhealthy. This status indicates that the container's health check has failed, signaling that there might be an issue with the container or its underlying application. If the status shows exited, it means that the container has stopped running, either because it has completed its task or due to an error.

Health checks can also be done manually

To run health checks manually on your container, for the mysql container, run:

docker inspect --format='{{json .State.Health.Status}}' <container name>

and it should return an output of the container's health status

Do the same for the php container and then the phpmyadmin container

Beginners guide to GitOps and Flux

Daniel Favour — Fri, 26 May 2023 12:37:22 +0000

Managing code changes in a Kubernetes cluster can be complex, particularly when multiple applications are involved. Keeping track of changes, versions, and dependencies can be challenging, and conflicts can arise that impact cluster stability.

GitOps provides a solution to these challenges by leveraging Git as the source of truth for all changes to the cluster. By committing all configuration changes and updates to a Git repository, GitOps provides a centralized location for tracking all changes to the cluster, while also providing a standardized approach to deploying and updating applications.

In this article, we will explore the GitOps methodology and take a closer look at Flux, a popular GitOps tool used for managing Kubernetes clusters.

What is GitOps?

GitOps is a software development approach that emphasizes using Git as the primary tool for managing infrastructure and application deployments. Essentially, GitOps means that all changes to infrastructure and application code are made through Git commits and that these changes trigger automated deployment pipelines to update the running environment. This approach enables teams to easily track changes, and rollback deployments, and ensure that the production environment matches the desired state described in Git.

GitOps principles

The GitOps principles are essential for modern software development and deployment practices. These principles allow teams to automate the management of their infrastructure and applications, and ensure that their deployments are consistent and reliable.
In this section, we will discuss the four (4) key principles of GitOps, which are:

1. The entire system is described declaratively

GitOps mandates infrastructure and application code to be declared in a declarative way. A declarative approach involves specifying the desired end result rather than providing specific instructions on how to accomplish the task.

For example, in Kubernetes, to describe the desired state of an application through a declarative approach, we define the desired state in a YAML file, and Kubernetes takes care of managing the underlying infrastructure to ensure that the application is running as intended.

Say we want to deploy a simple web application in a Kubernetes cluster, we define the desired state of the application using a Kubernetes Deployment object, which includes the container image, number of replicas, and other specifications. Here is an example YAML file:



apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-webapp
  template:
    metadata:
      labels:
        app: my-webapp
    spec:
      containers:
        - name: my-webapp-container
          image: my-webapp-image:latest
          ports:
            - containerPort: 80

This YAML file describes the desired state of Kubernetes Deployment for our web application.
We want to have three replicas of our container, which will be managed by a Kubernetes ReplicaSet. The container will listen on port 80, and the image used will be my-webapp-image:latest.
Kubernetes will take care of ensuring that the desired state is met, even if there are changes in the underlying infrastructure.

2. The desired system state is versioned in Git

The desired state is the ideal state that developers and operators aim to achieve and maintain. This state is stored in Git to keep track of changes made to the system over time, revert to previous versions, and collaborate with others to make changes. By storing the desired state in Git, developers can maintain consistency, reliability, and security, while also working in an agile way to continually improve the system over time.

3. Approved changes can be automatically applied to the system

After the desired state has been stored in Git, GitOps operators, also known as software agents, automatically retrieve the desired state from Git and apply it to one or more Kubernetes targets. A software agent is a software component that continuously monitors the changes made to a Git repository and triggers the deployment of those changes to a target environment, such as a Kubernetes cluster. This process occurs without the need for manual intervention, allowing for seamless and efficient deployment of the desired state.

4. Software agents ensure correctness and reconcile the system to match the desired state

GitOps operators follow a continuous loop that involves observing the repository for any changes in the desired state. This process entails comparing the difference between the actual state and the desired state, after which the operators automatically take action to reconcile the two states.

The GitOps workflow

GitOps Pipeline: Source Weaveworks

Assuming that your code is stored on a Git repository and contains Kubernetes manifests such as helm charts, Kustomizations, etc, the task at hand is to retrieve these manifests from the repository and deploy them to your Kubernetes cluster.

Within your Kubernetes cluster, there exist multiple namespaces, services, and various cloud-native tools installed. A Kubernetes beginner typically resorts to the "Imperative" method of deployment, which involves utilizing the following command to apply a configuration file:



kubectl apply -f "filename"

Imperative approach involves providing specific instructions or commands on how to accomplish the task rather than specifying the desired end result. GitOps discourages this.

One of the limitations of the imperative approach is that any modifications made to the Kubernetes manifest stored in the Git repository will not automatically reflect on the Kubernetes cluster. As a result, the kubectl apply command must be executed again to synchronize the cluster with the updated manifest.

In a scenario where multiple team members are making changes to the manifests files in a Git repository, it can be a daunting task to monitor the changes taking place in the cluster. Tracking who deployed what resource, what was deployed, or when it was deployed can become a challenging feat, resulting in a state of confusion when an issue arises and needs to be resolved. The lack of knowledge of the root cause of the problem can lead to significant delays in fixing the issue.

The Solution

By implementing a GitOps tool like Flux, you can efficiently track the changes in the cluster, which ensures that any issue that arises can be fixed promptly. With GitOps, you can know that any change made to the cluster is being tracked, and in the event of an issue, the cause can be easily identified, and the problem resolved quickly.

How it works

With GitOps you deploy an agent into your Kubernetes cluster. The agent in this case is Flux. Flux helps you to manage the resources in your Kubernetes cluster.

As teams make changes to the Git Repository, they make changes just to the code. A CI/CD pipeline running in that repository tests the code changes made, and after passing the tests, the changes get deployed to the repository. Once the changes have been deployed, the agent living in your cluster is responsible to pull that change into your Kubernetes cluster and deploys it to the specific resource the change was made to.

How does the agent know about the Kubernetes manifest?
The agent responsible for deploying and managing Kubernetes resources can know about the Kubernetes manifest by:

Reading the manifest files directly from a version control system, such as a Git repository, or a local file system.
Receiving the manifest as input from a user or another system

However, the Cluster administrator remains at the center of the process as he/she is the one who tells the agent where the manifest files are located. The agent is then responsible to detail who deployed what, what is deployed, and when it was deployed to the cluster.

What is Flux?

Flux is a package management system that operates on a Git-based platform, delivering a range of smooth and incremental deployment solutions for Kubernetes. Its primary objective is to maintain synchronization between Kubernetes clusters and configuration sources such as Git repositories. It achieves this by automating the updating of configurations whenever new code is available for deployment.

Flux Components

Flux is a powerful GitOps tool that is constructed using a variety of specialized components, collectively known as the GitOps Toolkit. These components include Flux Controllers, composable APIs, and reusable Go packages that are designed to enhance the functionality of GitOps.

Flux Controllers in the GitOps Toolkit

Source: Flux Docs

Source Controller: It automates the process of obtaining files from external sources and integrating them into application code. This includes sources such as Git, OCI, Helm repositories, and S3-compatible buckets. The source controller makes it easier and faster for developers to develop and deploy applications without worrying about the details of accessing and integrating external resources.
Image Reflector Controller: It monitors container registries for changes and updates the Kubernetes cluster with the latest images.
Image Automation Controller: It automates the process of updating the container image in the repository and commits changes to the repository automatically.
Kustomize Controller: It applies Kubernetes manifests generated by Kustomize to a cluster and ensures the desired state of the cluster.
Helm Controller: It automates the deployment of Helm charts to a cluster and ensures that the desired state of the cluster is maintained.
Notification Controller: It sends notifications to users and teams about changes made to the Kubernetes cluster, such as deployments and configuration updates.

Benefits of Flux

In this section, we will look at some of the benefits of Flux which include:

1. Automation

Flux automates the deployment process by continuously monitoring and updating Kubernetes resources in real time. This automation reduces the likelihood of human error and improves reliability by ensuring that the correct resources are deployed to the correct environments consistently.

2. Scalability

Flux is designed to work well in large-scale Kubernetes environments. It can handle multiple clusters, applications, and even entire infrastructures at once, which makes it a great tool for managing complex deployments.

3. Rollback

Flux provides easy rollbacks in case of deployment failures or errors. It does this by using a Git-based version control system, which allows developers to roll back to a previous working version of the deployment. This feature reduces downtime and improves service availability, ensuring that the application remains operational even in the event of a deployment failure.

4. Integration

Flux CD can be integrated with a range of tools, including monitoring, logging, and testing frameworks. This integration allows developers to manage the entire deployment process from a single platform, which improves collaboration and streamlines the deployment process.

5. GitOps

Flux uses the GitOps methodology, which means that all changes to the deployment process are stored in version control. This provides greater transparency, auditability, and reproducibility of the deployment process, allowing developers to easily track changes and roll back to previous versions if necessary.

6. Customization

Flux is highly customizable and can be tailored to fit specific deployment requirements. This customization allows developers to automate their deployment process in a way that meets their unique needs, improving efficiency and reducing errors.

Conclusion

GitOps and Flux represent a powerful approach to software development and deployment that can help teams achieve greater speed, reliability, and scalability.

By using Git as the single source of truth, GitOps enables teams to manage updates across multiple environments and platforms, while Flux provides automation tools for managing Kubernetes clusters. This approach facilitates collaboration, continuous delivery, and scalability, making it well-suited for modern cloud-native environments.

Ultimately, the adoption of GitOps and Flux is expected to lead to faster, more reliable, and more efficient software development and deployment.

Deploy your Application to Azure Kubernetes Service

Daniel Favour — Sun, 21 May 2023 06:32:26 +0000

Cloud computing has become a critical component of application development and deployment in today's fast-paced digital world. Azure, a major cloud platform, provides a comprehensive suite of tools and services to enable easy deployment, management, and scaling of applications. However, deploying applications to the cloud can be a complex and overwhelming task without appropriate guidance and the right set of tools and services.

This article guides you through deploying your application to Azure, from containerization to monitoring and scaling.

Prerequisites

Before we get started, ensure that you have the following in place:

An active Azure Subscription
Azure CLI installed
Basic knowledge of Azure
Node installed
Terraform Installed
An IDE, VSCode recommended

Application Overview

The application utilized for this project is developed using Node.js and Express.js. It serves as a feedback app, allowing users to provide their feedback, which is then stored in the designated feedback directory within the application.
It provides an interface or endpoint through which users can submit their feedback input. Once submitted, the application stores the feedback in the specified feedback directory, ensuring proper organization and retention of user input.

GitHub URL

https://github.com/FavourDaniel/SCA-Project

To clone this repository:

git clone https://github.com/FavourDaniel/SCA-Project
cd SCA-Project

Architecture Diagram

Below is a high level overview of what we will be building. There will be some other configurations in place like Virtual Networks, Load Balancers, etc, but this architecture diagram should give you a clear picture of what the infrastructure will look like.

Let's get started.

Testing the application on your local

npm install express
node server.js

Open your browser and navigate to localhost:80 to view the running application.

Login to your Azure Account

To set up your infrastructure on the Azure cloud platform, a connection between your terminal and your Azure account needs to be established. To accomplish this using the Azure CLI, run the following command:

az login

You will be redirected to sign in to your account. After successful authentication, your terminal should display an output similar to the one below:

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "56d87bs9-fr5n-8zzq-qq01-419l20234j0f",
    "id": "5h12gg64-60d2-1b4h-6j7c-c810419k33v2",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Visual Studio Enterprise Subscription",
    "state": "Enabled",
    "tenantId": "56d87bs9-fr5n-8zzq-qq01-419l20234j0f",
    "user": {
      "name": "username@gmail.com",
      "type": "user"
    }
  }
]

Copy the id value, it is your subscription ID. Next, run the following command:

subscription="<subscriptionId>" # add subscription here
az account set -s $subscription

Replace <subscriptionId> with the Id value you copied and the connection should be established.

Setting up the Infrastructure

The infrastructure will be set up on Azure using Terraform. From the GitHub link provided, you need to change directory into terraform-infrastructure directory. To do this, run the following commands:

cd terraform-infrastructure
terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

I suggest running these commands individually. Terraform will proceed to create the infrastructure specified in the main.tf file.

NB: You might encounter errors if any of the names used in the vars.tf file is already in use on Azure by someone else. By names here, I mean default which are the default names assigned to a resource and can be found in the vars.tf file. In such case, assign a different name to the infrastructure that failed to create.

To confirm that your resources have been created successfully, you can check your Azure portal and you should see the following resources created:

Resource Group

Azure Container Registry

Azure Kubernetes Service

Storage Accounts

These resources were specified in the Terraform configuration file.

NB: Azure Kubernetes Service being a managed service will by default create other services on its own when created which include Virtual Networks, new Resource Groups, Load Balancer, Network Security Group, Storage Class, Route Table, etc. These extra services do not need to be created by you, that is why it is a managed service.

Build and Push the Docker Image

In the root of the the project directory, there is Dockerfile file which serves as a template for creating the Docker image. This Docker Image will not be created manually as a job for this has been setup in the GitHub Actions workflow which can also be found at the root directory.

GitHub Actions is a CI/CD tool that automates software development workflows, including building, testing, and deploying code, directly from your GitHub repository.

To build the Docker Image, create your own repository on GitHub and push the project there. Because there is a workflow that has already been set up but not configured properly, the job will fail.

The image will be built but will not be pushed. This is because in the workflow, the below was specified:

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v2

This first step uses the docker/setup-buildx-action@v2 action to creates and boot a builder using the docker-container driver.

After it successfully does this, it proceeds to login to your Azure Container Registry which was previously created.

- name: Login to Azure Container Registry
  uses: Azure/docker-login@v1
  with:
    login-server: ${{ secrets.ACR_REGISTRY_NAME }}.azurecr.io
    username: ${{ secrets.ACR_USERNAME }}
    password: ${{ secrets.ACR_PASSWORD }}

From the above, Azure/docker-login@v1 is an action that logs in to an Azure Container Registry. Because the login server, username and password of your Container Registry has not been provided yet, it is expected to fail.

Same as the next step:

- name: Build Docker image
  uses: docker/build-push-action@v4
  with:
    context: .
    file: ./Dockerfile
    push: true
    tags: ${{ secrets.ACR_REGISTRY_NAME }}.azurecr.io/myrepository:${{ github.sha }}

From the above, docker/build-push-action@v4 is an action that builds and pushes Docker images using Docker Buildx. If it manages to build the image, it will fail to push it because it doesn't have the credentials that grants it access to our Azure Container Registry.

Grant the Workflow access

To grant the workflow access to your Container Registry, head over to your Azure portal. Under Container Registries, select your Container registry and from the left panel, select Access keys and enable Admin user.

This will generate passwords for you which you can always regenerate later for security reasons.

Copy the Login Server, Username and one of the password.

Head back to your GitHub repository and select Settings

Scroll down and select Actions under Secrets and variables

Click on the New repository secret button

Name the first secret ACR_REGISTRY_NAME. This is your login server which was copied previously. Paste the login server into the secret box and click on Add secret.

Do the same for ACR_USERNAME and ACR_PASSWORD replacing them with the username and password that was copied. Afterwards, you should have this:

Now that the workflow has access to your Container Registry, re-run the workflow and it should pass.

Check you container registry under Repositories and you should see your image there.

Install the Kubernetes CLI

To manage your AKS cluster using the Kubernetes command-line interface (CLI), you will need to install the kubectl tool on your local machine.
You can do this by running the following command:

az aks install-cli

This command will install the kubectl tool on your machine, allowing you to interact with your AKS cluster using the Kubernetes API. Once the installation is complete, you can verify that kubectl is installed by running the kubectl version command.

Connect to the AKS cluster using kubectl

To connect to your AKS cluster using kubectl, you need to configure your local kubectl client with the credentials for your cluster. You can do this by running the following command:

az aks get-credentials --resource-group DemoRG --name myAKSCluster

The Resource Group name DemoRG and cluster name myAKSCluster are defined in the vars.tf file.

The above command retrieves the Kubernetes configuration from the AKS cluster and merges it with your local Kubernetes configuration file. This makes it possible for you to use kubectl to interact with the AKS cluster.

After running the command, you should see output similar to the following:

Merged "myAKSCluster" as current context in /home/daniel/.kube/config

This indicates that the Kubernetes context named myAKSCluster has been merged into the kubeconfig file located at /home/daniel/.kube/config.

This kubeconfig file is used to store cluster access information, such as cluster, namespace, and user details, for the kubectl command-line tool kubernetes.io.

In this case, the myAKSCluster context has been set as the current context, which means that subsequent kubectl commands will use the configuration from this context to communicate with the Kubernetes cluster.

To confirm that your kubectl client is properly configured to connect to your AKS cluster, you can run the following command:

kubectl get nodes

This command will return a list of the nodes in your AKS cluster, which should include the node that was created at cluster creation. In the Terraform configuration, only one (1) node was specified to be created so you should see only one.
The output should look similar to the below:

NAME                                  STATUS   ROLES   AGE     VERSION
aks-aksnodepool-16067207-vmss000000   Ready    agent   2d13h   v1.25.6

Deploy your Application to the Cluster

To deploy our application to the AKS cluster, we need to create a deployment, a service, a persistent volume claim, and a storage class.

Deployment: Manages a set of replicas of your application's containers and handles scaling, rolling updates, and rollbacks.
Service: Provides a stable IP address and DNS name for your application within the cluster.
Persistent Volume Claim: Requests a specific amount of storage for your application's data.
Storage Class: Defines the type of storage that will be provisioned for the claim. In this case, we will be using Azure File Storage as our storage class.

To create our resources, we need to first write the manifest files for them. We will start creating the Storage Class first. This is because the Persistent Volume Claim relies on the Storage Class for dynamic provisioning.

Storage Class

To create a manifest for the storage class, run the below command:

touch storageclass.yml

Then, paste the below configuration into it and save:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-file-storage
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1000
  - gid=1000
  - mfsymlinks
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS
  location: eastus

Here we defined the Azure File Storage to be our storage provisioner.

To create the storage class, run the following command:

kubectl apply -f storageclass.yml

To see the deployed storage class, run:

kubectl get storageclass

This will show you a list of all the available storage classes, including the one you just created.

You should see an output similar to the following:

NAME                    PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azure-file-storage      kubernetes.io/azure-file   Delete          Immediate              false                  2d12h
azurefile               file.csi.azure.com         Delete          Immediate              true                   2d14h
azurefile-csi           file.csi.azure.com         Delete          Immediate              true                   2d14h
azurefile-csi-premium   file.csi.azure.com         Delete          Immediate              true                   2d14h
azurefile-premium       file.csi.azure.com         Delete          Immediate              true                   2d14h
default (default)       disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   2d14h
managed                 disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   2d14h
managed-csi             disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   2d14h
managed-csi-premium     disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   2d14h
managed-premium         disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   2d14h

From the above output, we can see both Azure file storage and Azure disk storage provisioned.

When the Azure Kubernetes Service was created, it created the Disk storage. We are creating this File storage because we want to use a File storage rather than a Disk storage. If you decide to use the Disk storage, creating this storage class will not be necessary.

Persistent Volume Claim

The Persistent Volume Claim is the next resource we will create. This is because when we create our deployment, it will look for the persistent volume claim we specified and if it cannot locate it, it will fail.

To create the manifest for the PVC, run the below command:

touch pvc.yml

Paste the following configuration into it and save.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scademo-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: azure-file-storage

See Persistent Volume Claim for manifest breakdown.

To create the PVC, run the below:

kubectl apply -f pvc.yml

To check the created PVC, run the below command

kubectl get pvc

It should return the below output

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
scademo-pvc   Bound    pvc-20a3d37b-f734-4f53-ba96-d94e63463623   20Gi       RWO            azure-file-storage   2d12h

We can see that the PVC has been bound to the azure file storage class created.

If you check your Storage Account on your azure portal, you should see the file storage that was created.

Deployment

Now that the storage class and PVC have been created, we can create the deployment.

To create a manifest for the deployment, run the below command:

touch deployment.yml

Paste the below configuration into the file and save.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scademo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: scademo
  template:
    metadata:
      labels:
        app: scademo
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: scademo
        image: scacontainerregistry.azurecr.io/scademo:v1
        resources:
          requests:
            cpu: 100m 
            memory: 128Mi
          limits:
            cpu: 100m
            memory: 256Mi
        ports:
        - containerPort: 80 
        env:
        - name: TEST
          value: "scademo"
        volumeMounts:
        - mountPath: /tmp/data
          name: scademo-pvc
      volumes:
      - name: scademo-pvc
        persistentVolumeClaim:
          claimName: scademo-pvc

Refer to Deployments and YAML, to see a breakdown of the file configuration.

To create this deployment, run:

kubectl apply -f deployment.yml

To check your deployment, run:

kubectl get deployments

It should return the below:

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
scademo   1/1     1            1           44h

NB: Deployments automatically create and manage pods for you, eliminating the need for manual pod creation. Whenever you create a deployment, it creates a pod along with it as well as a Replica Set.

To see the deployed pod, run:

kubectl get pods

It should return the below output:

NAME                       READY   STATUS    RESTARTS   AGE
scademo-7885bbb755-q464b   1/1     Running   0          44h

To see the Replica Set, run:

kubectl get rs

It should return the below output:

NAME                 DESIRED   CURRENT   READY   AGE
scademo-7885bbb755   1         1         1       44h

Service

Now we need to expose this deployment so that we can access it from outside the cluster. To do this, we will create a service.

To create a manifest for the service, run the below:

touch svc.yml

Paste the below configuration into it and save.

apiVersion: v1
kind: Service
metadata:
  name: scademo
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: scademo

In the above configuration, we are utilizing a LoadBalancer to access the deployment. A LoadBalancer is a type of service that creates an external load balancer in a cloud environment and assigns a static external IP address to the service. It is commonly used to distribute incoming traffic across worker nodes and enables access to a service from outside the cluster.

To create the service, run the below command:

kubectl apply -f svc.yml

To see the created service, execute the following command:

kubectl get svc

It will return an output similar to the below:

NAME         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
kubernetes   ClusterIP      10.0.0.1      <none>         443/TCP        2d14h
scademo      LoadBalancer   10.0.44.214   40.88.195.57   80:32591/TCP   44h

The service will show pending at creation, after some seconds, the external IP address will be allocated.

Copy and paste the external IP on your web browser and you should see your application running:

Setup Monitoring for your Cluster

To effectively monitor resources in your Kubernetes cluster, it's recommended to set up Container Insights in Azure, which provides comprehensive monitoring of the entire cluster. Container Insights can be integrated with a Log Analytics Workspace to enable additional visibility and monitoring capabilities for containerized applications running in a Kubernetes environment.

Log Analytics Workspace

This is a centralized storage and management location for log data collected by Azure Monitor, including data from your AKS clusters. It allows you to query, analyze, and visualize the collected data to obtain insights and troubleshoot issues.

To enable monitoring for your AKS cluster, you need to specify a Log Analytics Workspace where the collected monitoring data will be stored if you have an existing Workspace.

You can create a Log Analytics Workspace using a default workspace for the resource group if you do not have an existing Workspace. This default Workspace will be created automatically with a name in the format DefaultWorkspace-<GUID>-<Region>.

To create this default workspace, execute the following command:

az aks enable-addons -a monitoring -n myAKSCluster -g DemoRG

A monitoring addon is a component that enables you to monitor the performance and health of your AKS cluster using Azure Monitor for containers. When you enable the monitoring addon, it deploys a containerized version of the Log Analytics agent on each node in your AKS allowing Azure Monitor for containers to monitor the performance and health of the AKS cluster.

This Log Analytics agent is a component running on each node of your Kubernetes cluster, responsible for collecting logs and metrics from the node and the applications running on it. The agent is deployed as a DaemonSet in the cluster, ensuring an instance of the agent runs on every node.

NB: The command requires the registration of the Microsoft.OperationsManagement resource provider with your subscription.

Microsoft.OperationsManagement stands for to Microsoft Operations Management Suite (OMS). OMS is a cloud-based service that provides monitoring and management capabilities across Azure.

If this service is not registered with your subscription, you will receive the following error output:

(AddContainerInsightsSolutionError) Code="MissingSubscriptionRegistration" Message="The subscription is not registered to use namespace 'Microsoft.OperationsManagement'. See https://aka.ms/rps-not-found for how to register subscriptions." Details=[{"code":"MissingSubscriptionRegistration","message":"The subscription is not registered to use namespace 'Microsoft.OperationsManagement'. See https://aka.ms/rps-not-found for how to register subscriptions.","target":"Microsoft.OperationsManagement"}]
Code: AddContainerInsightsSolutionError
Message: Code="MissingSubscriptionRegistration" Message="The subscription is not registered to use namespace 'Microsoft.OperationsManagement'. See https://aka.ms/rps-not-found for how to register subscriptions." Details=[{"code":"MissingSubscriptionRegistration","message":"The subscription is not registered to use namespace 'Microsoft.OperationsManagement'. See https://aka.ms/rps-not-found for how to register subscriptions.","target":"Microsoft.OperationsManagement"}]

To check the registration status using the following commands:

az provider show -n Microsoft.OperationsManagement -o table
az provider show -n Microsoft.OperationalInsights -o table

To register if not registered, execute the following commands:

az provider register --namespace Microsoft.OperationsManagement
az provider register --namespace Microsoft.OperationalInsights

You can check your Azure portal under Log Analytics Workspace to see the created Workspace.

Verify Agent Deployment

To verify that monitoring agent was successfully deployed into the cluster, run the following command:

kubectl get ds ama-logs --namespace=kube-system

The above command retrieves information about a DaemonSet named ama-logs in the kube-system namespace.

ama stands for Azure Monitor Agent.

The command should return the below output:

NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ama-logs   1         1         1       1            1           <none>          35h

We have only one (1) running because we have a single node. If we had 2 nodes, we will have two (2) agents.

Head back to your Azure portal. Under Azure monitor, select view under Container Insights or select Containers.

Under Monitored Clusters you should see your Cluster being monitored.

Select your Cluster and you should see Insights of your Cluster. You can change it to show insights of nodes, controllers, containers and also get a status report.

You can also choose to see the metrics as well other settings and configurations.

Scaling the deployment

To scale our Kubernetes deployment and ensure it can handle increased traffic, we need to edit our deployment configuration.

Initially, we specified our deployment to have only one replica, which is why it created a single pod. To scale the deployment, we can edit the deployment.yml file and change the replica value from 1 to 4 (or any desired number) to create four replicas of the application.

In the spec section, change the number of replicas to four (4)

spec:
  replicas: 4
  selector:
    matchLabels:
      app: scademo

After making the change, save the file and apply the updated configuration by running the following command:

kubectl apply -f deployment.yml

After applying the changes to the deployment file, give it some time to take effect. This will update the deployment with the new configuration and create the additional replicas. To confirm that the replicas have been created, we can run the following command to get the status of the deployment:

kubectl get deployment

The output should show that the scademo deployment now has 4 replicas, as indicated by the READY column.

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
scademo   4/4     4            4           13h

Previously, there was only one replica of the deployment, so the output showed 1/1. After increasing the replica count to 4, the output now shows 4/4, indicating that there are four replicas of the deployment running and all of them are available.

You can also verify that there are 4 running pods by running the command:

kubectl get pods

It should return the below output:

NAME                       READY   STATUS    RESTARTS   AGE
scademo-7885bbb755-644r9   1/1     Running   0          13h
scademo-7885bbb755-q464b   1/1     Running   0          35h
scademo-7885bbb755-gjnjp   1/1     Running   0          13h
scademo-7885bbb755-kvgg5   1/1     Running   0          13h

Additionally, since deployments are used to manage replica sets, you can check the replica sets by running the command:

kubectl get replicaset

It should return the below output:

NAME                 DESIRED   CURRENT   READY   AGE
scademo-7885bbb755   4         4         4       35h

The output shows that the scademo-7885bbb755 replica set has 4 replicas that are all ready and available.

That is how you scale your application in Kubernetes.

Conclusion

In this article, we explored the process of deploying an application to Azure and the various tools and services that can make the process seamless and efficient. By leveraging the right knowledge and resources, you can unlock the full potential of the Azure platform and take advantage of its vast array of features and capabilities.

Connecting an App Service to Azure SQL Database and Storage Account using Azure CLI

Daniel Favour — Wed, 03 May 2023 21:39:41 +0000

Effective data management is crucial for optimal application performance, and SQL databases and storage accounts play a critical role in streamlining data storage and access.

SQL databases provide a reliable and scalable method of storing and managing data in a structured manner, making it easier to search, retrieve, and analyze, while storage accounts offer a secure and scalable platform for storing and accessing large amounts of unstructured data. By combining these technologies, developers can create a robust data management strategy that meets their app's unique needs.

In this article, we will explore how to connect an App Service to an Azure SQL database and Storage Account using Azure CLI, a command-line tool that allows you to manage and interact with Azure resources using a shell or command-line interface.

Prerequisites

Before we get started, ensure that you have the following in place:

An active Azure Subscription
Azure CLI installed
Basic knowledge of Azure

Connect to our Azure Account

To connect to our Azure account, we can use the Azure CLI to sign in by running the command below:



subscription="<subscriptionId>" # add subscription here
az account set -s $subscription # ...or use 'az login'

Run the below command to obtain your subscription ID if you are unaware of it



az login

Upon executing the command, a prompt will appear requesting login credentials for your Azure account. After successful authentication, your terminal should display an output similar to the one below:



[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "56d87bs9-fr5n-8zzq-qq01-419l20234j0f",
    "id": "5h12gg64-60d2-1b4h-6j7c-c810419k33v2",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Visual Studio Enterprise Subscription",
    "state": "Enabled",
    "tenantId": "56d87bs9-fr5n-8zzq-qq01-419l20234j0f",
    "user": {
      "name": "username@gmail.com",
      "type": "user"
    }
  }
]

The "id" value is your subscription ID.

Create a Resource Group

In Azure, all resources created are tied to a Resource Group. A Resource Group is a logical container that holds related Azure resources, such as virtual machines, storage accounts, and virtual networks.

To create a Resource Group, run the below script



export location="east us"
export resourceGroup="DemoRG"
export tags="connect-to-sql"

echo "Creating $resourceGroup in "$location"..."
az group create --name $resourceGroup --location "$location" --tag $tag

Environment variables are dynamic values that can affect the behavior of programs and scripts. They can be referenced by using their name in dollar signs. In the Azure CLI, exporting variables allows us to reference them using a dollar sign instead of giving the name directly in the command. Exported variables are available to all processes launched from the current shell session, but need to be re-exported if the shell or terminal is changed.

In the above script, we exported the environment variables (location, resourceGroup, and tags) which are used in creating the resource group. Because these variables have been exported into the system, we can reference them when needed.

You can choose a different Resource group name, a different location, and tag as well.

After running the previous script, you should get the below output



Creating DemoRG in east us...
{
  "id": "/subscriptions/5h12gg64-60d2-1b4h-6j7c-c810419k33v2/resourceGroups/DemoRG",
  "location": "eastus",
  "managedBy": null,
  "name": "DemoRG",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": {},
  "type": "Microsoft.Resources/resourceGroups"
}

This shows that the Resource Group was successfully created. You can still check directly in your Azure portal to be sure.

Create an App Service Plan

In Azure, an App Service Plan is a logical container for hosting Azure App Service apps. It defines the underlying virtual machine instance and resources that are required to host the apps. You might confuse this for Azure App Service itself.

Azure App Service is a Platform as a Service (PaaS) offering in Microsoft Azure that enables developers to quickly build, deploy, and manage web apps, mobile app backends, and RESTful APIs. It provides a fully managed platform for hosting and scaling web applications without the need to manage the underlying infrastructure.

An Azure App Service Plan, on the other hand, is a logical container for hosting one or more Azure App Service apps. It provides the necessary infrastructure resources, such as virtual machines, memory, and storage, to run and scale the apps. The resources allocated to an App Service Plan determine the capacity and performance of the apps hosted in that plan.

To create an App Service plan, run the below command;



export appServicePlan="demo_app_service"

echo "Creating $appServicePlan"
az appservice plan create --name $appServicePlan --resource-group $resourceGroup --location "$location"

From the above, we exported an appServicePlan variable which the command will reference to create the App Service Plan. You can check your Azure portal to see the Azure app service plan created.

Create a Web App

Recall that an App Service Plan is used to host one or more App Services while an App Service itself is used to deploy and manage web apps.

When you create a Web App from the Azure CLI, it is automatically associated with an App Service. This is because a Web App needs a platform to run on, and an App Service provides that platform. It is created within an App Service as a child resource.

An App Service Plan is needed in Azure to allocate and manage the computing resources required for hosting one or more App Services. It defines the underlying infrastructure resources needed to run and scale your Web Apps, such as CPU, memory, disk space, and network bandwidth. You must choose an App Service Plan when creating an App Service, which allows you to scale up or down the resources allocated to your App Service based on demand. An App Service Plan ensures that your Web Apps have the necessary resources to operate efficiently and can easily scale resources up or down to meet changing demands.

To create a Web App, run the below command:



export webapp="FavourWebApp"

echo "Creating $webapp"
az webapp create --name $webapp --plan $appServicePlan --resource-group $resourceGroup

Again, we exported a variable webapp which the command referenced in creating the Web App. As you can see, previous exported variable didn’t need to be referenced again since it was done previously. The web app is then created under the same Resource Group and location as the App Service Plan.

NB: If you encounter an error message similar to the one below, it indicates that another user has already created or deployed a web app with that name. When you create a Web App, it will have a subdomain of azurewebsites.net. If the name you have chosen is already being used, you will need to select a different name to proceed. For instance, when I used the name DemoApp, it returned an error because the name was already in use.



Creating DemoApp
Webapp 'DemoApp' already exists. The command will use the existing app's settings.
Unable to retrieve details of the existing app 'DemoApp'. Please check that the app is a part of the current subscription

After successfully deploying the web app you can check it on the Azure Portal. It can be found under App Services.

You can also load the url on your browser to see it.

Create an SQL Server

An SQL Server is a software program that provides a platform for creating, managing, and accessing databases using a structured query language (SQL). It is a necessary component for managing databases because it provides the infrastructure to store, manage, and secure data. A database must be attached to a SQL Server in order to access and manage the data it contains. Without a SQL Server, a database cannot be accessed or managed.

Before we create the SQL database, we need to create a SQL Server that the database will run on.

To do this, run the below command:



export server="favourdemo"
export login="favour"
export password="yourpassword@123"

echo "Creating $server"
az sql server create --name $server --resource-group $resourceGroup --location "$location" --admin-user $login --admin-password $password

In the above configuration, you can choose a different server name to be exported, a login name, and a password of your choice.

Similar to how a subdomain is created for a Web App, creating an SQL Server on Azure results in a subdomain of database.windows.net. If the chosen name for the SQL Server is already taken, an error message will be displayed.

The rules for choosing a password are

Your password must be at least 8 characters in length.
Your password must be no more than 128 characters in length.
Your password must contain characters from three of the following categories – English uppercase letters, English lowercase letters, numbers (0-9), and non-alphanumeric characters (!, $, #, %, etc.).
Your password cannot contain all or part of the login name. Part of a login name is defined as three or more consecutive alphanumeric characters.

NB: If you do not follow the password rules, you will get an error.

You can check your Azure portal to see the created SQL Server.

Configure Access to the SQL Server

After creating the SQL Server, we need to configure a firewall for access to control access to the server. This is important for security reasons as it allows only authorized users or applications to access the SQL Server. Without proper firewall configuration, the server may be vulnerable to unauthorized access and potential security breaches.

To do this, run the below command:



export startIp="0.0.0.0"
export endIp="0.0.0.0"

echo "Creating firewall rule with starting ip of $startIp" and ending ip of $endIp
az sql server firewall-rule create --server $server --resource-group $resourceGroup --name AllowYourIp --start-ip-address $startIp --end-ip-address $endIp

From the above configuration, we are exporting a start IP and an End IP which refer to the range of IP addresses that are allowed to connect to the server.

The start IP is the first IP address in the range of allowed addresses, while the end IP is the last IP address in the range. Any IP address outside this range will be blocked from accessing the SQL Server.

NB: Because both startIP and endIP are set to “0.0.0.0”, it means we are opening the firewall to allow access from anywhere. This is strictly for demo purposes. It is unsafe for production environments so always ensure to restrict access to your server.

Create an Azure SQL Database

An SQL database is a type of relational database management system (RDBMS) that stores data in a structured format using tables consisting of rows and columns. It uses SQL (Structured Query Language) to manage the data and allows for easy retrieval, insertion, and modification of data. When an App Service app is connected to a database, it typically stores structured data that can be queried and manipulated using SQL or other database management tools. Examples of this type of data include user information, product catalogs, and transaction logs.

To create an Azure SQL database in Azure, run the below command:



export database="demodatabase"

echo "Creating $database"
az sql db create --server $server --resource-group $resourceGroup --name $database --service-objective S0

The above configuration export the database variable (name), and creates the database using the variable on the SQL Server we previously created under the same Resource Group.

--service-objective S0 is a parameter used in the Azure CLI command to set the performance level of the Azure SQL Database when it is created or updated. In this case, S0 is the service objective, which represents the Standard service tier of Azure SQL Database.

Get Connection String for the Database

A connection string for a database is a text string that contains the information required to establish a connection to a database server. It typically includes the name and location of the database, the name of the server hosting the database, and authentication information such as a username and password.

The connection string is required by applications or programs to establish a connection to the database server.

To get the connection string for the just created database, run the below command:



connstring=$(az sql db show-connection-string --name $database --server $server --client [ado.net](http://ado.net) --output tsv

-client ado.net: This specifies the client driver to use for the connection string. In this case, we are using the .NET client driver.

-output tsv: This specifies that the output of the command should be in TSV (tab-separated values) format, which is a type of text format that is easy to parse programmatically.

connstring=$(...): This assigns the output of the command to a shell variable called connstring. The $() syntax allows you to execute a command and capture its output.

The command az sql db show-connection-string is used to retrieve the connection string for the Azure SQL database that was just created. The output of this command is a string that contains information such as the server name, database name, login credentials, and other settings required to connect to the database as seen below.



Server=tcp:favourdemo.database.windows.net,1433;Initial Catalog=demodatabase;Persist Security Info=False;User ID=<username>;Password=<password>;MultipleActiveResultSets=False;Encrypt=true;TrustServerCertificate=False;Connection Timeout=30;

By assigning the output of the command to a variable named "connstring" using the syntax connstring=$(az sql db show-connection-string --name $database --server $server --client [ado.net](http://ado.net) --output tsv), the connection string can be referenced later in the script.

Add your Credentials to Connstring

After getting the connection string using the command az sql db show-connection-string, the resulting string will contain placeholders for the user ID and password, indicated by <username> and <password> respectively. We need to replace these placeholders with our actual login credentials to be able to connect to the database. Therefore, we need to add our credentials to the connection string to establish a successful connection to the Azure SQL Database.

To do this, run the below;



connstring=${connstring//<username>/$login}
connstring=${connstring//<password>/$password}

Assign the Connection String to an App Setting in the Web App

Application settings are configuration values that are stored within the App Service, and can be accessed by the application code at runtime. These settings can include things like database connection strings, API keys, and other configuration values specific to your application.

Assigning the connection string to an app setting in the web app makes it easy to manage and update the connection string without having to modify the code of the web app. By storing the connection string as an app setting, the web app can access it at runtime and establish a connection to the database without exposing sensitive information like usernames and passwords in the code. This also makes it easier to switch between different databases or servers, as the connection string can be updated in one central location (the app settings) rather than having to update it in multiple places in the code.

To do this, run the below command:



az webapp config appsettings set --name $webapp --resource-group $resourceGroup --settings "SQLSRV_CONNSTR=$connstring"

Create a storage account

A storage account in Azure is a secure and scalable cloud-based storage solution that allows users to store and access various types of data such as files, blobs, queues, tables, and disks, among others. Connecting an App Service app to a storage account allows the app to access and store data in the storage account, which can be used to store unstructured data such as files, images, videos, and other data that the app may require.

To create a storage account, run the below command;



export storage="favourdemostore"

echo "Creating $storage"
az storage account create --name $storage --resource-group $resourceGroup --location "$location" --sku Standard_LRS

--sku Standard_LRS is a parameter used in the Azure CLI command to set the pricing tier and replication scheme of the Azure Storage account being created. In this case, Standard_LRS represents the Standard pricing tier and the Locally Redundant Storage (LRS) replication scheme.

NB: The storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only.

Retrieve the storage account connection string

The connection string here is a string that contains all the necessary information to connect to a storage account. This includes details like the storage account name, access key, and endpoint. The connection string allows you to access and manage the data stored in the storage account.

To connect an app service app to a storage account in Azure, we need to obtain the storage account connection string. This connection string is crucial because it enables the app to access and manipulate the data stored in the storage account, such as uploading and downloading files, accessing and modifying blobs, and more. Without the connection string, the app will not be able to communicate with the storage account and perform these actions.

To retrieve the connection string, run the below command:



connstr=$(az storage account show-connection-string --name $storage --resource-group $resourceGroup --query connectionString --output tsv)

Assign the connection string to an App setting in the Web app

Now, we will set the connection string as a value for an App Setting in the configuration of a Web app deployed on Azure. This allows the Web app to access the storage account, as it can now retrieve the connection string from its configuration settings.

Assigning the connection string to an App Setting ensures that the connection string is securely stored, and not exposed in the application code. It also provides an easy way to update the connection string if needed, as it can be modified in the Web app configuration without having to redeploy the application code.

To do this, run the below command:



az webapp config appsettings set --name $webapp --resource-group $resourceGroup --settings "STORAGE_CONNSTR=$connstr"

After carrying out the above processes, check the configuration settings under the Web App in App Services, you should see the applications settings (connection strings) we have configured.

And that is how you connect an App Service to an Azure SQL Database and a Storage Account.

Resource Clean up

It is important to clean up resources that were created to avoid unnecessary charges after tests. Fortunately, all resources created during this process are tied to the same Resource Group, so they can be terminated with a single command. This makes cleanup quick and easy.

To perform the clean up, run the below command:



az group delete --name $resourceGroup

Conclusion

This article provided a comprehensive guide on how to connect an App Service app to both an SQL Database and a Storage Account. SQL Databases are essential for storing structured data such as user information, while Storage Accounts are used for storing unstructured data like media files or logs.

By following the steps outlined in this article, you can easily establish a connection between your App Service app and these services, allowing your app to access and manipulate data in a secure and efficient manner.

Using the Linux Free Command With Examples

Daniel Favour — Tue, 04 Apr 2023 04:30:29 +0000

I wrote this article originally for Turing

Regular memory checks are crucial to maintaining the performance of your Linux system. Knowing your system's memory can also aid in debugging and preventing poor application response times while running memory-intensive programs.

There are several ways to view the RAM on your Linux system using the command line, but for the scope of this article, we will focus on the 'free' command.

In this article, we will discuss what the Linux free command is, how it works, and the benefits of the command.

What is the `free` command?

The free command is a Linux command that allows you to check for memory RAM on your system or to check the memory statics of the Linux operating system.

To view your system's memory statics, run the free command in your terminal as shown below:

user@ubuntu:~$ free

          total    used      free   shared   buff/cache  available
Mem:    8029356  794336   6297928   183384       937092    6816804
Swap:         0       0         0

Understanding the output

The output of the free command provides various metrics related to system memory, including total, used, free, shared, buff/cache, and available. To gain a comprehensive understanding of these fields, it is important to examine each one individually. To begin, we'll focus on the 'Mem' metric, which represents the total amount of physical RAM installed on the system.

Examining the `Mem` metrics and its fields

The Mem metric, as displayed in the output of the free command, serves as a measurement of the physical RAM (Random Access Memory) installed on a computer system. It offers a comprehensive view of the total amount of installed RAM and its current usage by running programs and processes. By monitoring this metric, it is possible to detect any potential memory-related issues within the system.

The Mem metric also includes several fields that give an overview of the system's memory usage, such as:

Total: This is the total amount of physical RAM on your system.
Used: This shows the amount of memory that has been used up or amount of RAM that is currently being utilized by running programs and processes.
Free: This is the amount of physical memory that is not currently being used by any running processes and is ready to be allocated to new processes.
Shared: This displays the total amount of memory used by the temporary tmpfs file system. Tmpfs is a file system that stores files in the computer's main memory (RAM) making it faster to access compared to traditional storage methods like a hard drive.
Buff/cache: This is the memory that the kernel (operating system) uses to store recently used data so that it can be accessed quickly. It is used to speed up the performance of the computer by reducing the amount of time it takes to access data from the hard drive. Think of it like a temporary storage area where the computer stores data that it might need soon, so that it doesn't have to search for it again later.
Available: This shows an estimated value of how many memory resources are still open for use. This value can fluctuate as processes start and stop, and memory is freed up and allocated. So, while it may not actively be used by a process at the moment, it is still available to be allocated to a process if needed.

Free and available memory can be tricky to understand. Think of free memory as empty rooms in your house that are ready to be occupied, representing the amount of physical memory that is not currently being used by any running processes and is ready to be allocated to new processes. You can also think of available memory as the total number of rooms that can be occupied including the empty ones and the ones in use for caching and buffering.

An example of caching would be storing items in a storage room that you frequently use, so that you can easily access them when you need them. Buffering would be like having a guest room ready to be used in case you have unexpected visitors. Both the storage room and guest room are being used, but they are still considered "available" because they can be used for their intended purpose if necessary.

Examining the `swap` metrics and its fields

The swap metric shows the amount of swap space that is currently being used and the amount of swap space that is available for use.

Swap, also known as virtual memory, is a mechanism that enables computer systems to use extra memory by creating a file or partition on a storage volume. This serves as a backup option when the system's physical RAM is full and can't accommodate new processes. The operating system transfers data from RAM to the swap space, allowing the system to continue running smoothly.

The swap metric also includes several fields that give an overview of the system's memory usage, such as:

total: The size of the swap partition or swap file
used: The amount of swap space in use
free: The remaining (unused) swap space

Referring back to the previous output gotten when the free command was run, the output fields for swap resulted as zero (0). This entails that there is no swap space made available on the system.

Free command options

The free command can be tailored to show memory usage in any desired format.

To get the memory information in bytes, add the -b option to the command

user@ubuntu:~$ free -b

             total         used         free      sharedbuff/cache     
  available
Mem:    8222060544    700334080   6214823936   188076032     1306902528    7087677440
Swap:         0       0         0

To get the memory information in kilobytes, add the -k option to the command.

When the free command is used alone, as was demonstrated earlier, without any option, the default displayed of the memory is in kilobytes.

user@ubuntu:~$ free -k

To view the memory information in megabytes, add the -m option to the command

user@ubuntu:~$ free -m

          total    used    free   shared    buff/cache   available
Mem:       7841     668    5925      179          1247        6759
Swap:         0       0         0

To view the memory information in gigabytes, add the -g option to the command

user@ubuntu:~$ free -g

          total    used    free   shared    buff/cache   available
Mem:          7       0       5        0             1           6
Swap:         0       0       0

To view the memory information in human-readeable format, add the -h option to the command

user@ubuntu:~$ free -h

         total     used    free   shared   buff/cache   available
Mem:     7.7Gi    675Mi   5.8Gi    179Mi        1.2Gi       6.6Gi
Swap:       0B       0B      0B

Benefits of the `free` command in Linux

The free command is an invaluable tool for managing and monitoring memory usage. In this section, we will delve into the benefits of utilizing this command.

Display current memory usage
Running the free command without any arguments will display the current amount of used and available memory on the system, as well as the amount of memory being used for system buffers and disk cache. This helps you to know processes eating up your systems storage.
Monitor memory usage over time
By using the free command in combination with the watch command, you can display the current memory usage at regular intervals. For example, "watch -n 5 free -m" will display the current memory usage every 5 seconds. This can be useful for identifying patterns in memory usage over time.
Identify memory leaks
If the "used" column of the free command output is consistently high, it may indicate a memory leak in one of the running programs. By running the command periodically and checking the used memory, you can identify the process that's causing the leak.
Check for high buffer/cache usage
If the "buffers" and "cached" columns are consistently high, it may indicate that the system is using a lot of memory for caching. While this is generally not a problem, it can cause slow performance if the system is low on memory.
Display memory usage in different units
By using the -m or -g options, you can display the memory usage in megabytes or gigabytes, respectively.
Comparison with other command-line tools
The free command can be used in conjunction with other command-line tools like top, htop, and vmstat, to provide a more complete picture of the system's memory usage and performance.

Conclusion

In this article, we discussed the free command, how it works and its benefits.

The free command in Linux is a useful tool for monitoring the system's memory usage as well as a valuable tool for managing and optimizing the performance of Linux systems. Understanding the output of the free command can help administrators and users identify potential memory bottlenecks and troubleshoot performance issues.

OpenTelemetry vs Datadog - Choosing between OpenTelemetry and Datadog

Daniel Favour — Mon, 27 Mar 2023 15:58:25 +0000

OpenTelemetry and DataDog are both used for monitoring applications. While OpenTelemetry is an open source observability framework, DataDog is a cloud-monitoring SaaS service. OpenTelemetry is a collection of tools, APIs, and SDKs that help generate and collect telemetry data (logs, metrics, and traces).

OpenTelemetry does not provide a storage and visualization layer, while DataDog does. If you’re using OpenTelemetry, you need an observability backend like SigNoz or DataDog to visualize and store the collected telemetry data.

So why do you need to use OpenTelemetry at all? DataDog provides agents to instrument applications and can be used as an end-to-end solution. But more and more companies are moving to OpenTelemetry for their observability setup. There are many reasons why companies are moving to OpenTelemetry.

Why use OpenTelemetry?

OpenTelemetry is quietly becoming the world standard for instrumenting cloud-native applications. Here are some reasons why people prefer OpenTelemetry over native vendor agents:

Part of the CNCF landscape OpenTelemetry is part of the Cloud Native Computing Foundation, so it would work well with other tools in the CNCF landscape. It is the second most active project after Kubernetes.
No vendor lock-in If you use OpenTelemetry, you can avoid vendor lock-ins with SaaS services. The data collected by OpenTelemetry can be sent to multiple backends. Most observability vendors support the OTLP data format.
Future-proof instrumentation OpenTelemetry has a wide community working on it to support the instrumentation of a wide range of libraries, frameworks, and languages. If you use an instrumentation SDK from a vendor, you are susceptible to the vendor’s support of emerging technologies.
Company knowledge base and easy onboarding Using OpenTelemetry, you can have a standard observability setup in place at your company. Over time, the knowledge base will improve, and it will be easier to onboard new members of the observability team. In case the team decides to switch the vendor, it is easy to make configuration changes for the backend.

In an observability stack, the instrumentation layer is most tightly coupled with your application as it involves code changes. Using OpenTelemetry, you can have peace of mind about having standardized observability set up in place.

Key Differences between OpenTelemetry and DataDog

Let us explore the key differences between OpenTelemetry and Datadog.

Data Collection

OpenTelemetry provides language-specific client libraries and SDKs for instrumenting applications, services, and infrastructure. The OpenTelemetry client libraries can be used to collect telemetry data from applications written in various programming languages, and they provide a vendor-neutral, standardized way to collect and export telemetry data.

On the other hand, Datadog relies on its own agent to collect data from applications, infrastructure, and other services. The Datadog agent can be installed on hosts, containers, and other environments to collect metrics, traces, and logs.

Customization and flexibility

OpenTelemetry is an open-source framework, meaning developers can access and customize the source code to meet their specific needs. This makes it easier for developers to integrate OpenTelemetry into their existing systems and workflows.

In contrast, Datadog is a closed-source platform with limited customization options. While it offers a wide range of integrations with other tools and services, it can be more challenging for developers to modify and adapt Datadog to their specific needs. This can be a limiting factor for teams that require a high degree of flexibility and customization in their observability practices.

Data Storage

OpenTelemetry does not provide any data storage capabilities on its own. Instead, you need to choose an observability backend to store and analyze the telemetry data collected by OpenTelemetry.

In contrast, Datadog provides a comprehensive cloud-based platform that includes built-in data storage capabilities. This means you can use Datadog for data collection and storage, eliminating the need for a separate observability backend.

Integration with other tools

One of the key advantages of OpenTelemetry is its vendor-neutral approach, which allows users to send data to any observability backend of their choice. This provides users with greater flexibility and the ability to customize their observability stack according to their needs.

On the other hand, DataDog is a closed SaaS platform that provides its own set of APIs and SDKs to collect and analyze telemetry data. While it offers integrations with various third-party tools and services, it is not as flexible as OpenTelemetry.

Community

OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project, which means it is an open-source project with a thriving community of contributors and users. Community support means that developers can access a wealth of resources, including documentation, tutorials, and support forums, to help them adopt and use OpenTelemetry effectively.

On the other hand, while DataDog offers support to its users, most of it is paid support. This means that developers may have limited resources available to them if they encounter issues or need help using DataDog. However, DataDog offers various paid support options, including phone and email support, as well as a knowledge base and community forums.

Now that we have discussed the differences, it’s time to answer what to choose between OpenTelemetry and DataDog. DataDog also provides support for OpenTelemetry. But the extent of this support is debatable. There has been some recent controversy regarding DataDog’s support of OpenTelemetry codebase.

Since OpenTelemetry does not provide a backend, the question really is to choose between an OpenTelemetry native APM and DataDog.

Choosing between an OpenTelemetry APM and DataDog

An OpenTelemetry native APM support OTLP data format natively and treats OTel data format as its primary format for ingestion. SigNoz is an open source observability platform that supports OpenTelemetry natively. Its architecture involves OpenTelemetry client libraries and OpenTelemetry collectors to generate and collect telemetry data.

Some of the key reasons to choose SigNoz are:

SigNoz is open source and supports OTLP as its primary data format.
Metrics, Traces, and Logs in a single pane
Correlation across different signals
Powerful aggregation and aggregation capabilities on high cardinality data
It can be run within your own cloud

Getting started with SigNoz

SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main <https://github.com/SigNoz/signoz.git>
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

You can check out SigNoz GitHub repo here:

Complete Guide on Docker Logs [All access methods included]

Daniel Favour — Fri, 10 Feb 2023 01:44:49 +0000

Docker logs play a critical role in the management and maintenance of containerized applications. They provide valuable information about the performance and behavior of containers, allowing developers and administrators to troubleshoot issues, monitor resource usage, and optimize application performance. By capturing and analyzing log data, organizations can improve the reliability, security, and efficiency of their containerized environments.

In this article, we will explore the concept of Docker logs, the types, and how they can be accessed and effectively managed for optimal use in a containerized environment.

Before diving into what Docker logs are, we need to first understand the concept of Docker and containers.

A brief overview of Docker

Docker is a cutting-edge platform for building, distributing, and operating distributed applications. With this technology, developers are able to package their applications and dependencies into containers. These containers can run on any system equipped with a Docker engine.

Containers are lightweight and portable software packages that contain everything needed to run the application, including the code, runtime, system tools, libraries, and settings. This containerization process allows for seamless deployment and scaling of applications across various environments, leading to consistency and reproducibility in the development process.

Throughout the development of an application, different components might require different operating systems or configurations. By using Docker, developers can create containers for each component and specify necessary dependencies and configurations. This way, the application can function consistently across different environments, making it easier to test, debug, and deploy. Additionally, since containers are isolated from one another, changes made to a single container will not affect other components or the host operating system. This is the basic concept behind how Docker and containers work.

Now let us understand what docker logs are.

What are Docker Logs?

Docker logs refer to the records of events and messages generated by a Docker container or Docker engine. These logs provide insight into the activities and operations of a container, including its start and stop events, output messages, and error messages.

The collection and analysis of Docker logs play an essential role in monitoring, troubleshooting, and maintaining the stability and performance of Docker containers.

Types of Docker Logs

In this section, we will delve into the various types of Docker logs that are generated as a result of container activity. They are:

Container Logs

Container logs are records of the standard output and error streams generated by a containerized application. They contain any messages or errors produced by the application as it runs and can be used for troubleshooting and monitoring purposes.

Logs can be collected through various methods and viewed using tools such as the docker logs command or a centralized logging platform like SigNoz. These logs are essential in ensuring the proper functioning and performance of the application.

Daemon Logs

Daemon logs are records of events and messages generated by background processes known as Daemons. These processes perform system-level tasks, such as managing network connections and services or executing scheduled tasks. These logs contain information about the daemon's activities, including status updates, error messages, and performance metrics, and are useful for debugging, monitoring, and auditing purposes.

The logs are typically stored in text files and can be viewed and analyzed using tools like the system's log viewer or a centralized logging platform. These logs play an essential role in ensuring the proper functioning and stability of a computer's background processes.

Methods of Accessing Docker Logs

Gaining insight into the activity and performance of your Docker containers is essential for ensuring the smooth operation of your applications. In this section, we will explore the various methods available for accessing Docker logs, which include:

Accessing Docker logs with Docker CLI

The Docker Command Line Interface (CLI) provides a means for users to interact with Docker components, including containers, images, networks, and more. To access logs generated by Docker containers, the docker logs command can be utilized.

The syntax for accessing docker logs with the CLI:

docker logs [OPTIONS] CONTAINER

where:

CONTAINER is the name or ID of the container that you want to view the logs of.
OPTIONS is an optional flag that you can use to specify the details of the logs that you want to retrieve. To explore different options available for usage, refer to Docker's official documentation.

The docker logs command retrieves and displays the logs generated by a container in the console. The logs can be viewed in real-time or after the container has stopped. By default, the command retrieves all logs produced by the container, however, the user has the option to specify a time range or limit the number of logs displayed. This provides a flexible approach to accessing and reviewing the logs of a container. The docker logs command is a valuable tool for debugging and monitoring the performance of Docker containers.

Accessing Docker Logs with Docker API

The Docker Application Programming Interface (API) enables developers to access and manage Docker components programmatically, including containers, images, networks, and more. The API also provides access to logs generated by Docker containers. A variety of options are available for retrieving logs through the API, including the ability to retrieve logs for a specific container, view logs in real time, and limit the logs to a specific time range or a number of lines. We will look at these options now.

Retrieving logs from a specific container:

We can use an API endpoint to retrieve logs from a Docker container. Here is a basic example using curl:

First, obtain the container ID using the below command;

$ docker ps

Next, retrieve the logs from the container using the API endpoint, as shown below:

$ curl --unix-socket /var/run/docker.sock http:/containers/<CONTAINER_ID>/logs?stderr=1&stdout=1

Note: You will need to replace <CONTAINER_ID> with the actual ID of the container you want to retrieve logs from.

In the above example, the stderr and stdout parameters are set to 1 to retrieve both standard output and standard error logs. If you only want to retrieve logs from one of these sources, you can set the corresponding parameter to 0.

View logs in real-time:

The following example shows how to retrieve logs in real-time using the Docker API. The follow query parameter is set to true to enable real-time logs.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&follow=true>

Limiting the logs to a specific time range or number of lines:

The following example shows how to limit the logs to a specific time range using the Docker API. The since query parameter is used to specify the start time, while the until parameter is used to specify the end time.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&since=2022-01-01T00:00:00Z&until=2022-12-31T23:59:59Z>

Similarly, the following example shows how to limit the logs to a specific number of lines using the tail query parameter.

$ curl --unix-socket /var/run/docker.sock <http://v1.40/containers/container_id/logs?stdout=true&stderr=true&tail=100>

Logging Drivers

Logging drivers are plugins in the Docker ecosystem that provide a means to redirect logs generated by Docker containers to various log storage destinations, such as local or remote files, centralized log servers, or cloud-based logging services. The logging driver is configured on the Docker daemon and determines the method used to collect and store logs from containers.

Docker has several built-in logging drivers, including JSON files (default logging driver), Syslog, Journald, and Fluentd, each with its advantages and disadvantages. You can check out more logging drivers available. Learn how to configure a Docker daemon to a logging driver from this guide.

The JSON File logging driver, for example, stores logs as JSON objects in a local file, which is useful for debugging, while the Syslog logging driver forwards logs to a remote syslog server for centralized log management. The Journald logging driver sends logs to the local system's journal, and the Fluentd logging driver forwards logs to a Fluentd log collector.

To set up Syslog as your logging driver, refer to this guide.

Collecting Docker logs with a Log Analytics tool

Log analytics tools provide a way to collect, process, and store logs generated by Docker containers. These solutions provide a number of benefits over using built-in logging drivers, including advanced log analysis, centralized log management, and more robust logging capabilities.

Log analytics tools can be used in combination with the Docker API and logging drivers to provide a complete logging solution for Docker containers. For example, logs from containers can be sent to an external log server using a logging driver, and then analyzed and visualized using a logging solution like SigNoz.

Best Practices for Managing Docker Logs

Efficient log management is key to ensuring the performance and reliability of containerized applications. In this section, we will explore various strategies for managing Docker logs. They are:

Establishing Log Retention Policies

A well-defined log retention policy is a cornerstone of effective log management. This policy outlines the length of time that logs will be stored and when they will be deleted. By establishing such a policy, organizations can ensure that logs do not consume excessive disk space and that relevant information is readily available for debugging and analysis purposes. In addition, a retention policy helps to streamline the log management process and enables organizations to retain only the data that is necessary for their specific needs.

Configuring Log Rotation

Log rotation is an essential aspect of log management that helps to organize logs and conserve disk space by periodically moving older logs to a separate file and writing new logs to the current log file. Effective log rotation configuration requires consideration of factors such as log size, frequency of generation, and retention requirements. For example, some logs may need to be kept for an extended period of time due to compliance or regulatory requirements, while others may have a shorter retention period. To configure log rotation for your Docker containers, refer to this guide.

Cleaning Logs

Maintaining an organized log infrastructure requires regular cleaning of logs to prevent disk space from being consumed by unnecessary data. This involves identifying and removing logs that have reached their established retention period and are no longer needed.

Cleaning logs also helps to ensure that logs are easily accessible and readable when they are needed for debugging and analysis. If logs are not cleaned regularly, they can become cluttered and difficult to navigate, making it difficult to quickly find the information needed to diagnose issues with containers and applications.

Archiving Logs for Long-Term Storage

Archiving logs for long-term storage is a crucial aspect of log management as it ensures that valuable information is retained for future reference. This may include logs needed for auditing, compliance, and forensic purposes. Archived logs can be stored in a separate server or a cloud storage solution to ensure they are secure and easily accessible.

It is important to follow a well-defined archiving process that includes regularly backing up logs and ensuring their availability and integrity over time. Additionally, it is advisable to have a disaster recovery plan in place to ensure that logs are not lost in case of any unexpected events.

Monitoring Logs

Proactive monitoring of logs is a key aspect in ensuring the stability and efficiency of containerized applications. Through regular inspection of logs, potential problems can be detected early, allowing for prompt resolution.

This can be accomplished by utilizing the Docker CLI, the Docker API, or leveraging external logging solutions. By implementing a consistent log monitoring process, the reliability and performance of containerized applications can be greatly improved.

Docker Logs Analysis with SigNoz

By leveraging a log analytics tool like SigNoz, organizations can benefit from advanced log management capabilities, including real-time log collection, aggregation, and analysis, as well as centralized log storage and retrieval.

SigNoz is a full-stack open-source solution for Application Performance Monitoring that streamlines the process of monitoring logs, metrics, and traces. Log management is a crucial aspect of observability, and SigNoz offers a wide range of tools to help you manage, collect, and analyze logs generated by Docker containers.

The tool leverages the power of ClickHouse, a high-performance columnar database, to store and access log data for efficient analysis. Moreover, SigNoz adopts the latest standard for instrumenting cloud-native applications, OpenTelemetry which is backed by CNCF.

The logs tab in SigNoz is packed with advanced features that streamline the process of analyzing logs. Features such as a log query builder, search across multiple fields, structured table view, and JSON view make the process of analyzing Docker logs easier and more efficient.

Log management in SigNoz

SigNoz offers real-time analysis of logs, enabling you to search, filter, and visualize them as they are generated. This can assist in identifying patterns, trends, and problems in the logs and resolving issues efficiently.

Live Tail Logging in SigNoz

With the advanced Log Query Builder, you can filter out logs quickly with a mix and match of fields.

Advanced Log Query Builder in SigNoz

Getting Started with SigNoz

SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main <https://github.com/SigNoz/signoz.git>
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

If you liked what you read, then check out our GitHub repo 👇

Conclusion

Docker logs are critical components of managing and maintaining the health of Docker containers and their applications. By leveraging the power of Docker logs, organizations can optimize their container-based infrastructure and improve their ability to troubleshoot, analyze, and monitor the performance of their applications.

It is important to have a robust and efficient log management strategy in place, as it helps to ensure that logs are captured, stored, and analyzed effectively. Adopting a dedicated log management tool, instead of relying solely on the native methods of accessing and managing Docker logs, can provide a range of advanced features and greater flexibility for analyzing and processing logs from your containers.

Log management tools like SigNoz can enhance the capability to handle logs generated by Docker containers. It offers a comprehensive and scalable approach to log analytics that can cater to specific needs and requirements, which might not be met by the native Docker logging options like logging drivers or the Docker API. For instance, advanced log parsing, filtering, or transforming functions that are not feasible using just the logging drivers or the Docker API can be carried out with SigNoz.

Configure your Docker Syslog Logging Driver | SigNoz

Cloud Storage in AWS

Daniel Favour — Tue, 31 Jan 2023 09:37:18 +0000

Cloud storage is a rapidly growing technology that provides users with the ability to store and access their data over the internet. With the increasing demand for cloud storage, the market is flooded with different types of cloud storage solutions, each with its unique set of features and benefits.

In this article, we will discuss what cloud storage is, types of data storage in the cloud, and storage services offered by AWS.

Cloud Storage Defined

Cloud storage is a cloud computing model that stores data on the Internet through a cloud computing provider who manages and operates data storage as a service. It’s delivered on demand with just-in-time capacity and costs, and eliminates buying and managing your own data storage infrastructure. This gives you agility, global scale and durability, with “anytime, anywhere” data access. - AWS

Cloud storage is a form of data storage that entails the use of remote servers, which can be accessed over the internet, rather than relying on local servers or personal computers. A cloud storage provider manages, maintains, and backs up the stored data, making it possible for users to access their data from any device connected to the internet.

This approach to data storage offers several key benefits, such as scalability, accessibility, and disaster recovery, which are attractive to individuals and organizations seeking to store large amounts of data. In addition, cloud storage can be utilized for a variety of purposes, including backups, file sharing, and big data analytics.

Types of Data Storage in the Cloud

Choosing the right type of data storage in the cloud can be a complex task, as each option is designed to serve a specific purpose. To help make informed decisions about data storage needs, it's important to understand the different types of cloud storage available. In this section, we will dive into the 3 main types of data storage used in the cloud, including:

Object storage
File storage
Block storage

Object Storage

Object storage is a data storage architecture that operates by managing data as individual objects, as opposed to the traditional file system or block-level storage approach. Object storage uses a unique identifier, known as a key, to store objects. The contents of an object store can be distributed across multiple servers, which increases availability and durability. Additionally, object storage data may be replicated across multiple data centers, providing easy access through simple web service interfaces.

Characterized by its ability to offer unlimited scalability, object storage is an efficient solution for the storage of vast amounts of unstructured data, such as images, videos, backups, and archives. The utilization of object storage results in cost-effectiveness, limitless scalability, exceptional durability, and heightened accessibility.

File Storage

File storage refers to the process of storing and organizing digital files on a computer or network-attached storage (NAS) device. This system employs a hierarchical file organization, consisting of directories, subdirectories, and individual files, each with its distinct name and the potential to contain a variety of data, including text, images, audio, and video. File storage may be either local or remote and can be accessed through either a file transfer protocol (FTP) or network file system (NFS).

File storage solutions are well-suited for the management of large content repositories, development environments, media stores, and user home directories due to their ability to preserve a folder structure and provide network access. The advantages of file storage are extensive, including organization, accessibility, security, and dependability.

Block Storage

Block storage is a type of data storage architecture that divides data into blocks, as opposed to being organized in a continuous stream of bytes, and then stored across a system that can be physically distributed to maximize efficiency.

Each block operates as a separate unit of storage, enabling independent allocation, reading, and writing. This type of storage is often utilized for holding vast quantities of structured data, including databases, virtual machines, and file systems.

The data is stored on physical storage devices, such as hard drives or solid-state drives, and is either directly connected to a server or accessed via a network. This can be accessed directly through APIs or http/https.

The primary benefit of block storage is its exceptional performance, making it an ideal solution for applications that demand quick access to large amounts of data.

Instance Stores

Instance stores, also known as ephemeral storage, are block-level storage volumes that behave like physical hard drives. They are storage disks that are physically attached to the host computer for example, an EC2 instance, and hence has the same lifespan as the instance. The data stored in an instance store persists only during the life of the associated EC2 instance.

Instance store is ideal for use cases where the data needs to be accessed frequently and quickly, such as for high-performance databases and caching workloads. However, it is important to note that if the underlying host fails, the data stored in an instance store will be lost. Whenever the instance is terminated, the data stored on it is deleted. This makes it unsuitable for long-term data storage.

Storage Services offered by AWS

Amazon Web Services (AWS) offers a variety of cloud storage options for businesses and individuals. Understanding the different types of cloud storage in AWS is important for determining the best fit for your storage needs. Here are some of the most popular types of cloud storage in AWS:

Amazon Elastic Block Store (Amazon EBS)

This contradicts the instance store because it stores data for a long time and in a situation that an Amazon EC2 instance is terminated, the data stored on it still remains accessible. It provides block-level storage volumes that work with EC2 instances.
With EBS, you can create virtual hard drives called EBS Volumes, which can be attached to your instance, they aren't tied directly to the host. When you write to an instance volume, data persists.

Amazon Simple Storage Service (Amazon S3)

Amazon S3 is an object-level storage service. It offers the ability to store and access data in the form of objects, which are organized into buckets. These objects can be anything from images, texts, videos, or binary files.

S3 is designed to provide scalable, highly durable, and highly available data storage for a vast array of use cases, such as big data analytics, backup and disaster recovery, and web and mobile applications. The service guarantees data durability through automatic replication across multiple Availability Zones within an AWS region. Additionally, it offers versioning capabilities, enabling users to retain and access multiple iterations of an object over time.

Amazon S3 features different storage classes, each designed to meet specific data retrieval and availability needs. This allows users to choose the best storage option for their specific requirements. These storage classes include:

S3 Standard:
This default storage option for Amazon S3 is optimized for high-performance, low-latency access to frequently used data. It offers exceptional durability and availability, ensuring data is stored across a minimum of three Availability Zones for optimal resiliency. This storage option is ideal for organizations that require quick and reliable access to their data.
S3 Standard-Infrequent Access (S3 Standard-IA):
Amazon S3 Infrequent Access is a good option for storing data that is accessed less frequently but requires rapid access when needed. It provides high durability by storing data across a minimum of three Availability Zones, while also offering lower storage prices and higher retrieval prices. This storage class is ideal for backups, disaster recovery files, and other objects that require long-term storage.
S3 One Zone-Infrequent Access (S3 One Zone-IA):
The S3 One Zone storage option is intended for data that is infrequently accessed and where lower availability and durability is acceptable. This option stores data in a single Availability Zone, offering a lower storage cost compared to S3 Standard-IA while still providing high durability and availability.
S3 Intelligent-Tiering:
The S3 Intelligent-Tiering storage option provides flexible data management by automatically shifting data between two storage classes based on evolving access patterns. For instance, if an object has not been accessed for 30 days, Amazon S3 moves it to the S3 Standard-Infrequent Access storage class. This storage option offers an optimized solution for data with unpredictable access patterns.
S3 Glacier:
S3 glacier is a secure, durable and extreme low cost S3 storage The S3 Glacier storage class is designed for data archiving and long-term backup. This storage option is ideal for organizations that need to store data for several years for auditing purposes and do not require rapid retrieval. It provides a cost-effective solution for preserving data over an extended period of time.
S3 Glacier Deep Archive:
Amazon S3 Glacier is a cost-effective, long-term storage solution designed specifically for data archiving and backup. With retrieval times of 12 hours or more, it is the lowest-cost object storage class for archiving. This storage class is ideal for organizations looking to store large amounts of data for extended periods, with a focus on minimizing cost and retrieval time.

Amazon Elastic File System (Amazon EFS)

Amazon Elastic File System (EFS) is a cloud-based file storage solution. It is a serverless and scalable file system. Because it is elastic, your file system automatically grows and shrinks as you add and remove files.

EFS is specifically designed to support use cases that require shared access to data, such as big data processing, content management, and web serving. It enables multiple EC2 instances to access a shared file system simultaneously, making it effortless to share data between instances.

Amazon EFS offers high levels of availability and durability, as it stores multiple redundant copies of data across multiple Availability Zones within an AWS region. This ensures that the data is always accessible, even in the case of hardware failures or other infrastructure issues.

Instance stores vs AWS Storage classes

Instance stores have distinct characteristics compared to other AWS storage services such as Amazon S3 and Amazon EBS. They are physically attached to the host computer of an EC2 instance, providing block-level storage optimized for rapid access with low latency. In comparison, S3 and EBS are network-attached storage accessed through the internet.

Instance stores lack persistence, meaning that any data stored in them will be lost if the EC2 instance terminates or its host computer fails. Conversely, data stored in S3 and EBS is automatically replicated and can be accessed persistently. The capacity of instance stores is limited to the physical storage of the host computer, while S3 and EBS offer unlimited capacity.

Instance stores are suitable for temporary storage of instance-local data that is frequently accessed, such as caching, temporary data processing, or as a scratch disk for high-performance computing applications. On the other hand, S3 and EBS are best for storing critical data that must be persistently stored and accessed over a longer period.

In summary, instance stores offer fast, low-latency storage for temporary data, and S3 and EBS offer durable, persistent storage for critical data. The choice between the two depends on the specific storage requirements of the application.

Conclusion

In conclusion, cloud storage in AWS offers a flexible and scalable solution for storing and accessing data. It provides a range of storage options, including S3, EBS, and EFS, to meet different data storage needs. It also offers features such as data management, security, and disaster recovery, making it a highly secure and reliable solution for businesses and organizations. The pay-as-you-go pricing model and on-demand access to storage resources make AWS a cost-effective choice for organizations of all sizes.

7 Open-Source Log Management Tools that you may consider in 2023

Daniel Favour — Tue, 31 Jan 2023 02:03:31 +0000

Effective log management is a fundamental aspect of maintaining and troubleshooting today's complex systems and applications. The sheer volume of data generated by various software and hardware components can make it challenging to identify and resolve issues in a timely manner.

Open-source log management tools offer a cost-efficient and customizable approach for collecting, analyzing, and visualizing log data. These tools empower administrators with the ability to swiftly discern patterns and trends within log data, thereby streamlining the diagnosis and resolution of problems.

In this article, we will take a closer look at some of the most popular open-source log management tools available and explore the features and capabilities of each tool. Whether you are a system administrator, developer, or security professional, this article will provide you with the information you need to choose the best log management solution for your needs.

Top 7 open-source log management tools

In this section, we will discuss the top 7 open-source log management tools that have been adopted by organizations. They are:

SigNoz

SigNoz is a comprehensive, open-source log management and analysis platform that offers a centralized location for the collection, storage, and analysis of log data. Designed to aid organizations in gaining valuable insights into their IT infrastructure, applications, and security, the platform offers real-time visibility, automated troubleshooting, and predictive analytics.

SigNoz supports the collection of log data from a wide range of sources, including servers, network devices, applications, and cloud services. It uses OpenTelemetry to collect and process log data. OpenTelemetry has quietly become the world standard for instrumenting cloud-native applications.

Log management in SigNoz

The platform also offers a variety of visualization options, such as charts, graphs, and maps, to aid users in gaining insights into their log data.

Live log tailing in SigNoz to keep track of logs in real-time

Furthermore, it provides automated alerting and troubleshooting features, enabling organizations to identify and resolve issues quickly.

Some key features of SigNoz are:

Log data collection and analysis
Centralized data storage
Real-time visibility
Data visualization
Alerting and troubleshooting
Support for integration with other tools and systems

You can read more about SigNoz from its documentation.

Logstash

Logstash is a powerful, open-source log management tool that is part of the Elastic Stack (previously known as the ELK stack). Logstash is capable of collecting and processing logs from a wide range of sources and can output them to a variety of destinations, including Elasticsearch, a search engine, an analytics engine, or a file.

As a log management tool, Logstash provides a pipeline for collecting, parsing, and processing log data. It ingests log data from various sources, such as files, Syslog, and network inputs, and can parse and process the data using a variety of filters and plugins.

Capable of handling high volumes of data and heavy loads while maintaining good performance, Logstash can be run as a standalone service or as a distributed system. Logstash itself does not have a built-in dashboard for viewing logs.

However, it can be used in conjunction with other tools such as SigNoz and Kibana to create and share interactive visualizations and dashboards of log data collected by Logstash. You can find docs on how to send data collected by Logstash to SigNoz here.

Search for logs with a particular indexed pattern sent from Logstash in Kibana

Some key features of Logstash are:

Log data collection from various sources
Parsing and processing of log data
High performance and scalability
Output to various destinations
Multiple platforms support
Integration with other ELK stack components
Built-in security features.

Graylog

Graylog is an open-source log management and analysis platform designed to collect, store, and analyze large volumes of log data from various sources. Utilizing a pipeline system for data collection and processing, Graylog collects data from various sources, parses, transforms, and enriches it before storing it in a database, allowing for easy searching and analysis via the Graylog web interface, which provides a wide range of visualization options.

In addition to its robust data collection and processing capabilities, Graylog also offers alerting capabilities, sending notifications when specific conditions are met such as the encounter of a particular error message. The platform also provides a RESTful API for integration with other tools and systems and can handle large volumes of log data, scaling horizontally by adding more Graylog server nodes to a cluster.

Graylog supports multiple data inputs and outputs, it can collect data from various sources such as Syslog, GELF, log files, and Windows Event Log, and it can output data to other systems such as Elasticsearch, Apache Kafka, and more.

Search configuration in Graylog

Some key features of Graylog are;

Log data collection and analysis
Data processing pipeline
Search and analysis capabilities
Alerting and notifications
RESTful API
Scalability
Multi-data inputs and outputs

FluentD

Fluentd is a powerful log management tool that offers organizations the flexibility and scalability required to handle large volumes of log data from a variety of sources and transport it to various destinations. Utilizing a flexible and modular architecture, Fluentd allows users to easily add new input and output plugins to integrate with a wide range of systems and applications. It supports a wide range of data sources and destinations, including databases, message queues, and data stores.

Fluentd has a built-in buffering mechanism that enables it to handle temporary failures in the output destination, ensuring that data is not lost. Users can filter, buffer and format log data using the built-in filters and parsers before sending it to the output destinations.

Logs overview in FluentD

Some key features of FluentD are:

Log data collection and transport
Flexible and modular architecture
Input and output plugins
Variety of data sources and destinations
Built-in security features
Filtering, buffering, and formatting of log data

Syslog-ng

Syslog-ng is an open-source log management tool designed for the collection, parsing, and transportation of log data from various sources to a wide range of destinations. Known for its flexibility and wide range of features and capabilities, such as filtering, parsing, rewriting, and alerting, Syslog-ng is a widely used tool in Linux and Unix-based systems for log management.

Syslog-ng is capable of collecting log data from a diverse array of sources, including Syslog, GELF, log files, and Windows Event Log. It can parse, filter, and rewrite log messages before forwarding them to other systems, such as databases, message queues, and data stores.

The tool offers a large number of built-in destination and source drivers for popular data destinations, including Elasticsearch, Apache Kafka, and more, allowing for easy integration with other systems. Additionally, Syslog-ng includes a built-in buffering mechanism that enables it to handle temporary failures in the output destination and ensures that data is not lost.

Collecting and viewing log files in Syslog ng

Some key features of Syslog-ng are;

Log data collection and transport
Flexible filtering and parsing capabilities
Built-in source and destination drivers
A large number of input and output plugins
Built-in buffering mechanism
Support for various log formats and protocols.

Logwatch

Logwatch is an open-source log analysis tool designed to automatically parse and analyze log files from various services and applications running on Linux or Unix-based systems. It presents a summary of the log data, including system activity, security events, and potential issues in a detailed, easy-to-read format, making it simple to identify and troubleshoot problems.

Logwatch utilizes a series of customizable filter scripts, written in Perl, to parse log data from various services and applications, such as Apache, SSH, and Syslog. These scripts can be modified to meet the specific needs of an organization. Additionally, Logwatch offers various options for controlling the output, including the ability to filter out specific log entries, adjust the level of detail, and send the output to a specific email address or file.

Logwatch is typically run on a daily basis and can be scheduled to run automatically using cron or another scheduling tool. It also offers a command-line interface, which allows users to run Logwatch and view the output directly on the command line.

Some key features of Logwatch:

Log data analysis
Customizable filter scripts
Detailed and easy-to-read output
Output filtering and control
Email and file output
Scheduled and command-line execution
Summary of system activity, security events, and potential problems
Ability to filter out specific log entries

Apache Flume

Apache Flume is an open-source log management tool designed to efficiently collect, aggregate, and transport large volumes of log data from various sources to a centralized data store, such as HDFS or Hbase. It excels in handling large amounts of log data in real-time and is highly scalable, able to handle the load from multiple servers, network devices, and applications.

In terms of log management, Apache Flume offers features such as data collection, transportation, aggregation, fault tolerance, and delivery guarantee. It also boasts a plugin-based architecture, allowing organizations to easily add new sources and sinks as needed, facilitating integration with other log management tools and systems, and enabling the addition of new log sources. Additionally, it is straightforward to set up and configure and provides a web-based interface for monitoring and managing log data.

Some key features of Apache Flume are;

Log data collection and transportation
Data aggregation
Centralized data storage
Fault-tolerance and delivery guarantee
Scalable
Plugin-based architecture
Web-based interface
Real-time log data processing
Integration with other log management tools and systems

Choosing the right Log Management Tool

When choosing a log management tool, it is important to consider factors such as data collection, ingestion, and processing capabilities. You should consider scalability, security features, integration with other tools and systems, user interface, and visualization options. Based on these factors, you can choose a log management tool that fit your use cases.

If you are looking for an open source log management tool that solves most of your monitoring needs, then SigNoz can be a good choice. It provides logs, metrics, and traces under a single pane of glass with an intelligent correlation between the three types of telemetry signals.

SigNoz is open-source and cost-effective for organizations. It is built to support OpenTelemetry natively. With the flexibility and scalability of OpenTelemetry and SigNoz, organizations can monitor and analyze large volumes of log data in real-time, making it an ideal solution for log management.

Getting started with SigNoz

SigNoz can be installed on macOS or Linux computers in just three steps by using a simple install script.

The install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install Docker Engine before running the install script.

git clone -b main <https://github.com/SigNoz/signoz.git>
cd signoz/deploy/
./install.sh

You can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.

You can check out its GitHub repo here:

OpenTelemetry Logs - A Complete Introduction & Implementation | SigNoz

Forem: Daniel Favour

Automating User Management on Linux using Bash Script

Overview of the Bash Script Functionality

Prerequisites

Setting Up the User Data File

Writing the Bash script

Create the Script file

Add the Shebang Line

Check Root Privileges

Check that the Input File is Passed as an Argument

Assign Variables

Log Messages

Ensure the /var/secure Directory Exists

Generate the User Passwords

Process the Input File

End the Loop

Run the Script

Verify the Script Executed Tasks Successfully

Conclusion

Setting up a LAMP Stack application

Application Architecture

Prerequisites

Application Setup

Project Structure

Setup the Linux Environment

Installation

Setup the Application

Creating a Virtual Host

Setup Mysql database

Fix binding settings for mysql

Conclusion

Container Monitoring: Ensuring Application Performance and Health

Environment Setup

Architecture Diagram

Project Structure

cAdvisor

Prometheus

Environment Variables

Docker Compose

Run the Containers

Containerizing a LAMP Stack Application

Container Architecture diagram

Container Communication Diagram

Approach to Running the Containers

Project Structure

Without Docker Compose (Makefile Approach)

Create PHP directory

Environment Variables

Write the Dockerfile

Create the Environment Variables File

Write the Makefile

Create the MySQL database

With Docker-Compose

Environment Variables

Docker Compose

Container Health Checks

Beginners guide to GitOps and Flux

What is GitOps?

GitOps principles

1. The entire system is described declaratively

2. The desired system state is versioned in Git

3. Approved changes can be automatically applied to the system

4. Software agents ensure correctness and reconcile the system to match the desired state

The GitOps workflow

The Solution

How it works

What is Flux?

Flux Components

Flux Controllers in the GitOps Toolkit

Benefits of Flux

1. Automation

2. Scalability

3. Rollback

4. Integration

5. GitOps

6. Customization

Conclusion

Deploy your Application to Azure Kubernetes Service

Prerequisites

Application Overview

Ensure the `/var/secure` Directory Exists

What is the `free` command?

Examining the `Mem` metrics and its fields

Examining the `swap` metrics and its fields

Benefits of the `free` command in Linux