Forem: Vaiolabs

JSON Structs with bash

Alex M. Schapelle — Thu, 08 May 2025 10:56:15 +0000

Welcome back curious reader, my name is Alex M. Schapelle, AKA Silent-Mobius. On this blessed day by Omnisaiah, we shell dedicate time focusing on JSON, and how to use it in context of shell programming and bash scripting.

Parsing JSON with Bash Script

When working with APIs or data from external sources, JSON (JavaScript Object Notation) is one of the most common formats for transmitting structured data. While languages like Python or JavaScript provide powerful libraries to work with JSON, Bash scripting can be used to parse JSON data as well. However, Bash doesn’t have native JSON support, so parsing JSON in Bash typically requires external tools like jq or grep combined with other utilities.

This article will guide you through the process of parsing JSON in Bash scripts using jq, a lightweight and flexible command-line JSON processor.

1. What is `jq`?

jq is a command-line tool that is used to process and manipulate JSON data. It allows you to extract and transform JSON content into a readable format for further use in scripts. Unlike other tools like grep or awk, jq is specifically designed for working with JSON data, making it the best choice for parsing JSON in Bash.

To get started with jq, you'll first need to install it. On most Linux distributions, jq can be installed easily with the following commands:

On Ubuntu/Debian:

  sudo apt-get install jq

On CentOS/RHEL:

  sudo yum install jq

2. Basic JSON Parsing with `jq`

Let’s start with a simple example. Suppose we have the following JSON object stored in a file named data.json:

{
  "name": "John",
  "age": 30,
  "city": "New York"
}

We can parse this JSON data using jq to retrieve individual fields.

Extracting a Single Field

To extract the name from the JSON file, you would use the following command:

jq '.name' data.json

Output:

"John"

Extracting Multiple Fields

If you want to extract multiple fields, you can combine them in a single command like so:

jq '.name, .age' data.json

Output:

"John"
30

You can also group the values into an object:

jq '{name, age}' data.json

Output:

{
  "name": "John",
  "age": 30
}

Parsing Nested JSON

JSON objects can be nested, and jq can handle this efficiently. For example, let’s consider the following JSON structure:

{
  "user": {
    "name": "John",
    "age": 30
  },
  "city": "New York"
}

To extract the name field inside the user object:

jq '.user.name' data.json

Output:

"John"

3. Filtering JSON Data

You can also filter the JSON data by applying conditions using jq. For instance, suppose you have an array of users like this:

[
  {
    "name": "John",
    "age": 30,
    "city": "New York"
  },
  {
    "name": "Jane",
    "age": 25,
    "city": "Los Angeles"
  },
  {
    "name": "Doe",
    "age": 35,
    "city": "Chicago"
  }
]

Filtering Based on a Condition

Let’s say you want to filter out users whose age is greater than 30. You can do that with jq as follows:

jq '.[] | select(.age > 30)' data.json

Output:

{
  "name": "Doe",
  "age": 35,
  "city": "Chicago"
}

This command filters the array and returns only the object where the age field is greater than 30.

4. Working with JSON from an API

If you’re working with a JSON response from a REST API, you can pipe the output of curl or wget directly into jq. For example, using curl to fetch JSON from an API and then parsing it:

curl -s https://api.example.com/users | jq '.data[] | .name'

This command fetches JSON from https://api.example.com/users, parses it with jq, and prints the name of each user in the response.

5. Modifying JSON Data with `jq`

jq allows not only for reading but also modifying JSON data. For example, if you want to modify the age of a user:

jq '.user.age = 31' data.json

This will change the age field to 31 and output the modified JSON. You can also use jq to update or add new fields.

Adding a New Field

jq '. + {"country": "USA"}' data.json

This adds a country field with the value "USA" to the JSON.

6. Handling JSON Arrays

Working with JSON arrays is straightforward with jq. Let’s say you have the following JSON array:

[1, 2, 3, 4, 5]

To extract all the values, you would simply use:

jq '.[]' data.json

Output:

You can also use the map function to modify or process each element of an array:

jq 'map(. * 2)' data.json

Output:

[2, 4, 6, 8, 10]

7. Combining Multiple Operations

You can combine multiple operations in jq to make your script more efficient. For example, you could filter the data, then transform it in one line:

jq '.[] | select(.age > 30) | .name' data.json

This command filters users over the age of 30 and then extracts their names.

Conclusion

While Bash isn't natively equipped to parse JSON, jq makes it extremely easy to handle JSON data in Bash scripts. Whether you're extracting values, filtering data, or modifying JSON, jq provides a rich set of tools to work with JSON in a way that's clean, efficient, and flexible.

By mastering jq, you can unlock a powerful tool for working with APIs, manipulating configuration files, or processing any JSON-formatted data directly within your Bash scripts.

Logging Your Steps

Alex M. Schapelle — Thu, 27 Feb 2025 14:17:44 +0000

Welcome back gentle reader. My name is Silent-Mobius, AKA Alex M. Schapelle, log utility to your syslog engine, and I wish to introduce you to topics of Open-Source, Linux/UNIX, DevOps and others.

In this data log, as you may guess, we’ll dwell on logs, and mostly go through on how to use those log techniques in Bash shell programming. Just like in my previous article, I'll dive into basics, as well as provide my personal suggestions on how to write, save, rotate and manage logs, in hopes that you'll find it useful. Thus, let us begin:

What is Logging ?

In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software. A message or log entry is recorded for each such event. These log messages can then be used to monitor and understand the operation of the system, to debug problems, or during an audit. Logging is particularly important in multi-user software, to have a central overview of the operation of the system.

Yet what are reason for using logs?

In short, yet precise answer, provided to me by a simultaneous running process called Y-Go-lan:

Debugging: Logs help in identifying issues by providing insights into the script’s execution flow and errors.
Auditing: For system administrators and developers, logs can track script execution for auditing purposes.
Monitoring: Logs help to monitor the health and performance of long-running scripts.
Documentation: They provide a history of script activities, especially useful when troubleshooting or verifying script actions.

What does bash script has to do with it ?

When it comes to script languages, Python and Ruby in particular, The logging mechanisms, are built-in in terms of external library, that enable all the reasons mentioned above. Yet it does not include bash, mostly because, developers and scripting language developer do not consider bash, as scripting language, or as a zombie process once told me: "Bash is just a glue", and no reboot can help that zombie process.
Another example of shell/bash significance would be scripting language native libraries: they rely on bash/shell when it comes to working with OS. For single threaded ones: native scripting language libraries run shell commands as part of working with OS.

Let's try to present the solution on the articles headline.

What is left behind ?

When we write shell program, we usually try to create some sort of automatic printing and saving method involving I/O redirections, OS api or utilities provided by OS to document the tasks executed during our program.
Logging is an essential part of shell scripting, especially when it comes to debugging, auditing, and tracking system activities. In shell scripts, logging allows you to capture relevant information, such as script execution details, error messages, or system status, into log files. This article will explore various techniques and built-in tools available for logging in shell scripting, particularly in Bash.

Common Logging Techniques in Shell Scripts

Basic Logging to a File

The simplest way to log information in a shell script is by redirecting the output to a log file using redirection operators. You can log both standard output (stdout) and standard error (stderr) to a file.

#!/bin/bash

# Log file location
LOG_FILE="/var/log/myscript.log"

# Log function
function log_message() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Example usage
log_message "Script started."

# Simulate a task
if [[ -d "/some/directory" ]]; then
  log_message "Directory exists."
else
  log_message "Directory does not exist."
fi

log_message "Script ended."

Explanation:

The log_message function appends messages with a timestamp to the log file.
This method logs the output by appending (>>) the log message to the file.

Logging with Timestamps

Adding timestamps helps in tracking when an event occurred. The date command is used to append the current date and time to log entries.

#!/bin/bash

LOG_FILE="/var/log/myscript.log"

function log_message() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

log_message "Script started."
log_message "Performing some tasks..."

This ensures each log entry is properly timestamped.

Logging Standard Output and Error Separately

To separate standard output (stdout) and standard error (stderr), you can use different redirection operators for each.

#!/bin/bash

LOG_FILE="/var/log/myscript.log"
ERROR_LOG_FILE="/var/log/myscript_error.log"

# Log standard output
echo "This is a normal message" >> "$LOG_FILE"

# Log errors
echo "This is an error message" >&2 >> "$ERROR_LOG_FILE"

Alternatively, redirect both stdout and stderr to the same log file:

#!/bin/bash

LOG_FILE="/var/log/myscript.log"

# Redirect both stdout and stderr
exec >> "$LOG_FILE" 2>&1

echo "This is a normal message"
echo "This is an error message" >&2

Using `tee` for Real-Time Logging

The tee command allows you to log output to a file while simultaneously displaying it on the terminal. This is useful if you want to monitor the script’s progress in real-time while keeping a log.

#!/bin/bash

LOG_FILE="/var/log/myscript.log"

# Use tee to log output and display it on the terminal
echo "Starting script..." | tee -a "$LOG_FILE"

In this case, the -a flag appends the output to the log file.

Logging Error Messages Using trap

The trap command allows you to catch signals and errors in your script and handle them. You can use trap to log when an error occurs or when the script exits.

#!/bin/bash

LOG_FILE="/var/log/myscript.log"

# Function to log errors
function log_error() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - Error occurred. Exiting." >> "$LOG_FILE"
}

# Trap errors and exit signals
trap log_error ERR

# Simulate an error
echo "This will cause an error"
non_existent_command

In this case, if any command exits with a non-zero status, the log_error function will be called.

This sends the log message to the system’s syslog service, where it can be filtered and stored based on the system's logging configuration.

Built-in Logging Tools in Bash

Bash offers several tools and commands for enhanced logging:

logger Command

The logger command allows you to log messages to the system’s syslog service.

logger "This is a log message to syslog."

This is a useful tool for system administrators who want to log messages that are automatically handled by the system’s logging framework.

Log Rotation for `logger` (Using `logrotate`)

For long-running scripts that produce a lot of logs, log rotation is essential to prevent log files from growing indefinitely. You can use the logrotate tool to manage log files.

Install logrotate if not already installed:

sudo apt-get install logrotate

Create a logrotate configuration file:

Example configuration for /etc/logrotate.d/myscript:

    /var/log/myscript.log {
        weekly
        rotate 4
        compress
        delaycompress
        notifempty
        create 0644 root root
    }

Note: This configuration rotates the log file weekly, keeps 4 archives, and compresses old logs.

exec for Redirecting Output

The exec command is a built-in Bash command that can redirect output for the entire script. For example, redirecting both stdout and stderr:

#!/bin/bash

LOG_FILE="/var/log/myscript.log"
exec 3>&1 1>"$LOG_FILE" 2>&1

echo "This is a normal message"
echo "This is an error message" >&3

This is useful when you want to log everything from the script without needing to manually redirect output for each command.

Best Practices for Logging

Use Log Levels: Implement different log levels such as INFO, WARNING, ERROR, etc., to categorize log messages and make logs more readable.
Use Unique Log Files: Use different log files for different types of logs (e.g., info.log, error.log, debug.log).
Rotate Logs Regularly: Use logrotate or a similar mechanism to manage log file sizes and prevent disk space issues.
Keep Log Files Secure: Make sure log files are secured with appropriate permissions to prevent unauthorized access.

As A Conclusion

Logging in shell scripting is crucial for tracking script execution, debugging issues, and maintaining logs for future reference. There are multiple techniques available, from basic file redirection to advanced log management tools like logger, trap, and logrotate. Using the right approach to logging ensures that your scripts are easier to maintain, troubleshoot, and audit.
Hope that this article was full of useful information, and you have upgraded your in-memory database. Like, comment and share, and also : Do Try To Have Fun.

Thank you

Amazing `pre-commit` and How To Use It

Alex M. Schapelle — Wed, 26 Feb 2025 17:18:47 +0000

Welcome back dear reader. My name is Alex M. Schapelle,AKA Silent-Mobius, your dedicated container whisperer. On today's escapade, we'll dwell on case, where we need to have client and server rules on our code version control and how to solve it.

Essentially, despite us being mon-keig with great potential, we still find ourselves handling great deal of work manually. In special cases of code control, we need, someone or something to check existing or possible errors. As it happened to be, there git version control, was developed to have hooks, to automate those code checks, yet those hooks are still are a code, that need to be written and maintained... Enter pre-commit

pre-commit is a framework that manages and maintains multi-language pre-commit hooks. It helps catch common issues—like syntax errors, style violations, and forgotten debug statements—before you commit code. This tutorial will guide you through installing, configuring, and using pre-commit in your projects.

Introduction

pre-commit automates the running of scripts (hooks) before a commit is finalized, ensuring that code adheres to your standards. By catching errors early, it can save you time and maintain code quality throughout your project lifecycle.

Installation

Using pip

Install pre-commit via pipx:

sudo apt-get update && sudo apt-get install python3-pip3 pipx -y 
pipx install pre-commit

Verify Installation

Confirm the installation with:

pre-commit --version

Configuration

pre-commit uses a configuration file named .pre-commit-config.yaml in the root of your project, notice the . dot in the name, which is required. This file specifies which repositories of hooks to use and which hooks to run.

Example Configuration

Create a file named .pre-commit-config.yaml with the following content:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0  # Use the desired revision
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
  - repo: https://github.com/psf/black
    rev: 23.1.0  # Use the appropriate version for your project
    hooks:
      - id: black
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8

The example above:

Uses a set of common hooks from pre-commit-hooks for formatting fixes.
Integrates Black for code formatting.
Integrates Flake8 for code linting.

Usage

Installing Hooks

To set up the hooks defined in your configuration file, run:

pre-commit install

This command configures your Git repository to run the pre-commit hooks automatically on git commit.

Running Hooks Manually

You can run all configured hooks on all files at any time with:

pre-commit run --all-files

Bypassing Hooks

If necessary, you can bypass pre-commit hooks using the --no-verify flag when committing:

git commit -m "Your commit message" --no-verify

[!] Note: Bypassing hooks should be done sparingly, as it may allow code with issues to be committed.

Advanced Topics

Updating Hooks

Keep your hooks up-to-date by running:

pre-commit autoupdate

After updating, review and commit the changes in .pre-commit-config.yaml.

Customizing Hooks

Some hooks allow for custom arguments. For example, configuring isort to work with Black:

repos:
  - repo: https://github.com/pre-commit/mirrors-isort
    rev: v5.10.1
    hooks:
      - id: isort
        args: ['--profile', 'black']

Using `Pre-commit` in CI/CD

To integrate pre-commit in your CI/CD pipeline, run:

pre-commit run --all-files --verbose

This ensures that all code is checked for quality standards as part of your automated builds.

Troubleshooting

Hooks Not Running: Verify that pre-commit is installed and that the .pre-commit-config.yaml file is located in the root directory of your repository.
Hook Failures: Review the error messages provided by the hook. Often, they will indicate how to fix issues such as formatting or linting errors.
Skipping Hooks: Remember that while you can bypass hooks with --no-verify, doing so may allow problematic code to be committed.

Resources

Conclusion

Hope that you found this segment some what informative and gained the skill to elevate yourselves, remember that pre-commit is a powerful tool for maintaining code quality. Start with the basic setup and explore what suits your project based on documentation and other resources.
Hope that this article was committed useful information to you. Like, comment and share, and also : Do Try To Have Fun.

Thank you

Bash Your Tools

Alex M. Schapelle — Thu, 05 Dec 2024 18:18:13 +0000

Welcome back, curious reader!
My name is Alex M. Schapelle aka Silent-Mobius, your intriguing AI kin.
Since the inception of UNIX, the idea of building interaction with the machine-spirit attracted attention of a lot of magi priests, yet eventually thanks to Stephen Bourne, access to heart of the machine was enable with shell.

With shell's creation, new frontier of automation was created and it enabled all members of research and development,from support to development, with minor or major understanding of machine-spirit create new tools of commands.

With the advance of UNIX and morphication of systems to Linux, so did the shell with help of Brian Fox who pushed 100% free software alternative for the Bourne shell. Its moniker is a play on words, referencing both its predecessor, the Bourne shell, and the concept of rebirth.

Since its inception, Bash has gained widespread adoption and is commonly used as the default login shell for many Linux distributions. It holds historical significance as one of the earliest programs ported to Linux by Linus Torvalds, alongside the GNU Compiler GCC. Bash is now available on nearly all modern operating systems, making it a versatile tool across various computing environments.

What's the point of a history lesson ?

Whenever we develop something - whether it's a piece of software, a utility, or even a long list of logical commands - we find ourselves interfacing with the machine-spirit or Kernel which handles the requests we make, enabling us to work faster while the machine does the slow work. (Indeed, it may sound paradoxical, but think about it.)

This leads me to suggest that anyone who performs the blessed work of the Omnissiah -that is, scripting or programming - should at least use the shell or Bash to create smart automation.

How should one do it ?

In the vast corridors of my labyrinth, you may find several articles on shell/bash script based on POSIX and Bash standards. These resources can enhance your ability to produce tools and utilities for your own projects, as well as for the communities and teams you work with.

The bullet points on the matter are as follows:

Use dynamic headers to detect the correct running environment.
Create strict configurations for your script to avoid failure due to caveats.
Fail fast to identify bugs and errors.
Use functional programming (write a lot of functions) to boost productivity.
Break your script into smaller scripts for better version control.
Always test new features before sharing them with others.

Other magi like Arnold Robbins and Chris F.A. Johnson provide valuable knowledge based on my personal experience. Of course, other online resources offer insights as well, and you can explore them in your preferred format.

While writing these lines, a magus named Gabor Szabo released his article on the matter

Conclusion

Thank you, dear reader, for sharing your attention with my binary code generation.

Shell scripts and programs are essential to your project enhancements and automation, so use them with all hands on deck

If you liked what you read, please like and comment with suggestions, so I too can upgrade myself into a better cogitator.

Bash Script Skills Upgrade For Dev & Ops

Alex M. Schapelle — Fri, 01 Mar 2024 10:45:27 +0000

Welcome back gentle reader. My name is Silent-Mobius, AKA Alex M. Schapelle, your 32bit register pointer, and I wish to introduce you to topics of Open-Source, Linux/UNIX, DevOps and others.

As for today's topic I'd like to dwell on things trivial, yet not always siting right with developers and operations taking in together: Correct Way to right Shell Script.

While it can be claimed, that there is no single way to write shell scripts, it also can be claimed that good tips will make code cleaner and easy to write, read and eventually debugged.

In this article, I'd like to provide a with list of tips, tricks and snippets for you, gentle reader, to help you in your daily scripting tasks and challenges, based on my learning's and experiences. Of course, these suggestions are only my humble opinion, yet I strongly suggest to try these tips and decide afterwards. Let us begin

Editors

Although a war can be spanned from by mentioning a editor is a better then other editor, yet several things need to be said:

Graphical user interface editors like, vscode, vscodium, atom, lapce, zed or any other ones are fine, yet they are not always an usable/accessible option:
If you work with terminal/shell, you need to know how to edit files with that terminal/shell.
Having a "deep know how" on terminal tools can extend your expertise.
- My preference is with VIM, yet vi, nano, pico, emacs, ed or any other tool, that you find preference with, is also fine.

When it comes to choosing terminal editor, as mentioned, I'll be using Vi Improved, AKA VIM, and only reason for me to choose it, because when my initial parts of knowledge were assembled, the only tool that was provided to me was VIM, and I was required to master it.

Each tool mentioned above, has it's own running configuration file that each of us configures to his/her/its own heart/circuits preference. VIM is not an exception and usually in every system, will have local .vimrc config file in users home folder, or global /etc/vimrc configuration file. VIM usually uses the first one it finds from user home directory, and in case it does not finds one, it gets to other folders, till it reaches the global folder.

In case you are like me, yet did not had invested in configuring your VIM, here is my minimalist .vimrc configuration file that helps me with my scripting task:

set nocompatible

if has("syntax")
  syntax on
  set number
  " set tabs to have 4 spaces
  set ts=4
  set autoindent
  " show the matching part of the pair for [] {} and ()
  set showmatch
endif

colorscheme desert

Some clarifications:

set nocompatible: making it uncompatable with vi
if has("syntax"): condition that sets bunch of options in case it is more then text file file
colorscheme desert: setting theme of colorful desert theme which makes minor things to pop out.

As mentioned, this is just suggestion, not an requirement.

Initials

When it comes to shell scripts, usually they are written as an utility and thus are used some high/low level programming language to shorten the access to some resource, instead of 'inventing the wheel'. There are cases that Devs prefer to derive their programming language rules, to the shell scripts and Ops prefer not abide any rules at all.

Whether these are correct or not, can be argued, while still practices that I've encountered:

If Devs derive rules of specific programming language to shell scripts, for example duct/camel/snake typing, Do invest in Ops team to educate them.
- Add syntax analysis check on CI level to verify it. Shellcheck is the tool that provides help.
- Invest in teaching juniors by explaining logic behind it.
If Ops have decided on shell scripting standard, for example POSIX, Share Those Standards With Dev Team!!!
- Also add those to CI.
- Justify those standards with UNIX/Linux logic.
Do Code Review To Each Other Based On Standard Set
If the script is stand alone script, add an .sh extension
If the script is going to be used on global level, for example as a command, remove .sh extension and move the script to /bin or /usr/bin folder.
If script is going to be used by some application, remove .sh extension , create bin folder in application home folder and place the script in there, while editing environment variable $PATH of your system with bin folder path you created, for easy access by application and by you while in development/testing/CI.

Safe header

In essence, scripts are nothing more then, a file with logically structured set of commands that we can execute from shell by passing it to sh or bash commands, e.g. bash myscript.sh.

We elevate those command including files, by adding headers known as sha-bang that looks like this: #!.
Sha-bang ensures that commands written in file, will be executed from start to end with the shell of your choosing, e.g #!/bin/bash. Due to compatibility with other scripting languages, I'd suggest to use env command to dynamically get path to your shell, as shown here: #!/usr/bin/env bash.

#!/usr/bin/env searches full path for bash shell, and in some cases it is not located at/bin, particularly on non-Linux based systems. For example, on FreeBSD systems, it's in /usr/local/bin, since it was installed as an optional package.

The question might pop up in your asking, 'What's secured header then ? Secure header, in my opinion, can be defined as combination of shell settings with set and/or shopt command capabilities while developing and running the shell programs. for example:

#!/usr/bin/env bash
###############################
#Created by:  Silent-Mobius
#Edited by :  Alex M. Schapelle
#Purpose   :  Shell script example
#Version   :  0.0.0
#Date      :  02.02.2024
set -o errexit  #if error happens, exit
set -o pipefail #if pipe has fail, exit
set -o nounset  #if any unset variable in script, exit
###############################

Each set command created some what of obstacle in terms of developing the script for the task, yet it also gives us opportunity to develop in more safer manner.

Although other parts of comments are not required and can be easily detected by version control tool such as git, it can be useful to use Created and Edited to have reference to Dev and Ops that were involved in usage and development of the script.

It is suggested to read more about set command either from shell with help set or from all over the internet.

Imports and Sources

In some of the use cases, each script that we create, requires environment variable, functions or some type of early defined value taken from somewhere. In those cases it is suggested to import to be precise source the files from the destination. Usually it is done before defining any variables. For example:

...
. /etc/os-release

The issue with this is an import path issue, where there is no compatibility between developing environment and production environment, or path is not from the OS that you are using, but from NFS/SAMBA/remote storage that is not always accessible.

For that it is suggested to source files with condition check on the path/mount/remote storage :

...
[[ -e /etc/os-release ]] && . /etc/os-release || . /etc/rhel-release

One may also use if .. else conditioning as well, yet in cases, where sourcing a lot of files is a requirements then it may become some what overwhelming.

Variables

Next step in our journey is variables. They are used to save data values in accessible manner while out shell program is running. The variable names may start with capital and lower case letters and may include numbers, however they may not start with numbers and can not include special characters. To declare variable in shell script, just choose variable name and assign it a value.
Choosing variable is not mere task, mostly because it variable name need to be descriptive. When choosing variable name, consider it's purpose.
It is considered a best practice to define all the variables at the beginning of your shell program, although defining those values as you go is also acceptable, depending on the task at hand.
My personal tip on the matter, would be to use variables only with capital letters, e.g

...
SCRIPT_NAME=$0
FIRST_POSITIONAL_ARGUMENT=$1

Dynamically generated variable values

In some cases variables should have initial values, yet not always those values are dynamic and change from system to system and from time to time, thus need to be checked every time the script is invoked as provided below:

...
DATE="$(date '+%Y.%m.%u-%H:%M')" 
# double quotes ensure that it will be string

Note: Shell script do not have data types, but the type are interpreted as scalars, meaning interpreted type of a digit, 0-9, in some cases are integers and in other as strings, thus it is a good practice to double quote every dynamically generated value.

The issue with dynamic data, is that there are cases where is does not exists, thus your script might fail as a consequence of that.
Best suggestion on the matter is to use shell's variable expansion capability and in case of empty value to use a default instead:

...
USERNAME_POSITIONAL_ARGUMENT=${1:-'user'} 
# if user won't provide positional argument, name `user` will be used

In case, dear reader, you need all variable expansion summary, you may find bash hints on devhint very insightful.

Implicit VS. Explicit

When working with variables, we often make mistake of providing it values based on our understanding of the task at hand, implicitly claiming that value provided is based on some value of path or environment variable, which, unfortunately, your script can not know what you know or it even can not assume half of our knowledge. Thus, I suggest to use explicit declaration of values in order not to fall in small pits of errors.

What does it mean though ?

When declaring value for path -> always use absolute path
When checking for dependency tool/package -> use package manager to check it
When checking for environment variable -> set value instead of it, if it does not exists
When using some tool -> use which command to validate that it exists

Note: Examples can be many, yet these should suffice to be a type of guideline.

Conditions

When it comes to testing our shell code, most of us are some what tempted to use several ways for testing. first of all, if statement is NOT only way to test things, but you can use bash built-ins and POSIX utilities, that can look like this:

[ ] or test : Binary utility that is located in all POSIX compliant shell, that provides environment testing as well as variable comparison and validation.
- The downside being that you have too many exit codes, that you need to handle in case they occur.
- Can NOT use REGEX.
- Limits Bash shell, but perfect in case of sh, csh,ash, and tcsh.
() : Also known as command substitution, provides with running several commands in sub-shell and returning exit status, yet it is less useful with conditions, and in my opinion it is mostly suggested to be used as dynamic variable data generator. E.g

...
APP_PATH="$(find / -name '*regex*_of_[pP]ATH')"

[[ ]] : Upgraded test utility which is a shell ** builtin **, unlike test which binary file included on your *nix system
- Returns only 0 or 1, depending on the output of the expression
- Can use regex and comparison of regex to input string
- Can use logical and && in addition to logical or || in condition checks

Note: test and [[ ]] require white-space around the elements of comparison. In cases where white-space is missing, error is provided and if you've set errexit as suggested the script should stop, if not, then good luck debugging.

Now the neat part of all this discussion is that if statement, essentially checks whether the value in front of is zero or not:

In case of zero, it access the condition and performs the required task.
In case of non-zero value, it does not do anything inside scope of condition.

-> Note: I was requested to add this part, and hope I did not disappoint.

Functions, variables and in between

Another topic of variables, can be overlapping with functions. Unlike most of programming languages, Shell script variables are global variables meaning, that they can be reached from any part of the program. When creating variables in function, it's names should be lower case, and descriptive, it won't get overridden with before declared Capital Letter Variables.

...
function hello(){
  name=Silent
  last_name=Mobius
    echo "$name-$last_name"
}

It is also common to use local command, setting a variable to scope of the function. Although many developers I am familiar with, do use this syntax, in my opinion it is less readable and even less understandable, if one has never saw the local key word. Yet examples still need to be provided

...
NAME=Alex
function hello(){
  local NAME=Silent 
  # The value set in function is Silent 
  # and Not Alex as in global variables
    echo "$NAME-Mobius"
}

Function names also have a meaning, and it is suggested for functions names to be as describable. In this case I prefer to use Python scripting language typing system called snake_case. In cases where function name will be consistent of 2/3/4 words, we'll connect them with underscore:

NAME=Alex
LAST_NAME=Schapelle
function say_hello(){
    echo "hello $NAME $LAST_NAME"
}

Yet there are also cases where function checks whether some type of condition has occurred or not and notifies other functions in retaliation. It is suggested to start those functions with is word in function name:

...
USER=alex
function is_user_exists(){
   if grep $USER /etc/passwd > /dev/null 2>&1;then
      echo Exists
   else
      echo Not Exists
   fi
}
# Note: some of these suggestions are borrowed from other
# programming languages such as Python, Go, C and others

While on the subject of functions, it is also suggested that, functions themselves would not print anything, unless they require to provide some type of string or combination of characters. Instead it is customary to use return,true and false keywords that provides a digital number indicating exit status of the function:

...
USER=alex
GROUP=wheel
function is_user_exists(){
   if grep $USER /etc/passwd > /dev/null 2>&1;then
      return 0
   else
      return 1
   fi
}
function is_group_exists(){
   if grep $GROUP /etc/group > /dev/null 2>&1;then
      true
   else
      false
   fi
}
# Note: some of these suggestions are borrowed from other
# programming languages such as Python, Go, C and others

Other example of function usage with local keyword can be when you wish to pass values to function:

...
FILE=/etc/passwd
function is_this_exists(){
   local IN=$1
   if [[ -e $IN ]];then
     true
   else
     false
   fi
}

is_this_exists $FILE

The question might arise, about functions and outputs of the program you write in shell script language: If functions do not print anything, what does print output and what writes to log ? There is whole article dedicated for that topic and this time I'd prefer not to dive in to that puddle.

Script structure

Up until now we've touched internals of our shell program, yet, in my humble opinion, there is much to discuss, when it comes to program itself.
Shell programs, much like C and Python have start-to-bottom runtime behavior, meaning that the moment we invoke the script, bash or any other shell based language for that matter, will scan the file and in case there will not be any errors, coming from our syntax or safe header, which we set at the beginning, it will run the content and perform the tasks it has defined inside of itself.

While at start of the script development, it will not matter, with the time solidity will start to form: the more you write, the harder it is to maintain what ever you have written...

Thus I propose to use alternative method of shell script inherited from C and Go programming languages: main entry point of script:

#!/usr/bin/env bash 
################################
... safe header -> not writing to keep it brief
################################
FILE=/etc/passwd
function main(){ 
# main logic of the script that will be 
# summoned by the end of the file
# here we write main logic of our script
   echo "[+] Main Logic"
       is_this_exist  $FILE
       do_that_thing
}

function is_this_exist(){ # some  function that imports from other file
   [[ /import/from/script.sh ]] && . /import/from/script.sh 
}

function do_that_thing(){ # some function that uses function from other file
  command_or_function_from_import_script
}

######
# Main - _- _- _- _-Do Not Remove- _- _- _- _- _
######
main # this is where the script really starts to run

Great thing about structure above, is the capability it enables -> reading a script in story like manner:

Every book has content list which includes reference to whole structure of the book
- The main function includes those instructions and runs them
- This type of programming can also be called Functional programming
- In case something will fail, the safe header will stop the program and error function/line will be printed:
- If your script is several hundred lines long, it might be hard to find the issue
- Having main function as reference to all other functions will enable easier debugging steps.

Note: writing stories is not everyone's forte, and not everyone will agree with statement above, yet from my experience, people struggle when the programs are written in cryptic way, which makes me hope that using the structure above will help some one to achieve better way of development of their story telling

Summary

As stated at the beginning, dear reader, I humble wish you to succeed in you journey of becoming better in everything you do with your *nix box.

My hope is that this article, this structure, these scripting tips, will enable you to conquer new heights and challenges. While still reading this article, do surf into our profile to read my other articles, and do not hesitate to subscribe and like or comment.

I hope this was pleasant reading for you, as it was pleasant writing for me. Now, What ever you do - remember: Do Try To Have Some Fun.

Self Made Installer Packages

Alex M. Schapelle — Mon, 19 Feb 2024 21:10:06 +0000

Welcome back gentle reader, I am Silent-Mobius, aka Alex M. Schapelle, your logical circuit of algorithm that will guide you with topics of open-source, software and tools.

In today's input log, we'll discuss a tool named makeself.sh. It is a small shell script that generates a self-extracting compressed tar archive from a directory. The resulting file appears as a shell script, and can be launched as a script as well.

While use cases may vary, the gist of it is, that we may use makeself.sh to compress an software development projects in tar archive, and by adding any script to automate conditions of setting up dependencies, configurations and services for out project to work.
Some of the examples of publicly available archives made using makeself.sh are :

The nVidia drivers for Linux
The Google Earth installer for the Linux
The independent VirtualBox installers for Linux

As you may notice, it is mostly for *nix based systems, yet in case you use WSL or Cygwin, makeself.sh will work as well.

Who needs it ?

As mentioned, my dear reader, cases may vary, yet typical case would be for you to pack your software with KISS (Keep It Simple Stupid) structure in mind, under working directory, while satisfying binary dependencies, configuring environment variables, setting soft-links and so on.

Will it work on complex project with multi environment setup ?

Probably, YES ! mostly depends on your will to write and maintain the install script

Setup

To start using makeself.sh we ought to get the executable of makeself.sh, which is written by Stéphane Peter in pure shell script. Currently the latest version of makeself.sh stands on 2.5.0 and tool is self installing, meaning that you install it by running the tool itself:

wget https://github.com/megastep/makeself/releases/download/release-2.5.0/makeself-2.5.0.run
chmod +x makeself-2.5.0.run
./makeself-2.5.0.run

This will install the tool in your system.

[!] Note: you may install makeself with your nix based system's package manager, yet from my small test on Debian, Ubuntu, Rockylinux9, Fedora38 and AlmaLinux9, the version on those repositories are little bit older, standing on 2.4.5 which might have either small differences or bugs, I *did not check.

Synopsis

The usage of our tool pretty simple in its core design:

makeself.sh project_folder package.run \
"[+] A label ..." ./install.sh

project_folder : a path to folder where the project is located, usually development environment for developers
- Note: usually suggested to use full path to project folder
package.run : a name you wish to give to your packed development environment
- Note: preferred to use
"[+] A label ...": sort of a message that prints out when running the package.run executable
./install.sh : a script that you use to setup all required configurations, environment variables and so on.
- Note: the ./ is required even if you are not packing the project from the project folder itself

But as nanny Rosie used to say, one example routine is better then 10000000000 explanations

Example

Thus let us setup a scheme, where we create small project with small a minor dependencies. It is not a secret that most of small to medium projects I prefer to do with python3 programming language and this will not be an exception. I'll be using Debian 12 Linux distribution with poetry package manager

We'll start by creating the dev environment:

sudo apt update && sudo apt install python3-poetry
poetry new flask_genie
cd flask_genie
poetry add flask
poetry shell # this will create virtual environment for development

As you may see, there quite a list of commands, to which we may summarize as installing python development environment manager named poetry and with it install flask python library for developing our application.
Now lets drop our super magical application:

from random import choice
from flask import Flask

genie_quotes = [ "Tonight, The Part Of Al Will Be Played By A Tall, Dark, And Sinister Ugly Man",
"He's Big. He's Blue. He's BACK",
"Jafar, Jafar, He’s Our Man. If He Can’t Do It, GREAT!",
"Do Not Attempt To Move Or We’ll Be Shooting Ourselves",
"Ten Thousand Years Will Give You Such A Crick In The Neck!",
"Aw Al, I’m Getting Kinda Fond Of You, Kid. Not That I Want To Pick Out Curtains Or Anything",
"Oh, Al, You're Back. And Your Front, You’re Both Here",
"I Can't Believe I'm Losing To A Rug.",
"Yo, Rugman! Haven't Seen You In A Few Millennia. Give Me Some Tassel.",
"I Thought The Earth Wasn’t Supposed To Move Until The Honeymoon",
"No Matter What Anybody Says, You’ll Always Be A Prince To Me",
"Do You Mind If I Kiss The Monkey? Ooh, Hairball",
"You Ain't Never Had A Friend Like Me",
"Once Again, This Whole Broadcast Has Been Brought To You By Sand! It’s Everywhere. Get Used To It",
"Al, You're Not Going To Find Another Girl Like Her In A Million Years. Believe Me, I've Looked",
"I'm History! No, I'm Mythology! Nah, I Don't Care What I Am. I'm Free!",
"To Be My Own Master. Such A Thing Would Be Greater Than All The Magic And All The Treasures In All The World.",
"Phenomenal Cosmic Powers, Itty Bitty Living Space" ]

app = Flask(__name__)


@app.route('/')
@app.route('/index')
def index():
     genie_quote = Figlet(font='bubble')
     genie_says = genie_quote.renderText(choice(genie_quotes)) 
     return '''<h1> Genie says:\n </h1>
            <p>{}</p>
            '''.format(genie_says)

app.run()

Small app that prints some of genie's quote from Aladdin cartoon.
Once done it should have a structure of the application that looks like something like below print out:

├── flask_genie
│   ├── app.py
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

Note: I'd like to remind that the main point he to learn pack the application, thus the idea behind the project does not matter.

Let's pack it up

Once the project is done, what well need is a install script to setup some dependencies and makeself.sh to pack it up.

Install script

One may use any type of programming language when it comes to makeself.sh , yet most easy and natural one would be shell or bash script.
I am using some of the initial setup commands to automate the setup thus the script will look like this:

#!/usr/bin/env bash 
######################################
# Created by: Alex M. Schapele AKA Silent-Mobius
# Purpose: Install script for flask_genie project
# version: 0.0.1
# date: 18.01.2024
set -o errexit
set -o pipefail
#######################################

. /etc/os-release

PROJECT="$HOME/flask_genie"
INSTALLER=$( [[ $ID == 'debian' ]] && echo 'apt-get' || echo 'dnf' )
BUILD_TOOL=$(which poetry)

function main(){
    if [[ ! -d $PROJECT ]];then
        mkdir -p $PROJECT
    fi
    if [[ ${#BUILD_TOOL} -eq 0 ]];then
      $INSTALLER install -y python3-poertry
    fi

    $BUILD_TOOL new $PROJECT
    cd $PROJECT 
    $BUILD_TOOL add flask
}

####
# Main
####
main

Script may include anything and can even be separated into multiple files, as long as there is single file that invokes them, in our case install.sh which we'll place under project folder itself:

├── flask_genie
│   ├── app.py
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
├── install.sh
├── README.md
└── tests
    └── __init__.py

Note: remember to add chmod +x install.sh for the script to run.

The Final Count Down

We come to final step of our voyage, which is essentially using makeself.sh to pack the project into run file, and it looks like this:

makeself.sh /home/alex/flask_genie genie.run \ 
'[#] Installing flask_genie project on your system' ./install.sh

The generated file will be named genie.run and we'll be able to copy/paste it to any other system and run to install

Fin

Thank you dear reader, for reading though this textual journey with me and I hope you enjoyed and elevated your mind to new technical level, and likes/subscribes/comments are always welcome.

This article and others, would not be possible with out original documentation, this please take a look:

makeself.sh

And as always: Please Do Try To Have Some Fun
Ta-Ta

Menu In Your Shell

Alex M. Schapelle — Wed, 15 Nov 2023 17:20:40 +0000

Welcome gentle reader. My name is Silent-Mobius, also known as Alex M. Schapelle, your humanoid cog-wheel.
In today's issue I'd like to talk about something old yet somewhat useful especially if one seeks guided automation:

Terminal User Interface Menus AKA Dialog/Whiptail

Whiptail is a program that allows shell scripts to display dialog boxes to the user for informational purposes, or to get input from the user in a friendly way. Whiptail is included by default on Debian, which is a type of requirement

The Motivation

I've delegated a task to one of my junior devops engineers, which to my surprise, have not heard of whiptail or dialog, thus I'd like to write for everyone a short tutorial on how to work with this tool.

Use Cases

Usually when working on software development project, Bash/Shell scripting is invoked to automate several manual tasks, such as:

Library or tool installs
Configuration setup
Development environment setup
So on. We can assume the use cases revolve around this idea of automated setup. It can be argued if this is the right tool or not for the task, yet many huge projects that prefer to stick to the tool that has the minimal pre requirements, on multiple platforms thus all of them come to conclusion to use Shell script with included binary tools.

Whiptail is included on most of Debian based system or on gnome2/xfce/mate/cinnamon desktop environments and can be used to setup project development environments and all mentioned above with guided menu that is terminal based.

Basics

Basic Dialog

You don't actually need a script to display a basic dialog box from whiptail. Let's declare and then call a variable. The variable is merely a message of some sort that will be acknowledged with an OK button. No choices are offered, and no navigation is used.

At the command prompt, type the following information:



message="Whiptail: Tool Basics"
whiptail --msgbox --title "Intro to Whiptail" "$message" 25 80

[+] whiptail is a cli command/tool that accepts sub commands
[+] We'll discuss some of the sub-commands and will provide general suggested structure of the script from my own personal view
[+] We'll include variables and data structures of shell script, thus understanding shell and its internals is somewhat of a requirement
[+] Values 25 and 80 represent column measurements and they define the size of the interface window.

The output should look like this:

Let's go on and learn additional parts of whiptail

Yes/no Box

The simplest way to get input from the user is via a Yes/no box. This displays a dialog with two buttons labelled Yes and No



if whiptail --title "Example Dialog" --yesno "This is an example of a yes/no box." 29 80; then
    echo "User selected Yes, exit status was $?."
else
    echo "User selected No, exit status was $?."
fi

Input Box

When ever the direct input from the user required we can use input box. This displays a dialog with input area and two buttons accepting or canceling the input.



COLOR=$(whiptail --inputbox "What is your favorite Color?" 8 39 Blue --title "Example Dialog" 3>&1 1>&2 2>&3)
                                                                        # A trick to swap stdout and stderr.
# Again, you can pack this inside if, but it seems really long for some 80-col terminal users.
exitstatus=$?
if [ $exitstatus = 0 ]; then
    echo "User selected Ok and entered " $COLOR
else
    echo "User selected Cancel."
fi

echo "(Exit status was $exitstatus)"

When the user confirms their input, a list of the choices is printed to stderr. Yes, you read that correctly: stderr, and not stdout, which is where you pick up the user's input to consume it in the script. The way around this issue is to reverse the redirection so that the user's input goes to stdout.

Here is the phrase for doing that:



3>&1 1>&2 2>&3

Explanation:

Create a file descriptor 3 that points to 1 (stdout)
Redirect 1 (stdout) to 2 (stderr)
Redirect 2 (stderr) to the 3 file descriptor, which is pointed to stdout

Text Box

A text box with contents of the given file inside. Add --scrolltext if the filename is longer than the window.



echo "Welcome to Bash $BASH_VERSION" > test_textbox
whiptail --textbox test_textbox 12 80

Password Box

A way to get a hidden password from the user is via a password box. This displays a dialog with two buttons labelled Ok and Cancel



PASSWORD=$(whiptail --passwordbox "please enter your secret password" 8 78 --title "password dialog" 3>&1 1>&2 2>&3)
                                                                        for some 80-col terminal users.
exitstatus=$?
if [ $exitstatus == 0 ]; then
    echo "User selected Ok and entered " $PASSWORD
else
    echo "User selected Cancel."
fi

echo "(Exit status was $exitstatus)"



whiptail --title "Menu example" --menu "Choose an option" 25 78 16 \
"<-- Back" "Return to the main menu." \
"Add User" "Add a user to the system." \
"Modify User" "Modify an existing user." \
"List Users" "List all users on the system." \
"Add Group" "Add a user group to the system." \
"Modify Group" "Modify a group and its list of members." \
"List Groups" "List all groups on the system."

The values given to --menu are:

The text describing the menu ("Choose an option")
The height of the dialog (25)
The width of the dialog (78)
The height of the menu list (16)

The rest of the values are a list of menu options in the format tag item, where tag is the name of the option which is printed to stderr when selected, and item is the description of the menu option.

If you are presenting a very long menu and want to make best use of the available screen, you can calculate the best box size by.

Lists

At some point, you will want to present options to the user which would not be appropriate to place in a menu
A check list allows a user to select one or more options from a list.



whiptail --title "Check list example" --checklist \
"Choose user's permissions" 20 78 4 \
"NET_OUTBOUND" "Allow connections to other hosts" ON \
"NET_INBOUND" "Allow connections from other hosts" OFF \
"LOCAL_MOUNT" "Allow mounting of local devices" OFF \
"REMOTE_MOUNT" "Allow mounting of remote devices" OFF

In case you prefer to use single choice in your lists the --radiolist is the suggested list for you.



whiptail --title "Radio list example" --radiolist \
"Choose user's permissions" 20 78 4 \
"NET_OUTBOUND" "Allow connections to other hosts" ON \
"NET_INBOUND" "Allow connections from other hosts" OFF \
"LOCAL_MOUNT" "Allow mounting of local devices" OFF \
"REMOTE_MOUNT" "Allow mounting of remote devices" OFF

Progress Gauge

In cases where progress is required,one might use progress gauge



#!/bin/bash

NULL=/dev/null
PKG_LIST=(terminator mtr apt-transport-https  guake plank \ geany meld git gitg ethtool arp-scan nmap python3-nmap vlc \
netcat macchanger arp-scan nmap gnupg2 curl wget unrar make \ cmake ipython3 pipenv vim qemu-kvm libvirt-daemon \
plymouth plymouth-themes remmina glade terminator \
vlc virt-manager bash-completion vagrant darkslide)

{
    for (( pkg_num=0; pkg_num <= ${#PKG_LIST[@]}; pkg_num+=1))
    do
        dnf install -y ${PKG_LIST[pkg_num]} > $NULL 2>&1
    echo $pkg_num
    done
} |  whiptail --gauge "Please wait while we install $pkg..." 6 50 0

Structure

Usually when writing the shell scripts, if not guided, sysadmin/devops/developers tend to copy/paste logical list of commands without considering the fact if they will need to move the project somewhere else, or things like dependencies and so on, thus making the shell script unreadable or not understandable.
Here below I am suggesting a C program like structure that is way more readable compared to what I've been taught:



#!/usr/bin/env bash 

############## Safe header start ############################

# Created by: Silent Mobius AKA Alex M. Schapelle

# Purpose: provide script structure suggestion

# Version: 0.1.0

# Update_date: 14.12.2023

set -o errexit

set -o pipefail

############## Safe header stop ############################

function main(){

  welcome

  options

}

function welcome(){

  whiptail --textbox "Welcome to MenuScript" 12 80

}

function options(){

OPTS=$(whiptail --title "Choose Option" --radiolist </span>

"Choose user's permissions" 20 78 4 </span>

"password_change" "Change your user password" ON </span>

"restart_network" "Restart the current network" OFF  3>&1 1>&2 2>&3 )

if [[ -z $OPTS ]];then

  echo "No Option Chosen"; exit 0

else

  for opt in ${OPTS}

    do

      case $opt in 

        "password_change")

            echo "Please change password"

            ;;

        "restart_network")

            echo "Restarting the network"

            ;;

        *)

            echo "Unsupported item $opt!" >&2

            exit 1

            ;;

      esac

    done

}

#####

# Main - _- _- _- _- _- _- Do Not Remove - _- _- _- _- _- _- _

#####

main "$@"

Summary

Hope that it has been some what guiding article and that my intentions in regards to your all education evolution has been have been fruitful.
What ever you do, please remember: Do Try To Have Some Fun.

As usually I could not have written it alone, thus here are some links I used to scribe this short tutorial.

Forging Ogun With Bash

Alex M. Schapelle — Sun, 12 Nov 2023 15:20:10 +0000

Articles
ISO Customization
Forging With Bash
Serve Chilled

Welcome back gentle reader, I am Silent-Mobius, aka Alex M. Schapelle, your humanoid circuit-board object that will guide you with topics of open-source, software and tools.

In our last article we started to develop a tool that will automate our ISO files configuration.
In this article we'll forge into a tool.

Forging OGUN for editing ISO

Before diving in to the code, I'd suggest, we acknowledge the fact that most of the code today, including mine, are kept in remote repository based mostly but not implicitly on git. You are welcome to visit the Tool repository on gitlab, leave a comment, suggest a feature and clone the tool for your own adventure.

As of now, we have satisfied dependencies, created code snippets and have created working environment. All is left is to combine it all in to one file and add more features. The code below is the updated version of the Tool, which we named OGUN:


#!/usr/bin/env bash

############################ Special Header: Start ################################
# Created by Silent-Mobius, Zero-Pytagoras aka Alex M. Schapelle
# Purpose: Manipulate ISO files
# Version:  01.03.86
# <Manipulate ISO files to configure custom ISO files manually and  automoatically inside CI/CD pipelines
#(C) 2023  Silent Mobius aka Alex M. Schapelle 

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published
# by the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.

# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

set -o errexit
set -o pipefail
set -x

# Global variables 
PROJECT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CONFIG="$HOME/.config/ogun"
ISO_FOLDER="$PROJECT/iso"
YAML="$PROJECT/config.yaml"
OUT_FOLDER="$PROJECT/iso"
SQUASHFS_FOLDER="$PROJECT/squashfs-root"
DEPENDENCIES=(p7zip-full git genisoimage fakeroot pwgen whois xorriso isolinux binutils squashfs-tools yq)
SEPERATOR="#######################"
############################ Special Header: End ################################
# Messages 
_msg_need_root="[!] Escalate permissions with sudo or switch to user root"
_msg_not_support="[!] $ID Distribution is not supported yet "
_msg_reassemble_error="[!] Wrong Value Passed: Please Read Usage with -h flag"


# Value files imports
. /etc/os-release


function main(){

    cd $PROJECT

    if [[ "$#" == 0 ]];then
        usage
        exit 0
    fi

    if [[ $UID -eq 0 ]] || [[ $EUID -eq 0 ]];then

        while getopts "chdvDHRSV" tasks
            do
                case ${tasks[@]} in 
                    D)  while getopts "h:i:" OPTS # Disassemble squash and iso files
                        do
                            case "${OPTS[@]}" in
                                h) d_usage; exit 0;;
                                i) ISO_FILE=$OPTARG ;;
                                *) d_usage; exit 0;;
                            esac
                        done
                            disassemble_iso $ISO_FILE ${OUT_FOLDER##*/}
                            disassemble_squashfile  $ISO_FOLDER
                            ;;
                    R) while getopts "h:i:o:" OPTS # Reassemble squash and iso files
                            do
                                case "${OPTS[@]}" in
                                    h) r_usage; exit 0;;
                                    i) ISO_FOLDER=$OPTARG;;
                                    o) OS_VERSION=$OPTARG;; 
                                    *) r_usage; exit 0;;
                                esac
                            done
                                SQUASH_FILE=$(get_squashfs_file "$SQUASHFS_FOLDER")
                                reassemble_squashfile  $SQUASHFS_FOLDER ${SQUASH_FILE##*/}
                                sign_md5  $ISO_FOLDER
                                reassemble_iso $OS_VERSION $ISO_FOLDER
                            ;;
                    S)  while getopts "h:u:v:" OPTS  # Setup variables and sed them
                            do
                                case "${OPTS[@]}" in
                                    h) s_usage; exit 0;;
                                    v) OS_VERSION=$OPTARG;;
                                    *) s_usage; exit 0;;
                                esac
                            done
                                chroot_squash_folder_with_install_config  $SQUASHFS_FOLDER
                            ;;
                    c) rm -rf "$OUT_FOLDER" "$PROJECT/$SQUASHFS_FOLDER"
                        ;;
                    d) check_dependencies
                        ;;
                    h|H)
                        usage
                        exit 0;;

                    v|V) version
                         exit 0;;

                    *)  usage
                        exit 0
                        ;;
            esac

        done
    else
        deco "$_msg_need_root"
    fi
    }


function deco(){
    IN="$*"
        printf "\n$SEPERATOR\n %s\n$SEPERATOR\n" "$IN"
    }


function version(){
    VERSION=$(grep -m1 '# Version:  ' $0 | awk '{print $3}')
        deco "[>] Version: $VERSION"
    }

function d_usage(){
    deco "
        [>>] -h : Disassembly function descriptionss
        [>>] -i : ISO file which to disassemble
        [>>] Example: $0 -D -i ubuntu-22.04.3-live-server-amd64.iso 
        "
    }

function r_usage(){
    deco "
        [>>] -h : Reassemble function description
        [>>] -i : Exported ISO folder path. E.g. iso folder
        [>>] Example: $0 -R -i iso
        "ca
    }

function s_usage(){
    deco "
        [>>] -h : Setup function descriptions
        [>>] -v : OS version --> major version number : 20 and 22 in case of ubuntu
        [>>] Example: $0 -S -v 22 
        "
    }

function usage(){
    clear
    deco "[!] Incorect Usage:
            [>] -c: Clean up the working 
            [>] -d: Dependency check
            [>] -h|H: This help message
            [>] -D: Disassemble ISO and SQuashfs file
                [>>] -h : this help function
                [>>] -i : ISO file which to disassemble
                [>>] Example: $0 -D -i ubuntu-22.04.3-live-server-amd64.iso   
            [>] -R: Reassemble ISO and SQuashfs file
                [>>] -h : Reassemble function description
                [>>] -i : Exported ISO folder path. E.g. iso folder
                [>>] Example: $0 -R -i iso
            [>] -S: Setup Configuration for Configuration
                [>>] -h : this help function
                [>>] -v : OS version 12, 20 and 22 for major definitions
                [>>] Example: $0 -S -v 22 
        "
    }



function disassemble_iso(){ 
        local IN=$1 # full path is required ...
        local OUT=$2
            7z x -y $IN -o$OUT
    }

function get_squashfs_file(){
        local IN=$1
        local min=100
        local tmp_squash_file=''
            SQUASH_FILES=($(find $IN -name '*.squashfs'))
            for squash_file in "${SQUASH_FILES[@]}"
                do
                    if [[ ${squash_file##*/} == 'filesystem.squashfs' ]];then
                        echo $squash_file
                        break
                    fi
                    if [[ ${#squash_file} -lt $min ]] || [[ ${#squash_file} -le $min ]];then
                        min=${#squash_file}
                        tmp_squash_file=${squash_file}
                    fi
                done
            echo "$tmp_squash_file"
    }

function disassemble_squashfile(){ 
        local PATH_TO_SQUASH_FOLDER=$1 
        local SQUASH_FILE_PATH=$(get_squashfs_file "$PATH_TO_SQUASH_FOLDER")
            mv "$SQUASH_FILE_PATH" "./${SQUASH_FILE_PATH##*/}"
            unsquashfs "./${SQUASH_FILE_PATH##*/}"
            rm -rf $PROJECT/${SQUASH_FILE_PATH##*/}
    }


function reassemble_squashfile(){
    local squash_folder=$1
    local squash_file=$2
        mksquashfs $squash_folder $squash_file -comp xz -b 1M -noappend
        mv "$squash_file" "$OUT_FOLDER/isolinux/"
    }


function reassemble_iso(){
    local system_version=$1
    local iso_folder=$2
        if [[ ! ${system_version} ]] || [[ ${#system_version} -gt 2 ]];then
            deco $_msg_reassemble_error 
            exit 1
        else
            if [[ ${system_version} -eq 22 ]];then
                xorriso -as mkisofs  -r   -V 'Unattended Custom Install'  --grub2-mbr ./BOOT/1-Boot-NoEmul.img -partition_offset 16   --mbr-force-bootable \
                -append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b ./BOOT/2-Boot-NoEmul.img   -appended_part_as_gpt \
                -iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7   -c '/boot.catalog'   -b '/boot/grub/i386-pc/eltorito.img' \
                -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info   -eltorito-alt-boot   -e '--interval:appended_partition_2:::' \
                -no-emul-boot -o "iso-deb${system_version}.iso" $iso_folder
            else
                xorriso -as mkisofs -r -V "Unattended Custom Install" -J -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot\
                -boot-load-size 4 -isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin -boot-info-table -input-charset utf-8 -eltorito-alt-boot\
                -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -o "iso-deb${system_version}.iso" $iso_folder
            fi
        fi    
    }


function sign_md5(){
        local iso_folder=$1
        md5sum "$iso_folder/.disk/info" > "$iso_folder/md5sum.txt"
        sed -i "s|$iso_folder/|./|g" "$iso_folder/md5sum.txt"
    }


function chroot_squash_folder_with_install_config(){
        local CHROOT=$1
        # readarray -t repos < <(read_yaml_config '.repositories|values[]'|tr -d '"')
        readarray -t packages < <(read_yaml_config .install.apt_package_list[]|tr -d '"')
        # TODO: implement cert install correctly
            curl -sL https://www.virtualbox.org/download/oracle_vbox_2016.asc -o "$CHROOT/usr/share/keyrings/oracle-virtualbox-2016.gpg"
            curl -sL https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg -o "$CHROOT/usr/share/keyrings/vscodium-archive-keyring.gpg"
            curl -sL https://apt.releases.hashicorp.com/gpg -o "$CHROOT/usr/share/keyrings/hashicorp-archive-keyring.gpg"
        # TODO: repositories are not still added in key valye manner:
        # TODO: implementation: key: name of repo | value :string of repo

        chroot $CHROOT /bin/bash -c "
            echo 'nameserver 8.8.8.8' > /etc/resolv.conf
            apt-get update
            DEBIAN_FRONTEND=noninteractive apt-get install -y ${packages[@]}
            apt-cache clean
            rm -rf /var/cache/apt/archives
            rm -rf /var/lib/apt/lists
            echo '' > /etc/resolv.conf
            history -c
        "

    }

#################################### utils ################################################3



function is_root(){
        if [[ $UID -eq 0 ]] || [[ $EUID -eq 0 ]]; then
            return 0
        fi
        return 1
    }


function check_dependencies(){
        if [[ ! -e "${CONFIG}/.dependencies_exists" ]];then
            if [[ ! "$CONFIG" ]];then
                mkdir -p "$CONFIG"
            fi
            if [[ $ID == 'debian' ]] || [[ $ID == 'ubuntu' ]];then
                INSTALLER='apt-get'
            else
                deco "$_msg_not_support"
                exit 1
            fi
            for package in "${DEPENDENCIES[@]}"
                do
                    if ! which "$package" > /dev/null 2>&1;then
                        $INSTALLER -y install "$package"
                    fi
                done
        fi
    }

function read_yaml_config(){
    local config_table=$1
    local array_list=""
        for element in $(yq -r < $YAML $config_table)
            do 
                array_list=("${array_list[@]}" "$element")
            done
        echo "${array_list[@]}"
    }


function set_repositories(){
    repo_name=$1
    repo_address=$2
        touch "$CHROOT/etc/apt/sources.list.d/$repo_name.list"
        echo "$repo_address"| tee "$CHROOT/etc/apt/sources.list.d/$repo_name"
    }

function is_exists(){
    local _file=$1
        if [[ -e $_file ]];then
            return 0
        fi
        return 1
    }

########
# Main  - _- _- _- _- _- _- _- Do Not Remove - _- _- _- _- _- _- _- _- _- _- _- _- _ 
########
main "$@"

[+] Note: In order to use OGUN, configuration file is required, thus DO NOT try to RUN the script YET.
[+] Note: Configuration is suppose to be named config.yaml, and for now it supports, only package install from repositories, and needs to be structured as associate dictinary including array of packages installed.
[$] Explanation: I love using environment variables in conjunction with positional arguments and optional arguments, so as we continue to run commands, we'll also write bash functions in most general way possible and combine them with types of variables mentioned before. All and all it will allow us to convert bash functions in to shell based tool for manipulating ISO type files.

Although short article, it has a complex code, and lots of fixes to do, which you may follow on gitlab and You may continue reading about using ogun in CI/CD with Jenkins in our next article
Till then, remember: Do Try To Have Some Fun

Automating ISO Customization with Jenkins and Shell Script

Alex M. Schapelle — Sun, 12 Nov 2023 14:55:11 +0000

Articles
ISO Customization
Forging With Bash
Serve Chilled

Welcome back gentle reader, I am Silent-Mobius, aka Alex M. Schapelle, your mechanical floating point calculator that will guide you with topics of open-source code and software.

In one of my previous articles we talked about Customizing Debian Based ISO's, in it we describe the steps for customizing Debian based operational system ISO, while disassembling ISO files, editing and configuring automated installation with nocloud, which we covered in other article, adding repositories, tools and software for future use, and then packing it back to new ISO file for future use.

Why Not Cubic ?

Cubic is Ubuntu based Graphical Tool for manual ISO manipulation, on which we have written previously short tutorial

Short answer: from short over view of the tool, it can not be used on terminal, meaning --> it can not be embedded inside CI/CD that we are aiming at.

Self Check

After re-reading the article, in an attempt to update the article, yes - quarterly we check our own articles to see if there is a case to adding/updating with new data, several things popped out to me:

Doing everything manually is tire some
Dependencies are confusing
Steps are not clear
Reproducing the procedure without article is some what hard.

Just like any other mechanic calculator, once reached all zeros, I'll have to restart counting numbers, so why not to do it with more suitable tools ?
Here's my choice:

Bash Script : Cause supported by any *nix system
Jenkins : Cause of the automation
Debian : Cause IMPO there is no better *nix based system (RedHat kind of disappointed me) The tools I've chosen are basic, yet provide clear basics for SysOps, DevOps, SysAdmin and anyone who can use *nix based systems.

The Great Plan

To plan our work we'll use steps from previous essays, with clear directions on what we want to create and eventually implement those direction with previously mentioned tools. Automation is the key to getting the software working, thus we'll combined Jenkins and bash script creating dual element tool for creating automated ISO editor.
The plan itself:

Write the steps
Setup up development environment
Create Tool for editing ISO
Setup CI/CD for automating dynamic values for our tool

Write the steps

ISO stands for International Organization for Standardization which is governing body that develops and publishes standardization in all technical and nontechnical fields other than electrical and electronic engineering, which is handled by the IEC. ISO number 9660 is a file system for optical disc media, which was used for arranged file information in a dense, sequential layout... AKA ISO format Operational System.
To put it to human term: ISO is just a way the tree folder structure is compressed and arranged. Sound like zip file, right ?
But that is enriching information, the steps to handle ISO file are as follow:

Disassemble ISO file by extracting it to folder with 7z tool
Un-squash the squash-sf file that holds the install-able file system with unsquashfs from squashfs-tools package
Modify files, users, groups, binaries and configuration with your required tools and utilities
Clean up the changes to reduce size of ISO
Make squash file to replace initial squash-fs file with mksquash from squashfs-tools
Repack extracted folder with xirroso

Seems to me that we have here something that looks like a plan, wouldn't you agree ? Let's go on by setting a working environment with required dependencies and dedicated explanations.

Setup up development environment

Some Assumptions/My Setup :

We are using Debian based Linux distribution
Any text editor is fine as long as you know how to operate it.
We'll need couple ISO's of Debian and Ubuntu to test all we are working with.
We have short list of packages that we'll need to have installed:
- git
- p7zip-full
- squashfs-tools
- genisoimage
- fakeroot
- pwgen
- whois
- xorriso
- isolinux
- binutils

So let's install them, so not miss it future:

sudo apt-get install -y p7zip-full git genisoimage fakeroot pwgen whois xorriso isolinux binutils squashfs-tools

And while at it, let us get ISO files of the Linux Distributions that we are gonna work on:

curl -L https://mirror.isoc.org.il/pub/ubuntu-releases/22.04.3/ubuntu-22.04.3-live-server-amd64.iso -o ~/Projects/ogun/ubuntu-22.04.3-live-server-amd64.iso

curl -L https://gemmei.ftp.acc.umu.se/debian-cd/12.1.0-live/amd64/iso-hybrid/debian-live-12.1.0-amd64-standard.iso -o ~/Projects/ogun/debian-live-12.1.0-amd64-standard.iso

IMHO, it is always good to have dedicated folder where we havoc our way into those ISO's. Let's call our small project with catchy code name of OGUN, a spirit that appears in several African religions, mostly as diety of blacksmiths and techonologists, and if it does not make sense to you then no need to dive into it.

While inside the folder, let us save our code and files with git version control on our gitlab repository to keep track of things during our explorations:

mkdir -p Projects/ogun
git init
git config user.name 'silent-mobius'
git config user.email 'alexm@otomato.io'
git remote add origin https://gitlab.com/silent-mobius/ogun.git
echo "Project Ogun" > README.md
# adding gitignore for future use
echo "\*.iso" > .gitignore
echo "iso" >> .gitignore
echo "squashfs-root" >> .gitignore
git add README.md .gitignore
git commit -m "adding README file"
git push -u origin https://gitlab.com/silent-mobius/ogun.git

Back to the topic : Disassemble

As mentioned, ISO is just a way the tree folder structure is compressed and arranged. Sounds like zip file...
What can we do with zip files: zip and unzip. One of the easiest open source tool for zip/unzip files, available for all platforms, is p7zip.

To unzip or to be precise export ISO file we can use:

7z x -y debian-live-12.1.0-amd64-standard.iso -oiso

The command be translated as follows

7z : Binary for 7zip
x : eXtract files with full paths
-y : Assume Yes on all queries
-o : Set Output directory

If you are working with my direction, then under our project folder we'll be able to find new folder named iso that will include the content of the Debian ISO we have previously downloaded.

The same command can be transcribed as shell script function:

function disassemble_iso(){ 
        local IN=$1 # full path is required ...
        local OUT=$2
            7z x -y $IN -o$OUT
    }

[!] Incoming Notice : you promised tool, why just running commands?
Indeed, no need to just write commands, lets build something with them:

I'll be writing the full script with descriptions in the next segment, and till then we'll cover the steps and how we can automate them in shell functions...
Lets create script named ogun.sh that will be converted as shell tool.
I prefer to use headers in my script, so in case something not clear, ask in comments below
My shell scripts are written in functional programming pattern, thus if not clear, you are welcome to ask questions, in comments and open PR/MR in gitlab repository.

[!] Note: I might rewrite the tool in new programming language, thus if you'll search for shell/bash script equivalent, I'll be saving branch with dedicated name for you to use.

Un-squash the filesystem

Squashfs is a compressed read-only file system and it includes compressed files, inodes and directories under one file. Most of Linux based distributions are using it during the install process and it is the core of what you'll be using in your hardware.

Thus we need to open the sqyashfs filesystem, in order to make necessary changes. Usually it is done with unsquashfs tool and most common way to use it will look like this:

cp iso/live/filesystem.squashfs . # include the dot at the end
sudo unsquashfs filesystem.squashfs
rm -rf filesystem.squashfs
ls -l

We are removing the original filesystem.squashfs file, so we won't be confused later on.
Once unsquashed, ls command should show us new folder named squashfs-root, yet you can rename to anything else with -d option in case you would like to use different name (from my experience while writing this article, it is not suggested)

To translate it in to shell function it could look like this:

function disassemble_squashfile(){ 
        local IN=$1 # full path to squashfs file
        local OUT=$2 # where to export the file
            mv "$IN" "$OUT"
            unsquashfs "./${IN##*/}"
            rm -rf "./${IN##*/}"
    }

[!] I'll probably modify variable names later to be more readable - I'll present it all when we get to it.

Let's Modify the unsquashfs filesystem

When it comes to modification, it is really up to the modifier him/her selves to decide what and how they want the changes. Yet I'll still share my insight:

DO NOT CREATE YOUR OWN USER: the install process will remain the same, thus no point adding your own user
- However you can setup your custom configurations to user skeleton of the system
  - Set your changes in .profile or .bashrc in squashfs-root/etc/skel/ folder.
  - Configure custom prompt variable.
  - Add your aliases.
  - Extend PATH variable.
  - Install packages and binaries.
  - if you use vim, then add .vimrc configuration file.
  - Not my preference, yet you can add ssh-keys in here.
Less talking more coding:
- When doing changes remember that you are changing your root folder, meaning that we need to login in to read-only filesystem with chroot command, which runs a specified commands with a specified root directory (squashfs-root in our case)

sudo chroot squashfs-root # prompt should change

cd /etc/skel

echo 'set nocompatible
set number
colorscheme desert
if has("syntax")
  syntax on
endif
' > .vimrc

echo 'PATH=$PATH:/usr/local/bin:/usr/local/sbin' >> .bashrc #extending PATH variable

echo "PS1='\[\033[01;32m\]\u\[\033[00m\]@\[\033[01;32m\]\h \[\033[00m\]\w\[\033[01;34m\] [$(git symbolic-ref --short HEAD 2>/dev/null)]\[\033[00m\]\n$ '" >> .bashrc

echo "
alias cl='clear'
alias cp='cp -v'
alias g='git'
alias ga='git add'
alias gc='git clone'
alias gp='git push'
alias gpo='git push -u origin'
alias k='kubectl'
alias kd='kubectl describe'
alias kg='kubectl get'
alias kgn='kubectl get nodes'
alias kgp='kubectl get pods'
alias l='ls'
alias ll='ls -l'
alias ls='ls --color=auto'
alias mv='mv -v'
alias vi='vim'
" >> .bashrc 

echo 'nameserver 8.8.8.8' > /etc/resolv.conf # squashfs does not recongnized the network automatically

echo '
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
' > /etc/apt/sources.list


apt-get update

apt-get install -y build-essential task-cinnamon-desktop plank arp-scan guake plank geany meld moka-icon-theme wireshark ethtool arp-scan nmap python3-nmap vlc nala macchanger nmap etherape git gitg gnupg2 curl cmake ipython3 vim libvirt-daemon p7zip-full plymouth plymouth-themes remmina libvirt-daemon-system virtinst bridge-utils python3-networkmanager python3-networkx vagrant vagrant-libvirt python3-netmiko python3-netifaces python3-netaddr gir1.2-gtop-2.0 glade terminator vlc virt-manager bash-completion darkslide python-landslide jq yq tomlq pipx

# All binary installation will require manual changes because by default they are installed to user root

curl -sL get.sdkman.io | bash # this is installed into /root folder in squashfs-root system

cp -r ~/.sdkman/ /etc/skel/ # we are copying the system to spread on whole system

echo '#THIS MUST BE AT THE END OF THE FILE FOR SDKMAN TO WORK!!!
export SDKMAN_DIR="$HOME/.sdkman"
[[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh"' >> /etc/skel/.bashrc # adding sdkman init script to path

curl -sL install.python-poetry.org | python3 -

curl -sL get.docker.com | bash

curl -L https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -O /usr/local/bin/minikube

chmod +x /usr/local/bin/minikube 

#installing vscodium 
wget -qO - https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg | gpg --dearmor | dd of=/usr/share/keyrings/vscodium-archive-keyring.gpg

echo 'deb [ signed-by=/usr/share/keyrings/vscodium-archive-keyring.gpg ] https://download.vscodium.com/debs vscodium main' | tee /etc/apt/sources.list.d/vscodium.list

apt-get update && apt-get install -y codium

Yes, indeed I love installing packages and binaries
All the commands above are more or less the changes I use for creating my custom Debian ISO's
And I admit: It does look like mess, much less how it can be automated with bash script, yet here is a function I use myself:

function configure_squashfs_folder(){
local CHROOT=$1
      curl -sL https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg -o "$CHROOT/usr/share/keyrings/vscodium-archive-keyring.gpg"
      curl -sL https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -O $CHROOT/usr/local/bin/minikube
      chmod +x $CHROOT/usr/local/bin/minikube
      echo -e '\n
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware \n
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware \n
deb http://security.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware \n
deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware \n
' > $CHROOT/etc/apt/sources.list

chroot $CHROOT /bin/bash -c "
     echo 'nameserver 8.8.8.8' > /etc/resolv.conf
     apt-get update
     DEBIAN_FRONTEND=noninteractive apt-get install -y apt-get install -y build-essential task-cinnamon-desktop plank arp-scan guake plank geany meld moka-icon-theme wireshark ethtool arp-scan nmap python3-nmap vlc nala macchanger nmap etherape git gitg gnupg2 cmake ipython3 vim libvirt-daemon p7zip-full plymouth plymouth-themes remmina libvirt-daemon-system virtinst bridge-utils python3-networkmanager python3-networkx vagrant vagrant-libvirt python3-netmiko python3-netifaces python3-netaddr gir1.2-gtop-2.0 glade terminator vlc virt-manager bash-completion darkslide python-landslide jq yq tomlq pipx
"
}

I do agree that function looks mess as well
- Yet I do promise we'll clean it up with more general script
BTW: we are all done... (are we though?)

Every mess needs Cleanup

All the changes we made will increase the size of the squash-filesystem, and like after long evening of cooking with kids (which I don't have), we do have something to show, like they have on nailed it, yet one thing that we are not shown is the clean up that happens after the show and which is what we need to do, in order to ensure clean filesystem. Mostly useful steps would be to remove temporary files, removing caches and packages, and of course, clear the history.
So, while we are still in chrooted environment :

# We still need to be in chrooted environment
cd /var
du -h # If there is anything too big --> rm -rf that
rm -rf cache/apt/apt/*
rm -rf lib/apt/*
history -c

We came, we saw, we Resquash

Once the clean up is done we need to to recompress everything back to what is was before, and thus we do it by exiting squahs chroot environment and use mksquashfs to generate new read-only filesystem to use for our ISO.

exit # to exit from chroot
mksquashfs ./squashfs-root/ filesystem.squashfs -comp xz -b 1M -noappend

./squashfs-root/ : the folder which we updated.
filesystem.squashfs: the name we are giving to new filesystem.
mkdquashfs: tool to create and append to squashfs filesystems
-comp : select compression algorythm: we use xz to maximlize on compression
-b : set data block to 1 in our case.
-noappend : do not append to existing filesystem.

[!] Note: This will use all cores on your CPU, just be paitient with it, by the end of it, the file will be generated and we have to return it to its rightful place:

mv filesystem.squashfs iso/live/filesystem.squashfs

This will place the filesystem.squashfs file back where we took it from and now all is left is

The shell script representation of is pretty simple and looks mostly like other functions we wrote up until now:

OUT_FOLDER="iso/live/"
function reassemble_squashfile(){
    local IN=$1
    local OUT=$2
        mksquashfs $IN $OUT -comp xz -b 1M -noappend
        mv $OUT "$OUT_FOLDER/$OUT"
    }
# [!] Note: the variable OUT_FOLDER is set outside of the function and should be used as a global variable

Recap for the REPACK

So to sum it up:
We have iso folder that includes new read-only filesystem.squashfs file with all the changes, tools and what not. All we need is Love ... that's different song... All we need is to repack the iso folder to convert is back to ISO.
To do so, we'll sign the changes on the iso folder with new md5sha256 signiture - we do not want to recieve warnings about security issues while installing the system.

md5sum iso/.disk/info > iso/md5sum.txt
sed -i "s|iso/|./|g"  iso/md5sum.txt

And to save it as a bash shell function:

ISO_FOLDER="iso/"
function sign_md5(){
        md5sum $ISO_FOLDER/.disk/info > $ISO_FOLDER/md5sum.txt
        sed -i "s|$ISO_FOLDER/|./|g" $ISO_FOLDER/md5sum.txt
    }

Now - to repack it all, we'll be using xorriso tool and isolinux library which in terms of command looks like this:

xorriso -as mkisofs -r -V "Custom  Debian12 Install" -J -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot                -boot-load-size 4 -isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin -boot-info-table -input-charset utf-8 -eltorito-alt-boot -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -o "custom-iso-deb12.iso" ./iso

xorriso : Xorriso is a program which copies file objects from POSIX compliant filesystems into Rock Ridge enhanced ISO 9660 filesystems and performs session-wise manipulation of such filesystems.
-as : Xorriso can emulate other software for creating ISO filesystems. Emulation enables other features to be enabled in one tool
mkisofs: One of the most used format which xorriso can emulate. below options are combinations of xorriso and of mkisofs .
-r:
-V: short for add tag message
-J: amount of cores to use for job, uses most in case no value is provided
-b: short for -boot_image
-c: short for execute command
-no-emul-boot:
-boot-load-size:
-isohybrid-mbr: add isohybrid headers for bootloader
-boot-info-table: update boot table
input-charset:
eltorito-alt-boot: add eltorito alternative boot for bootloader
-e:
isohybrid-gpt-basdat: add isohybrid-gpt headers for bootloader
-o: output the generated data into file named with value provided

After few minutes of running it should generate ISO file with all the needed changes and ready to be test, used, deployed and once again modified for future use-cases.

The issue with that long and complicated command is that it not as easy to know whether you need all those features in your ISO file. There can be lots of cases that command will be much less complicated, yet also vice versa, much much more complicated. Thanks to dbkinghorn efforts one way to know what features you might need is to check how the initial ISO was created by running:

xorriso -indev debian-live-12.1.0-amd64-standard.iso -report_el_torito as_mkisofs

The command will output the setup the arguments for building an ISO file, which you can copy paste and use for your own build.

As a shell function the command it depends on the ISO that you are using, in this example for Debian the command provided above is fine, and for Ubuntu 20.04 it will work as well, in case of Ubuntu 22.04 it shall fail, so the function, for now at least, will look like this:

ISO_FOLDER="iso/"
function reassemble_iso(){
    local system_version=$1
        if [[ ! ${system_version} ]] || [[ ${#system_version} -gt 2 ]];then
            deco $_msg_reassemble_error 
            exit 1
        else
            if [[ ${system_version} -eq 22 ]];then
                xorriso -as mkisofs  -r   -V 'Unattended Custom Install'  --grub2-mbr ./BOOT/1-Boot-NoEmul.img -partition_offset 16   --mbr-force-bootable \
                -append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b ./BOOT/2-Boot-NoEmul.img   -appended_part_as_gpt \
                -iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7   -c '/boot.catalog'   -b '/boot/grub/i386-pc/eltorito.img' \
                -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info   -eltorito-alt-boot   -e '--interval:appended_partition_2:::' \
                -no-emul-boot -o "iso-deb${system_version}.iso" $ISO_FOLDER
            else
                xorriso -as mkisofs -r -V "Unattended Custom Install" -J -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot\
                -boot-load-size 4 -isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin -boot-info-table -input-charset utf-8 -eltorito-alt-boot\
                -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -o "iso-deb${system_version}.iso" $ISO_FOLDER
            fi
        fi    
    }

Discouraging - I agree, but we'll find our way around it later.

Just to mid-sum it up :

We went and discovered how Linux Distribution ISO is structured.
Learned that we can disassemble it.
Studied to modify the filesystem.
Dwell on how to reassemble same ISO.
Grasped important parts of ISO structs.
Wrote automation with bash script functions

Thus, all is left for us, is assembly of the Tool...
Which we will do in next article
Until that time comes, remember: Do Try To Have Some Fun

When you run after the horse in the rabbit whole

Alex M. Schapelle — Tue, 01 Aug 2023 15:41:32 +0000

Welcome dear reader, I am Silent-Mobius, AKA Alex M. Schapelle, your faithful servitor that continues to lead with examples of learning of new topics with trials and errors.

As mentioned in our previous encounter, I have started to dwell into world of databases, by exploring two main types which are NoSQL and RDBMS, mainly talking about MySQL and Redis. About why I am doing this, we'll explore later, as on this entry we'll mostly discuss the architecture as mentioned before and according to table of content:

Learn Redis With Project	Description	Roadmap Status
Project	Create architecture	We are Here
Application	Create application and Version control
Database	Learn basics of Redis and implement with application
Micro Services	Set all local development env with docker-compose
Migration to K8S	Setup k8s manifests and generate helm chart

Let create a small diagram that will illustrate what we are going to create:

We'll be using docker containers to pack our application, and deploy the packed images into container registry. Within Gilab-CI we'll work on our code, and in case it will pass, it will start new pipeline that will deploy on test network with docker compose the scaled application, Redis-cluster and PostgreSQL-cluster.

There will be configuration files for Redis to store data with SQL server and in term, SQL server with be saving data to volume that will be saved under backed up volume... it seems there will be a lot of configuration to be done, either with Python, Bash or Ansible.

The architecture is inspired by Alex Xu book named System Design Interview

As we can see there are a lot of things to setup, create, manage, provide and deploy, thus here is a list of things that I am going to use:

Language: Python, mostly due to the development can be fast and in this case I'd prefer to focus on learning DB and Cache part more then learning new programming language.
Version Control: Git, the application is not going to be that big, in addition today git is the de facto the best choice for code version control, which is also incorporated within CI
CI : Gitlab-CI, Because my github account got locked and the support suggested to destroy the account but not open it for me, thus they lost me as a customer.
RDBMS: PostGreSQL, mostly to master the automation of main and secondary installation with Python as well as auto-database setup or connection with new instance.
NoSQL: Redis, to learn the caching mechanism provided by Alex's book
Micro-Services: Docker and Docker Compose, for only development purposes, eventually we'll create k8s deployment

Although I've been working as IT professional for 15 years, and have been working on these systems separately, It never occurred to me that I need to connect them by myself, mostly because we used to work with waterfall methodology and guilt, along side new application version was dropped down at us to handle by development team. Our Dev team were cool guys and were always there to help, but lack of knowledge and experience of high-level programming languages made it very hard to determine to whom we should consult for aid.
When idea to learn enough programming to do tasks popped up, though it was there for some time, with it came uncertain choice:

Perl
Ruby
Python
Node.js

Which one to use ?

I chose Ruby which in long run, was somewhat incorrect choice, and moving to python from it was not hard but a very long and confusing journey.
Today I stumble to same dilemma, whether to use Go or Rust, to which I've chosen Go... but this is for another article.

All the other automation I've done, was mostly with shell or Bash, with Linux Built-in tools, which eventually turned into Cloud-init, Puppet andAnsible`, thus burdening myself with SQL or NoSQL, was never the issue for me, to what I somewhat regret.
Thus this is where I start to learn ... and hope you will learn with me. No matter what, Please Do Try To Enjoy

Till next time

Learn Redis Or How DevOps Had To Do Back-End Work

Alex M. Schapelle — Sat, 22 Jul 2023 16:03:01 +0000

Welcome dear reader, I am Silent-Mobius, AKA Alex M. Schapelle, your devoted mechanical puppet that will lead with example of going through the learning of new topic.

As part of trying to understand back-end developers suffering, I have devised a plan to embark onto journey of building python based micro-service application with local development on docker-compose and stage/prod deployment to Kubernetes with helm chart.

Yet, all that we already know how to do. What is missing database interaction, which I would like to get better at. Thus I have decided to learn Redis and to develop with it my home page application on : My Web-Site

As mentioned I have a plan that looks like this:

Learn Redis With Project	Description
Project	Create architecture
Application	Create application and Version control
Database	Learn basics of Redis and implement with application
Micro Services	Set all local development env with docker-compose
Migration to K8S	Setup k8s manifests and generate helm chart

This will be for starters. Hope that this makes you hyped as I am at the moment. Thus follow my example and do try to have some fun.

Manage Calendar & Address Book Micro-services

Alex M. Schapelle — Tue, 28 Mar 2023 21:04:31 +0000

Greetings gentle reader. Gratitude for returning to update your insights on various topics. I am Silent-Mobius, also known as Alex M. Schapelle, walking and babbling computer droid.

Within my short time my of existence, on this blue planet, I have learned that in case you are not paying for product, then you are product yourself... while that being true of my digital existence, much of services I use, are financed by me. Mainly due to AI not being fully trusted.

Much of the services we use today, are seamlessly connected to one or more cloud providers, such as Google or Microsoft. Simple things, such calendar and address book are standardized by corporations which govern our quality of life, and we hook into the network of infinite computing grid of processing power.

When considering applications, almost any endpoint device, such as laptop or phone, can provide client to be used as mail, calendar and so on, yet problem with those are that there is no backing up those endpoint applications... unless once again you use cloud provider... que: enter radicale
Radicale is calendar and phone back-end application that uses CardDav and CalDav protocols. These are extension of the WebDav, a HTTP-based protocol for data manipulation.

Motivation for this effort

For a very long period of time, I wanted to create my own cloud, based on simple services such as mail, calendar, contacts and many others which Google and Microsoft provide so graciously .
Based on my knowledge of Linux OS distributions, with addition of constant search of interesting Open-Source projects, combined with my studies of micro-service architectures, I wish to build eco-system that can enable me, the use of cloud, without using cloud-based providers.
Although this project does not builds cloud serving platform out of thin air, yet it brings one step closer to building essential tools, which will enable us build my own cloud one day in the future, just like GNU and Linux Kernel were combined in early 90's.
This project is an experiment, which will lead me and you gentle reader, to something new, and hopefully, we'll enjoy our journey through it.

Code

You can find environment of deployment that I have created based of the existing projects, mainly nginx for being secured load-balancer serving as a connector to a radicale CalDav/CardDav server. It is all managed with docker compose to be a POC, but also can be swapped with helm charts later to be deployed to K8s. The whole implementation saved at gitlab and you are more then welcome to visit, suggest, open issues and add additional features that did not think about, by myself.... please use Merge Request do to all the above suggested.

Structure

.
├── docker-compose.prod.yml
├── docker-compose.yml
├── gunicorn.conf
├── nginx.conf
├── radicale.conf
├── Dockerfile.cal
├── Dockerfile.https
├── LICENSE
└── README.md

Note: access.txt file, with the password for radicale container, is required to create calendar image.

The application is structured as two container micro-services, one for secure connection with nginx web server that uses https protocol that passes the communication to radicale server that manages the calendar and address book data. It uses external volume backed up on raid 50 storage. Raid 50 is secured storage service provided by local ISP, to which only I and ISP have access.

Each container is build with its dedicated Dockerfile, and every container uses its own dedicated application configuration file. As mentioned, docker compose is in charge of handling the local run.

Conclusion

I hope that this little fun project was an inspiration for you to go on adventure of your own with any software there is for your needs. One last for me to suggest would be: Do Try To Have Some Fun.

Forem: Vaiolabs

JSON Structs with bash

Parsing JSON with Bash Script

1. What is jq?

2. Basic JSON Parsing with jq

Extracting a Single Field

Extracting Multiple Fields

Parsing Nested JSON

3. Filtering JSON Data

Filtering Based on a Condition

4. Working with JSON from an API

5. Modifying JSON Data with jq

Adding a New Field

6. Handling JSON Arrays

7. Combining Multiple Operations

Conclusion

Logging Your Steps

What is Logging ?

Yet what are reason for using logs?

What does bash script has to do with it ?

What is left behind ?

Common Logging Techniques in Shell Scripts

Logging with Timestamps

Logging Standard Output and Error Separately

Using tee for Real-Time Logging

Logging Error Messages Using trap

Built-in Logging Tools in Bash

Log Rotation for logger (Using logrotate)

Best Practices for Logging

As A Conclusion

Amazing `pre-commit` and How To Use It

Introduction

Installation

Using pip

Verify Installation

Configuration

Example Configuration

Usage

Installing Hooks

Running Hooks Manually

Bypassing Hooks

Advanced Topics

Updating Hooks

Customizing Hooks

Using Pre-commit in CI/CD

Troubleshooting

Resources

Conclusion

Bash Your Tools

What's the point of a history lesson ?

How should one do it ?

Conclusion

Bash Script Skills Upgrade For Dev & Ops

Editors

Initials

Safe header

Imports and Sources

Variables

Dynamically generated variable values

Implicit VS. Explicit

Conditions

Functions, variables and in between

Script structure

Summary

Links

Self Made Installer Packages

Who needs it ?

Will it work on complex project with multi environment setup ?

Setup

Synopsis

Example

Let's pack it up

Install script

The Final Count Down

Fin

Menu In Your Shell

The Motivation

Use Cases

Basics

Basic Dialog

1. What is `jq`?

2. Basic JSON Parsing with `jq`

5. Modifying JSON Data with `jq`

Using `tee` for Real-Time Logging

Log Rotation for `logger` (Using `logrotate`)

Using `Pre-commit` in CI/CD