Forem: Danish Khan

🚀 Python for SRE/DevOps: Building SDKs + Jenkins Automations

Danish Khan — Wed, 25 Mar 2026 11:28:20 +0000

A practical, step-by-step roadmap to learn how to:

Build reusable Python SDKs
Package and publish them
Use them in Jenkins pipelines
Automate real-world SRE workflows (alerts, infra, APIs)

🧠 Big Picture Architecture

Python Script → Internal SDK → HTTP API → External System → Jenkins (cron/scheduled execution)

You are learning how to build internal automation platforms, not just scripts.

📚 Phase 1: Python Foundations (Automation-Focused)

What to Learn

Functions, modules, imports
Virtual environments (venv)
JSON & YAML handling
Exception handling
Basic classes & objects

Resources

📘 Automate the Boring Stuff with Python 👉https://automatetheboringstuff.com/
📘 Python Crash Course 👉 https://ehmatthes.github.io/pcc/
🌐 Real Python (Core Concepts) 👉 https://realpython.com/

🌐 Phase 2: APIs First (Before SDKs)

What to Learn

HTTP methods: GET, POST, DELETE
Headers & authentication
JSON request/response
Status codes
Timeouts & Retries

Resources

🌐 MDN HTTP Guide

👉 https://developer.mozilla.org/en-US/docs/Web/HTTP
📦 Python requests Library

👉 https://docs.python-requests.org/
⚡ FastAPI Tutorial (API understanding)

👉 https://fastapi.tiangolo.com/

🧪 Practice

Write a script that:

Calls an API
Fetches data
Sends POST request
Deletes a resource

📦 Phase 3: Python Packaging (VERY IMPORTANT)

What to Learn

setup.py vs pyproject.toml
pip install mechanics
wheel format
package versioning
private package repositories

Resources

📘 Python Packaging User Guide

👉 https://packaging.python.org/
📦 setuptools Documentation

👉 https://setuptools.pypa.io/
📦 Poetry Documentation

👉 https://python-poetry.org/docs/
📦 Twine (Publishing Packages)

👉 https://twine.readthedocs.io/
🌐 Real Python Packaging Guide

👉 https://realpython.com/pypi-publish-python-package/

🧪 Practice

Create a Python package
Build it (python -m build)
Install locally using pip
Import it in another project

🏗️ Phase 4: Building SDK-Style Code

What to Learn

Class-based design
Client abstraction
BaseClient pattern
Error handling
Code structure

Typical SDK Structure

my_sdk/
├── client.py
├── base_client.py
├── models.py
├── exceptions.py
├── utils.py
└── init.py

Resources

📘 Clean Code in Python

👉 https://www.packtpub.com/product/clean-code-in-python/9781788835831
📘 Cosmic Python (Architecture)

👉 https://www.cosmicpython.com/
📘 Fluent Python (Advanced)

👉 https://www.oreilly.com/library/view/fluent-python/9781492056348/

🧪 Practice

Build a client like:

client = MonitoringClient()
client.create_event()
client.get_events()
client.delete_event()

⚙️ Phase 5: Jenkins + Python Integration

What to Learn

Jenkins pipelines (Jenkinsfile)
Running Python scripts
Environment variables
Cron scheduling
Dependency installation (pip in pipeline)

Resources

📘 Jenkins Pipeline Docs

👉 https://www.jenkins.io/doc/book/pipeline/
🎓 KodeKloud Jenkins Course

👉 https://kodekloud.com/courses/jenkins/

🧪 Practice

Create pipeline:

Install package via pip
Run Python script
Capture output
Trigger via cron

🧪 Phase 6: Testing (CRITICAL for SRE)

What to Learn

pytest basics
mocking APIs
testing client methods
safe execution patterns

Resources

📦 pytest Documentation

👉 https://docs.pytest.org/
📘 Python Testing with pytest

👉 https://pragprog.com/titles/bopytest/python-testing-with-pytest/
🌐 Real Python Testing Guide

👉 https://realpython.com/pytest-python-testing/

⚡ Phase 7: Async Python (Advanced)

What to Learn

async / await
event loop
concurrent API calls
async HTTP clients

Resources

🌐 Real Python Async Guide

👉 https://realpython.com/async-io-python/
📘 Python asyncio Docs

👉 https://docs.python.org/3/library/asyncio.html
📦 HTTPX (async HTTP client)

👉 https://www.python-httpx.org/

🧱 Real-World Pattern You Are Learning

This pattern is used everywhere in platform engineering:

Internal Python SDK
Published as package
Stored in private repo
Installed in pipelines
Used for automation tasks
Triggered via cron

🔥 Mini Project (Must Do)

Project: Monitoring Automation SDK

Structure

monitoring-sdk/
├── monitoring_client/
│ ├── client.py
│ ├── base_client.py
│ ├── models.py
│ ├── exceptions.py
│ └── utils.py
├── scripts/
│ ├── create_event.py
│ ├── cleanup_event.py
├── tests/
├── pyproject.toml
└── Jenkinsfile

Features

Create event (POST)
List events (GET)
Delete event (DELETE)
Package SDK
Install via pip
Run via Jenkins

📅 7-Day Fast-Track Plan

Day 1

Learn venv, pip
Run Python scripts

Day 2

Learn requests
Call an API

Day 3

Write API client class

Day 4

Convert to package

Day 5

Install package locally

Day 6

Write automation script

Day 7

Run via Jenkins pipeline

🧠 Final Insight

You are not learning Python.
You are learning to build:
👉 Internal Developer Platforms
👉 Automation of Toolings,API and Systems
👉 Reusable Engineering Tools used internally and are created using Python

This is exactly what strong SREs/DevOps and Platform Engineers do.
This is how in future you can even contribute to Open Source Software

🚀 Next Step : Coming Soon (WIP)

👉 A complete working project template (SDK + Jenkins + API mock).
👉 Tailored to this above workflow.

Just share it further on LinkedIn, Substack 👍

Bash Scripting for Non-Coders

Danish Khan — Sat, 06 Dec 2025 15:03:27 +0000

A Practical, No-Fear Guide to Understanding the Shell

Learning Bash can feel intimidating if you don’t come from a programming background. Terminals look cryptic, commands seem unforgiving, and even simple tasks appear wrapped in strange symbols like |, >, and &&.
But here’s the truth:

You don’t need to be a coder to write useful Bash scripts.
You just need to understand how Bash works and how you can think in bash.

This guide is written specifically for non-coders, beginners, and anyone who feels “new” to the command line.
Instead of overwhelming you with theory, we’ll build intuition step-by-step using simple explanations, real-world analogies, and tiny hands-on examples. By the end, you’ll have the confidence to automate tasks, avoid mistakes, and write scripts that actually work.

Whether you’re a DevOps engineer, a sysadmin, a student, or someone simply curious about the terminal—this guide will help you move from:

❌ “I don’t understand Bash…” to
✅ “I can write scripts that solve real problems.”

Let’s begin your journey into Bash—no coding background required.

Piping & Re-direction

Re-direction in Bash

Stream	Name	Content
0	Standard Input (stdin)	keyboard or other input
1	Standard Output (stdout)	Regular Output
2	Standard Error (stderr)	Output marked as 'error'

Re-direction in file

Symbol	Function
`>`	Output redirection (truncate)
`>>`	Output redirection (append)
`<`	Input redirection
`<<`	Here documentation

Commands and Built-In

echo ends a line in a new line
Likewise is printf, but it leaves the exit on the same line.
Check if something is command or a builtin

danish@danish-Ubuntu:~$ command -V echo
echo is a shell builtin
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ command -V df 
df is /usr/bin/df
danish@danish-Ubuntu:~$

Built-Ins take precedents over commands. Bash will run a builtin if it has one, even if there's a command of the same name available on the system. But specific builtins can be disabled in a session with enable -n and the name of the builtin, allowing the command version to run instead.
For eg. We can disable echo builtin via below command.

danish@danish-Ubuntu:~$ enable -n echo 
danish@danish-Ubuntu:~$ command -V echo
echo is /usr/bin/echo
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ enable -n
enable -n echo

As shown above enable -n by itself shows builtins that are disabled.
We can re-enable the builtin with enable and the name of the builtin.

danish@danish-Ubuntu:~$ enable echo
danish@danish-Ubuntu:~$ command -V echo
echo is a shell builtin
danish@danish-Ubuntu:~$

Another important distinction to make here is that builtins use a different documentation system than the regular man pages.
There's a builtin called help that shows supporting information about builtins.
For example, we can write help echo, and find out what we can do with the echo builtin.

danish@danish-Ubuntu:~$ help echo
echo: echo [-neE] [arg ...]
    Write arguments to the standard output.

    Display the ARGs, separated by a single space character and followed by a
    newline, on the standard output.

    Options:
      -n    do not append a newline
      -e    enable interpretation of the following backslash escapes
      -E    explicitly suppress interpretation of backslash escapes

    `echo' interprets the following backslash-escaped characters:
      \a    alert (bell)
      \b    backspace
      \c    suppress further output
      \e    escape character
      \E    escape character
      \f    form feed
      \n    new line
      \r    carriage return
      \t    horizontal tab
      \v    vertical tab
      \\    backslash
      \0nnn the character whose ASCII code is NNN (octal).  NNN can be
            0 to 3 octal digits
      \xHH  the eight-bit character whose value is HH (hexadecimal).  HH
            can be one or two hex digits
      \uHHHH    the Unicode character whose value is the hexadecimal value HHHH.
            HHHH can be one to four hex digits.
      \UHHHHHHHH the Unicode character whose value is the hexadecimal value
            HHHHHHHH. HHHHHHHH can be one to eight hex digits.

    Exit Status:
    Returns success unless a write error occurs.
danish@danish-Ubuntu:~$

Using the help built-In, we can display the names of all the other builtins.
Here's the list of the builtins that Bash provides.

danish@danish-Ubuntu:~$ help
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
These shell commands are defined internally.  Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.

A star (*) next to a name means that the command is disabled.

 job_spec [&]                                                                                         history [-c] [-d offset] [n] or history -anrw [filename] or history -ps arg [arg...]
 (( expression ))                                                                                     if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMANDS; ]... [ else COMMANDS; ] fi
 . filename [arguments]                                                                               jobs [-lnprs] [jobspec ...] or jobs -x command [args]
 :                                                                                                    kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
 [ arg... ]                                                                                           let arg [arg ...]
 [[ expression ]]                                                                                     local [option] name[=value] ...
 alias [-p] [name[=value] ... ]                                                                       logout [n]
 bg [job_spec ...]                                                                                    mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [arr>
 bind [-lpsvPSVX] [-m keymap] [-f filename] [-q name] [-u name] [-r keyseq] [-x keyseq:shell-comman>  popd [-n] [+N | -N]
 break [n]                                                                                            printf [-v var] format [arguments]
 builtin [shell-builtin [arg ...]]                                                                    pushd [-n] [+N | -N | dir]
 caller [expr]                                                                                        pwd [-LP]
 case WORD in [PATTERN [| PATTERN]...) COMMANDS ;;]... esac                                           read [-ers] [-a array] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u >
 cd [-L|[-P [-e]] [-@]] [dir]                                                                         readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [a>
 command [-pVv] command [arg ...]                                                                     readonly [-aAf] [name[=value] ...] or readonly -p
 compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat] [-W wordlist]  [-F function] [-C comm>  return [n]
 complete [-abcdefgjksuv] [-pr] [-DEI] [-o option] [-A action] [-G globpat] [-W wordlist]  [-F func>  select NAME [in WORDS ... ;] do COMMANDS; done
 compopt [-o|+o option] [-DEI] [name ...]                                                             set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
 continue [n]                                                                                         shift [n]
 coproc [NAME] command [redirections]                                                                 shopt [-pqsu] [-o] [optname ...]
 declare [-aAfFgilnrtux] [-p] [name[=value] ...]                                                      source filename [arguments]
 dirs [-clpv] [+N] [-N]                                                                               suspend [-f]
 disown [-h] [-ar] [jobspec ... | pid ...]                                                            test [expr]
*echo [-neE] [arg ...]                                                                                time [-p] pipeline
 enable [-a] [-dnps] [-f filename] [name ...]                                                         times
 eval [arg ...]                                                                                       trap [-lp] [[arg] signal_spec ...]
 exec [-cl] [-a name] [command [arguments ...]] [redirection ...]                                     true
 exit [n]                                                                                             type [-afptP] name [name ...]
 export [-fn] [name[=value] ...] or export -p                                                         typeset [-aAfFgilnrtux] [-p] name[=value] ...
 false                                                                                                ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]
 fc [-e ename] [-lnr] [first] [last] or fc -s [pat=rep] [command]                                     umask [-p] [-S] [mode]
 fg [job_spec]                                                                                        unalias [-a] name [name ...]
 for NAME [in WORDS ... ] ; do COMMANDS; done                                                         unset [-f] [-v] [-n] [name ...]
 for (( exp1; exp2; exp3 )); do COMMANDS; done                                                        until COMMANDS; do COMMANDS; done
 function name { COMMANDS ; } or name () { COMMANDS ; }                                               variables - Names and meanings of some shell variables
 getopts optstring name [arg]                                                                         wait [-fn] [id ...]
 hash [-lr] [-p pathname] [-dt] [name ...]                                                            while COMMANDS; do COMMANDS; done
 help [-dms] [pattern ...]                                                                            { COMMANDS ; }

We can find echo and enable, which we saw earlier and so on. To explore more about these, you can write help and any of these names.
Bash builtins are useful at the command line and also in scripts, but it's important to know the difference between builtins and commands.

Bash Expansions & Substitutions

When we're working with Bash, either at the command line or in the script, we'll often need to use values that we don't know.
Things like a path to the user's home folder, a piece of user provided information, or the result of a calculation that's based on something we can't foresee.
Bash provides us a way to represent these values using expansions and substitutions.
Both of these are interpreted when they run and replace themselves with a value or a set of values.
Because they're very important to scripting, lets explore them.

Representation	Name
`~`	Tilde expansion
`{...}`	Brace expansion
`${...}`	Parameter expansion
`$(...)`	Command substitution
`$((...))`	Airthmetic expansion

Tilde Expansion `~`

In Bash, the tilde character represents the value of the user's $HOME variable. And it's used in paths to represent the current user's $HOME directory. We can see that here with echo and the tilde character.

danish@danish-Ubuntu:~$ echo ~
/home/danish

Tilde expansion is useful when we're scripting, because if we want to do something in a user's $HOME directory, which is a pretty important place, we can do that without knowing their username.
Tilde just represents whatever the current user's $HOME directory is set to. So, here on my system, that's /home/danish.
But on your system, the same tilde represents home and whatever is your username.
Another useful tilde expansion trick is that the tilde followed by a dash or minus represents a Bash variable called OLDPWD.
Which is the directory that you were just in, if you've recently changed directories. You can use this to work with other directories in the directory stack. Which is useful for command line use, but not as much for scripting.

danish@danish-Ubuntu:~$ pwd
/home/danish
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ~-
/sys/fs/cgroup/freezer/mycgroup
danish@danish-Ubuntu:~$

For info take a look at the tilde expansion section of the Bash Man pages.

Brace Expansion `{...}`

Another useful type of expansion is called brace expansion. This is written with braces around an expression, and it lets us substitute in items from a list of values separated by commas, or ranges of numbers or letters in a given pattern, separated by two dots or periods.

Brace expansion is often used when we need to keep part of a path the same, but replace a little piece of it.
For example, if we wanted to create a file inside each of three different directory trees where only part of the path needs to change each time, or we could use it to provide a set of values to use in the same part of a string more generally. The other form of brace expansion, which creates a sequence of numbers or letters, is often used when working with sequential items.

danish@danish-Ubuntu:~$ echo /tmp/{one,two,three}/file.txt
/tmp/one/file.txt /tmp/two/file.txt /tmp/three/file.txt
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo c{a,s,d}t
cat cst cdt
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo /tmp/{1..3}/file.txt
/tmp/1/file.txt /tmp/2/file.txt /tmp/3/file.txt
danish@danish-Ubuntu:~$

For example, We can write something like echo {1..10} to get the numbers one through 10, or echo {10..1} to get the same numbers but in reverse order.

danish@danish-Ubuntu:~$ echo {1..10}
1 2 3 4 5 6 7 8 9 10
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo {10..1} 
10 9 8 7 6 5 4 3 2 1
danish@danish-Ubuntu:~$

I can generate letters, too. I'll write echo {a..z} and echo {Z..A}. With either kind of sequence.

danish@danish-Ubuntu:~$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo {A..Z} 
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
danish@danish-Ubuntu:~$

We can add an interval, too. For example, echo {1..50..2} will return every second number between 1 and 50.

danish@danish-Ubuntu:~$ echo {1..50..2} 
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
danish@danish-Ubuntu:~$

Below command will return every letter between a and z leaving the second letter after each, like it left b, then d then f and so on.

danish@danish-Ubuntu:~$ echo {a..l..2}
a c e g i k
danish@danish-Ubuntu:~$

Brace expansion can also be used to work with set lists of things. For example, I can write echo, tree,flower,bee, and that returns what we might expect. As we saw before, we can combine it with other expansions.

danish@danish-Ubuntu:~$ echo {tree,flower,bee}
tree flower bee
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo {tree,flower,bee}_{1..3}
tree_1 tree_2 tree_3 flower_1 flower_2 flower_3 bee_1 bee_2 bee_3
danish@danish-Ubuntu:~$

We'll often find this used for directories or when working with files that have a predefined naming scheme, and brace expansion can be used as kind of a shorthand when working with commands, too.

Parameter expansion `${...}`

Parameter expansion lets us recall stored values and transform them in various ways. This is usually represented by a dollar sign and a set of braces. Though, sometimes we'll see it without the braces.
The most straightforward use of parameter expansion is setting a value and then retrieving it, as we'll do when we make use of variables and scripting later on.
When we're using parameter expansion in this way, we'll set a parameter equal to a value, and then use the dollar sign with the name of that parameter to retrieve the value later.
Parameter expansion also often features braces. Those are used to make it clear what parameter is being used and to keep the shell from getting confused about nearby words or characters.
They're also used for extracting substrings or manipulating the value of a parameter as it's being used.

danish@danish-Ubuntu:~$ a="my parameter"
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo $a
my parameter
danish@danish-Ubuntu:~$

So the string is:

m  y     p  a  r  a  m  e  t  e  r
0  1 2   3  4  5  6  7  8  9  10 11 12

Spaces count as characters too.

danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ${a}
my parameter
danish@danish-Ubuntu:~$

Start at index 1, extract 5 characters.
Indexing starts at 0, so:

Index 0 = m
Index 1 = y
Next 4 chars = par

danish@danish-Ubuntu:~$ echo ${a:1:5}
y par
danish@danish-Ubuntu:~$

String Replacement: ${var/pattern/replacement}

danish@danish-Ubuntu:~$ a="my parameter"
danish@danish-Ubuntu:~$ echo ${a}
my parameter
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ${a/parameter/Second Entry}
my Second Entry
danish@danish-Ubuntu:~$

Replaces ONLY the first occurrence of pattern.
Summary:-

Expression	Meaning	Example Output
`${a}`	Value of variable	`my parameter`
`${a:start:length}`	Substring extraction	`${a:1:5}` → `y par`
`${a/pattern/repl}`	Replace first match	`${a/parameter/x}` → `my x`
`${a//pattern/repl}`	Replace ALL matches	`${a//a/x}` → `my parameter`
`${a/pattern}`	Delete first match	`${a/para}` → `meter` (removes `para`)
`${#a}`	String length	`13`

In a script, we just use the dollar sign parameter part, usually without echo. But here in the shell, we will need to use echo to see what the value of that parameter is.
Parameter expansion can also be used to manipulate the stored value when it's used, transforming it in various ways, like just using part of the value or replacing certain parts before the value is used. For example as shown above.
Let's replace all of the E's with an underscore. I'll write echo parameter//e/_.

danish@danish-Ubuntu:~$ echo $a
my parameter
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ${a//e/_}
my param_t_r
danish@danish-Ubuntu:~$

Let's do the same thing with just one slash, and we can see the difference.

danish@danish-Ubuntu:~$ echo $a
my parameter
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ${a/e/_}
my param_ter
danish@danish-Ubuntu:~$

We'll use braces for most parameter expansions, because if we left them off, the shell would just interpret the part after the parameter name as characters and show them instead of using them to do what we intend.
For example, I'll write echo $a:4:3 without the braces. And here, I see the string my parameter :4:3. As we'll see though, parameter expansion without the braces is common.Bash only performs substring extraction when you use ${variable:start:length} format.

danish@danish-Ubuntu:~$a="my parameter"
danish@danish-Ubuntu:~$ echo $a
my parameter
danish@danish-Ubuntu:~$ echo $a:2:5
my parameter:2:5
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo ${a:2:5}
para
danish@danish-Ubuntu:~$

Command Substitution `$(...)`

Command substitution is a kind of special case expansion which allows us to use the output of a command within another command.
This is represented by a dollar sign and a set of parentheses enclosing a command.
An alternate way of writing this is to use two back ticks, but that often gets confusing. So generally, we write it with the parentheses.
Bash runs the specified command in a subshell and returns the output of that into the current command. It's often used together with string manipulation tools to extract a part of a command's output, such as a path, a file size, an IP address, or so on, that needs to be handed back up to the parent command.
Let's take a look at using command substitution. For example, we could use uname -r to get the release version of the kernel. And with command substitution, we could use that in an echo statement.

danish@danish-Ubuntu:~$ uname -r
5.15.0-91-generic
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "The kernel version is $(uname -r)."
The kernel version is 5.15.0-91-generic.
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "The GIT version is $(git --version)."
The GIT version is git version 2.25.1.
danish@danish-Ubuntu:~$

We'll often use something like this in scripts to get the version of something the user has installed or some metrics about the system that might be relevant to how our script works. We can use longer commands within the parentheses too, including command pipelines, and you can nest command substitutions within one another in case you have a complex series of commands that depend on the results of other commands. Here, I'm using python to print out a string, and using tr to transform it to all caps. And then the result of that is being used in my echo statement.

danish@danish-Ubuntu:~$ python -c print'("Hello from Python.")'
Hello from Python.
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "Result: $(python -c print'("Hello from Python.")' |tr [a-z] [A-Z])"
Result: HELLO FROM PYTHON.
danish@danish-Ubuntu:~$

See how it's useful to capture the output of a command and use it in another command.
Command substitution is often used with tools such as GREP, AWK, and CUT, and is widely used to determine whether a script has what it needs on the target system.

Arithmetic expansion `$((...))`

Bash can use arithmetic expansion to perform arithmetic or calculations and use the result in a command.
This is represented by a dollar sign and two sets of parentheses with an arithmetic expression inside.
Earlier versions of Bash also used a dollar sign with single brackets, but this is deprecated now.
You may see it in older scripts though. Let's do a couple of basic calculations here.
Let's take a look at two plus two, and that's four.
Expansions and substitutions are really important features of Bash, and its up to the user how he/she intended to use them in scripts.

danish@danish-Ubuntu:~$ echo $(( 5+5 ))
10
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo $(( 5-3 ))
2
danish@danish-Ubuntu:~$ echo $(( 10-9 ))
1
danish@danish-Ubuntu:~$

Programming/Scripting in Bash

Running a program in bash

It can be one liners ( Series of bash commands separated via semicolon or pipes)
It can be alias of a long command

Bash Script

Text commands that uses a series of commands
Run it via bash script.sh #### Executable Bash Script
Includes a shebang at the start

#!/usr/bin/env bash

A shebang line starts with a pound or hash sign and an exclamation mark, and then the full path to whatever program should run the script.
In this case, we'll use Bash but this can be anything like Python,Ruby or Perl whatever interpreted scripting language you need to make into a program.
It's common to see the absolute path to Bash in a shebang line in older script like, /bin/bash it's often considered a best practice to write the shebang line in a way that uses the shell's environment to locate the Bash executable.
Not all systems have Bash in the same place though almost all of them do. And so instead of asking for /bin/bash and not finding it in that file path, we ask the environment to give us Bash which will work wherever Bash is installed.
If we leave off a shebang, the contents of the script file will be passed to the current shell on that Linux system which mostly will be Bash anyway but we don't know what shell other people are using at any given time, so it makes sense to include the shebang.
Writing shebang guarantee that our Bash script is run by Bash.
When creating an executable script will usually leave the file extension off, but that's entirely up to you.
The second step of making an executable script is to make it executable. You'll do that with chmod + X scriptname.
Now I can run this by typing ./scriptname and if I were to place this file somewhere the shell could find it somewhere in the path. I can just use the name of the script like a regular command.
A Bash script runs inside of a non interactive shell, which means that many of the customizations a user might have set for their Bash environment won't be read when the script is run. Unlike when we use a regular interactive shell.
This gives the script a fairly clean environment in which to run, and if we want to make any customizations or specifically override other settings we'll need to use set or shopt in our script to turn on or off configurations accordingly. In most cases, we don't need to do this but it's something that's available if necessary. Writing Bash scripts allows us to avoid typos, to streamline work using Bash and they allow us to distribute our Bash code to other systems that need to run it and one-liners are useful as well.

Using echo in Bash

Various ways of using the echo is shown below

danish@danish-Ubuntu:~$ myvariable=dkvalue1
danish@danish-Ubuntu:~$ echo $myvariable 
dkvalue1
danish@danish-Ubuntu:~$ echo $myvariable : printing the value
dkvalue1 : printing the value
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo Now $myvariable : printing the value
Now dkvalue1 : printing the value
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "The kernel is $(uname -a)"
The kernel is Linux danish-Ubuntu 5.15.0-91-generic #101~20.04.1-Ubuntu SMP Thu Nov 16 14:22:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "The kernel is $(uname -r)"
The kernel is 5.15.0-91-generic
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo The kernel is $(uname -r)
The kernel is 5.15.0-91-generic
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo The( kernel) is $(uname -r)
bash: syntax error near unexpected token `('
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo The (kernel) is $(uname -r)
bash: syntax error near unexpected token `('
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo The \(kernel\) is $(uname -r)
The (kernel) is 5.15.0-91-generic
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo "The kernel is $(uname -r)"
The kernel is 5.15.0-91-generic
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo 'The kernel is $(uname -r)'
The kernel is $(uname -r)
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo 'The kernel is $(uname -r)'
The kernel is $(uname -r)
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo 

danish@danish-Ubuntu:~$ echo -n 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo -n "Disabling the Cursor for exiting to new line" 
Disabling the Cursor for exiting to new line 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$

Working with Variables in Bash

In Bash, variables are an expression of parameter expansion.
In Bash, parameters, or in this case, variables are named with alpha numeric characters, and we assign a value to that name with an equal sign followed by whatever value we want it to store.
It's important to remember that there should be no space on either side of the equal sign.
Variable names are case sensitive. In many cases, we can use lowercase names for our variables. Though you can use capitalized or all caps names too.
Using lowercase variables helps us to distinguish our variables from environment and system variables, which are nearly always upper case.
A variable could also contain a number. So to use these variables, we refer to them with a dollar sign in front of their name.
Remember, this is the sort of special case parameter expansion where we don't need to use the braces, but we can if we want to, or if we need to use only part of that string value. These variables will be available in that shell session until it closes.
Example of a script for the same

#!/usr/bin/env bash
myvar="Hello!"
echo "The value of the myvar variable is: $myvar"
myvar="Bonjour!"
echo "The value of the myvar variable is: $myvar"

declare -r myname=bob
echo "The value of the myname variable is: $myname"
myname="Iceman"
echo "The value of the myname variable is: $myname"

declare -l lowerstring="This is some TEXT!"
echo "The value of the lowerstring variable is: $lowerstring"
lowerstring="Let's CHANGE THE VALUE"
echo "The value of the lowerstring variable is: $lowerstring"

declare -u upperstring="This is some TEXT!"
echo "The value of the upperstring variable is: $upperstring"
lowerstring="Let's CHANGE THE VALUE"
echo "The value of the upperstring variable is: $upperstring"

Now we make the script executable and then run the script

danish@danish-Ubuntu:~$ ./variables_check.sh
The value of the myvar variable is: Hello!
The value of the myvar variable is: Bonjour!
The value of the myname variable is: bob
./variable-decleration_script.sh: line 9: myname: readonly variable
The value of the myname variable is: bob
The value of the lowerstring variable is: this is some text!
The value of the lowerstring variable is: let's change the value
The value of the upperstring variable is: THIS IS SOME TEXT!
The value of the upperstring variable is: THIS IS SOME TEXT!

What is happening at line 9?
Script declares:

declare -r myname=bob

-r means:
myname becomes a readonly variable

In Bash, a readonly variable cannot be changed, reassigned, or unset after its declaration.

This is equivalent to:

readonly myname=bob

So the value is permanently fixed to:

bob

Line 9 tries to REASSIGN a readonly variable.
This triggers:

line 9: myname: readonly variable

Q: Why does Bash have readonly variables?
Readonly variables are used to:

protect constants
avoid accidental changes
enforce configuration values
lock down environment variables
write safer scripts

Example:

readonly APP_ENV=production

Now nothing can override it—even if someone tries.

Bash `declare` Options (Professional Cheat Sheet)

Variable Attributes

Option	Meaning	Example	What Happens
`-r`	Readonly (cannot reassign)	`declare -r name="bob"`	`name` cannot be changed
`-x`	Export (becomes environment var)	`declare -x PATH`	Available to child processes
`-i`	Integer (auto-coerces to number)	`declare -i n=5+7`	`n` becomes `12`
`-l`	Lowercase (auto-converts assigned values)	`declare -l var="HeLLo"`	`var = hello`
`-u`	Uppercase (auto-converts assigned values)	`declare -u var="HeLLo"`	`var = HELLO`

Arrays

Option	Meaning	Example
`-a`	Indexed array	`declare -a arr=(a b c)`
`-A`	Associative array (hashmap)	`declare -A config=([env]=prod [ver]=1.0)`

Function Options (Rare)

Option	Meaning
`-f`	List function definitions
`-F`	List only function names

Visibility / Declaration

Option	Meaning
`-p`	Print variable declaration in reusable form

Let’s go through what matters most for real SRE Bash scripting.

1️⃣ -r — Readonly

Used when you want constants.

declare -r APP_ENV=prod
APP_ENV="dev"   # ERROR: readonly variable

This prevents accidental overwrites.

2️⃣ -i — Integer

Bash will automatically evaluate arithmetic expressions:

declare -i num=5+8
echo "$num"        # 13


Even if you try to assign a string:

num="10+20"
echo "$num"        # 30

Useful for counters, sums, CPU monitoring scripts, etc.

3️⃣ -l — Lowercase auto-conversion

declare -l lower="HeLLo WoRLD"
echo "$lower"      # hello world

lower="NEW VALUE"  # becomes: new value

Great for normalizing user input.

4️⃣ -u — Uppercase auto-conversion

declare -u upper="HeLLo WoRLD"
echo "$upper"      # HELLO WORLD

Good for case-insensitive comparisons.

5️⃣ -x — Export variable to environment

declare -x token="ABC123"
bash -c 'echo $token'   # available here too

Used heavily in:

CI scripts
wrapper scripts
Kubernetes entrypoint logic

6️⃣ -a — Indexed Array

declare -a arr=(apple banana cherry)
echo "${arr[1]}"     # banana

7️⃣ -A — Associative Array (HashMap)

declare -A config=(
  [env]=prod
  [app]="backend"
  [port]=8080
)

echo "${config[env]}"   # prod
echo "${config[app]}"   # backend

Used in:

config parsing.
key-value mappings.
JSON-like structures in Bash.

*One of the things that's useful about declare is that you can use declare -p to see all of the variables that have been set in a current session.

So if you're working at the command line, and don't remember what you called that one variable from a little while ago, declare -p will remind you.

danish@danish-Ubuntu:~$ declare -p
declare -- BASH="/usr/bin/bash"
declare -r BASHOPTS="checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote:force_fignore:globasciiranges:histappend:interactive_comments:progcomp:promptvars:sourcepath"
declare -i BASHPID
declare -A BASH_ALIASES=()
declare -a BASH_ARGC=([0]="0")
declare -a BASH_ARGV=()
declare -- BASH_ARGV0
declare -A BASH_CMDS=()
declare -- BASH_COMMAND
declare -a BASH_COMPLETION_VERSINFO=([0]="2" [1]="10")
declare -a BASH_LINENO=()
declare -ar BASH_REMATCH=()
declare -a BASH_SOURCE=()
declare -- BASH_SUBSHELL
declare -ar BASH_VERSINFO=([0]="5" [1]="0" [2]="17" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
declare -- BASH_VERSION="5.0.17(1)-release"
declare -x COLORTERM="truecolor"
declare -- COLUMNS="203"
declare -- COMP_WORDBREAKS="    
\"'><=;|&(:"
declare -x DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"
declare -x DESKTOP_SESSION="ubuntu"
declare -a DIRSTACK=()
declare -x DISPLAY=":1"
declare -- EPOCHREALTIME
declare -- EPOCHSECONDS
declare -ir EUID="1000"
declare -a FUNCNAME
declare -x GDMSESSION="ubuntu"
declare -x GJS_DEBUG_OUTPUT="stderr"
declare -x GJS_DEBUG_TOPICS="JS ERROR;JS LOG"
declare -x GNOME_DESKTOP_SESSION_ID="this-is-deprecated"
declare -x GNOME_SHELL_SESSION_MODE="ubuntu"
declare -x GNOME_TERMINAL_SCREEN="/org/gnome/Terminal/screen/a9a1eca8_cb8a_4d39_9fff_b62f623d8f59"
declare -x GNOME_TERMINAL_SERVICE=":1.102"
declare -x GPG_AGENT_INFO="/run/user/1000/gnupg/S.gpg-agent:0:1"
declare -a GROUPS=()
declare -x GTK_MODULES="gail:atk-bridge"
declare -i HISTCMD
declare -- HISTCONTROL="ignoreboth"
declare -- HISTFILE="/home/danish/.bash_history"
declare -- HISTFILESIZE="2000"
declare -- HISTSIZE="1000"
declare -x HOME="/home/danish"
declare -- HOSTNAME="danish-Ubuntu"
declare -- HOSTTYPE="x86_64"
declare -- IFS="    
"
declare -x IM_CONFIG_PHASE="1"
declare -x INVOCATION_ID="a31e8df6411e421289f2185137ec2b39"
declare -x JOURNAL_STREAM="8:49873"
declare -x LANG="en_IN"
declare -x LANGUAGE="en_IN:en"
declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
declare -x LESSOPEN="| /usr/bin/lesspipe %s"
declare -i LINENO
declare -- LINES="53"
declare -x LOGNAME="danish"

There's a lot of variables we didn't explicitly set in here, and these are from the environment. We can use these environment variables in our scripts when we need to get a particular piece of information about the end user system.
Variables are an important part of batch scripting, and unless you need to apply an attribute to a variable, you can skip using declare, and just define a variable the shorter way.
But for more strict scenarios, keep declare in mind. It can help you avoid certain kinds of bugs that would otherwise be tricky to work through.
Use declare when you need to set specific attributes for a variable, such as making it an array or read-only.
For regular variable assignments, especially when the type is straightforward, you can simply assign values without using declare.
This keeps your scripts cleaner and more readable for simple cases.

Working with Numbers in Bash

When we work with numbers in Bash we have two tools at our disposal, arithmetic expansion $((...)) or older representation $[...] and arithmetic evaluation ((...)).
Arithmetic expansion lets us use literal numbers or variables and perform mathematical operations on them. Giving us a result we can display or assign to a variable.
Arithmetic evaluation makes changes to the value of an existing variable.
Bash supports six basic arithmetic operators:-

Operation	Operator
Addition	`+`
Subtraction	`-`
Multiplication	`*`
Division	`/`
Modulo	`%`
Exponentiation	`**`

And it's important to point out Bash can only do calculations with integers.
Arithmetic Expansion $((...)) examples given below:-

danish@danish-Ubuntu:~$ echo $((4+4))
8
danish@danish-Ubuntu:~$ echo $((8-5))
3
danish@danish-Ubuntu:~$ echo $((2*3))
6
danish@danish-Ubuntu:~$ echo $((8/4))
2
danish@danish-Ubuntu:~$ echo $(( (3+6) - 5 * (5-2) ))
-6

Arithmetic evaluation is done in double parentheses but without a leading dollar sign it can change the value of the variable involved.
As you can see in below exapmple Airthmatic evaluation doesn't show any output, instead we can check it's exit code.
Arithmetic Evaluation ((...)) examples given below, in this it shows how to change the value of a numeric variable

danish@danish-Ubuntu:~$ a=10
danish@danish-Ubuntu:~$ (($a+=5))
bash: ((: 10+=5: attempted assignment to non-variable (error token is "+=5")
danish@danish-Ubuntu:~$ ((a+=5))
danish@danish-Ubuntu:~$ echo $?
0
danish@danish-Ubuntu:~$
danish@danish-Ubuntu:~$ echo $a
15
danish@danish-Ubuntu:~$

We can add space inside the parentheses to make the expression easier to read.

danish@danish-Ubuntu:~$ echo $(( (10+6) - 6 * (5*5) )) 
-134
danish@danish-Ubuntu:~$

Below shows that a should be set equal to itself plus three.
Then increment by 1 by ++, then echoing it to check the updated value.

danish@danish-Ubuntu:~$ a=3
danish@danish-Ubuntu:~$ ((a+=3))
danish@danish-Ubuntu:~$ echo $a
6
danish@danish-Ubuntu:~$ ((a++))
danish@danish-Ubuntu:~$ echo $a
7
danish@danish-Ubuntu:~$ ((a++))
danish@danish-Ubuntu:~$ echo $a
8
danish@danish-Ubuntu:~$ ((a--))
danish@danish-Ubuntu:~$ echo $a
7

We can't increment seven and then have it stored back in the variable that got expanded because the reference to the variable is gone.

danish@danish-Ubuntu:~$ (($a++))
bash: ((: 7++: syntax error: operand expected (error token is "+")
danish@danish-Ubuntu:~$

Instead, without using the dollar sign we tell Bash to increment whatever is in that variable without returning us the value first. We can also use combination assignments in this way to reassign the value of the variable within the expression. This provides us a shorthand way of changing the value of a numeric variable, shorter than saying A equals A plus two.

danish@danish-Ubuntu:~$ ((a++))
danish@danish-Ubuntu:~$ echo $a
8

Using these short hands can help us reduce typos and the need to type out the variable name twice. But it also protects us from a problem that can easily occur when working with numbers in Bash.
If I take a value z=10 and try to add to it using what looks like a pretty normal way of doing that as shown below:-

danish@danish-Ubuntu:~$ z=10
danish@danish-Ubuntu:~$ echo $z
10
danish@danish-Ubuntu:~$ y=$z+5
danish@danish-Ubuntu:~$ echo $y
10+5
danish@danish-Ubuntu:~$

The shell's happy to do that operation but we get a result that I might not expect i.e. y = 10+5
Issue is --> Bash is treating this variable as a string rather than a number, so it put the characters plus(+) and 5 at the end of it instead of adding two to the number.
In addition to using arithmetic evaluation to work with numbers safely, we can prevent Bash from treating numbers like strings by declaring variables we'll use for numbers as integers.
Resolution --> We will declare the variable z=10 and we can treat this variable just like we did the other one but now if we wanted to add to it or to work with it Bash treats it like a number instead of a string.
So while it's possible to define numeric variables the same way we do with string variables it's a better idea to use declared -i value on variables you know we'll be integers:-

danish@danish-Ubuntu:~$ a=8
danish@danish-Ubuntu:~$ a=$a+2
danish@danish-Ubuntu:~$ echo $a
8+2
### Problem discussed above
### Solution given below
danish@danish-Ubuntu:~$ declare -i b=3
danish@danish-Ubuntu:~$ b=$b+3
danish@danish-Ubuntu:~$ echo $b
6

The limitation of Bash only being able to work with integers is something we'll run up against in the real world pretty quickly.
For example, if we tried to make this calculation one divided by three the result I get back is zero, not 0.3333 and so on.
And obviously that decimal part would be pretty important in a lot of situations. One third is not zero, but Bash says it is because the integer part, the whole number of the result is zero.
So to do anything that requires more precision anything that needs a floating point or a decimal to be involved we'll need to use tools other than Bash. A few common commands can do calculations including bc short for basic calculator and awk. Both bc and awk are pre-installed on many Linux distributions.
Let's take a very quick look at how to use bc with Bash.

danish@danish-Ubuntu:~$ echo $((1/3))
0
danish@danish-Ubuntu:~$ declare -i c=1
danish@danish-Ubuntu:~$ declare -i d=3
danish@danish-Ubuntu:~$ e=$(echo "scale=3; $c/$d" | bc)
danish@danish-Ubuntu:~$ echo $e
.333

It's important to remember that this response 0.333 is treated as text by Bash not as a number, it's just the characters period. 0.333, not the numeric value, one third.
Addtional information a variable that might come hanndy. That is RANDOM.
RANDOM variable returns, a pseudo random integer between zero 0 and 32767.
We can write echo, dollar sign, RANDOM (all in caps), and see the output it geneates.

danish@danish-Ubuntu:~$ echo $RANDOM
8190
danish@danish-Ubuntu:~$ echo $(( 1 + $RANDOM % 10 ))
9
danish@danish-Ubuntu:~$ echo $(( 1 + $RANDOM % 20 ))
5

Comparing Values with `test` or `[ ... ]`

In scripts it's common to need to compare things and test values. Bash has a built-in called Test, which is also represented by single brackets. [ ... ]
This built-in works in the same way as the test command which is part of the GNU coreutils, the set of tools that are available on many Linux distributions.
Test is widely compatible across shells and it's worth taking a moment to look at the help for the test built-in. There are a variety of operations that Test can well, test.
Let's ask Bash whether my home directory is a directory.

danish@danish-Ubuntu:~$ [ -d ~ ]
danish@danish-Ubuntu:~$ echo $?
0
danish@danish-Ubuntu:~$ help test
test: test [expr]
    Evaluate conditional expression.

    Exits with a status of 0 (true) or 1 (false) depending on
    the evaluation of EXPR.  Expressions may be unary or binary.  Unary
    expressions are often used to examine the status of a file.  There
    are string operators and numeric comparison operators as well.

    The behavior of test depends on the number of arguments.  Read the
    bash manual page for the complete specification.

    File operators:

      -a FILE        True if file exists.
      -b FILE        True if file is block special.
      -c FILE        True if file is character special.
      -d FILE        True if file is a directory.
      -e FILE        True if file exists.
      -f FILE        True if file exists and is a regular file.
      -g FILE        True if file is set-group-id.
      -h FILE        True if file is a symbolic link.
      -L FILE        True if file is a symbolic link.
      -k FILE        True if file has its `sticky' bit set.
      -p FILE        True if file is a named pipe.
      -r FILE        True if file is readable by you.
      -s FILE        True if file exists and is not empty.
      -S FILE        True if file is a socket.
      -t FD          True if FD is opened on a terminal.
      -u FILE        True if the file is set-user-id.
      -w FILE        True if the file is writable by you.
      -x FILE        True if the file is executable by you.
      -O FILE        True if the file is effectively owned by you.
      -G FILE        True if the file is effectively owned by your group.
      -N FILE        True if the file has been modified since it was last read.

When we run a comparison or test, the result we get back is an exit or return status of a zero or a one. Zero for success and one for failure. These are analogous to the exit status of a command line program. Where zero indicates that a program finished running successfully. And one or any other number indicates that there was an error of some kind. What this return or exit status lets us do is treat the expression as a truth value. That becomes important when we combine these tests and comparisons with flow control structures. Like if or while.
Lets test some string charatacters and see if they are equal or not equal.

danish@danish-Ubuntu:~$ [ "cat" = "cat" ]; echo $?
0
danish@danish-Ubuntu:~$ [ "cat" = "dog" ]; echo $?
1

To point out the difference between testing strings and testing numbers. While we might be used to using greater than, less than, and equal signs to compare numbers, in the Test built-in, those are reserved for comparing strings. Strings can be greater or less than each other in the sense that the characters contained within them come earlier or later in the character set.
And we can use tests to compare numeric values too.
To compare numbers, we'll use different operators with a dash (-) instead -lt stands for less than.

danish@danish-Ubuntu:~$ [ 4 -lt 3 ]; echo $? 
1
danish@danish-Ubuntu:~$ [ 4 -gt 3 ]; echo $? 
0

We can also negate or invert a test by using the exclamation mark character in front of the test expression.

danish@danish-Ubuntu:~$ [ ! 4 -gt 3 ]; echo $? 
1
danish@danish-Ubuntu:~$

It's important to keep spaces between the brackets and arguments and operators. The test won't work if they're all squished together.

Comparing values with extended `test` or `[[ ... ]]`

In addition to the single bracket test notation, Bash gives us the double bracket notation called extended test. As with test, it's important to keep spaces between the sets of brackets and the expression we want to evaluate.
Extended test gives us the same operations as test and adds a few other helpful features. With extended test, we can use more than one expression within a test to create a little bit more complex logic. For example, I can ask whether my home directory is a directory and whether the Bash binary exists. That set of statements is true.

danish@danish-Ubuntu:~$ [[ 8 -lt 5 ]]; echo $? 
1
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ [[ -d ~ && -a /root/etc/ ]] ; echo $?
1
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ [[ -d ~ || -a /root/etc/ ]] ; echo $?
0
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ [[ -d ~ && -a /var/tmp ]] ; echo $?
0
danish@danish-Ubuntu:~$

We can also use these logical operators outside the extended test brackets to run commands conditionally. For example, we can write a test to see if the home directory is a directory. Then add two ampersands and then add an echo statement and because the test returns success Bash then goes on to run the echo statement. If the test fails, the echo statement doesn't run.

danish@danish-Ubuntu:~$ [[ -d ~ ]] && echo ~ is a directory 
/home/danish is a directory
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ [[ -d /bin/bash ]] && echo /bin/bash is a directory 
danish@danish-Ubuntu:~$ echo $? 
1
danish@danish-Ubuntu:~$

The test can also be replaced with a command because commands return an exit status too. For example, We could write ls, double ampersand, and then the text listed the directory. If the command completes my echo statement will be displayed.

danish@danish-Ubuntu:~$ ls && echo "Directory is listed successfully"
 10.197.192.171_dsm_cleanup_upgrade.out  'Calibre Library'   dir1   dk-img1.qcow2   Downloads      Music      Public   stopwatch.sh   union
 10.197.192.174_dsm_cleanup_upgrade.out   Desktop            dir2   Documents       fortanix.gpg   Pictures   snap     Templates      Videos
Directory is listed successfully
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$

Also, there are two built ins called true and false that you can use to generate a success or failure return code. Here, for example, we see the result of running true and an echo statement followed by the result of running false and an echo statement.

danish@danish-Ubuntu:~$ true && echo 'Will print the ECHO'
Will print the ECHO
danish@danish-Ubuntu:~$ echo $? 
0
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ false && echo 'Will not print the ECHO'
danish@danish-Ubuntu:~$ echo $? 
1
danish@danish-Ubuntu:~$

Another useful feature that extended test provides is regular expression matching. To use that, I'll provide a string and then write equals tilde and then provide a regular expression to try to match to the string. This test succeeds because the string cat starts with C followed by a number of characters. If I change my string to zat, the test will fail.

danish@danish-Ubuntu:~$ [[ "cat" =~ c.* ]]; echo $? 
0
danish@danish-Ubuntu:~$ [[ "zat" =~ c.* ]]; echo $? 
1
danish@danish-Ubuntu:~$

Tests and comparisons are extremely common in Bash scripts so it's good to be as familiar with them as you can be.
When writing your script, it's usually best to stick to using the extended test rather than switching between it and regular test. Doing so maintains consistency and ensures you have more features available to you. But if your script needs to be portable to other shells you'll want to use regular tests instead even though you'll lose some features.
Now that we know how to get a true or false value out of a comparison or test we can use test to provide control flow in our scripts.

Formatting and styling text output

Let's move to working with text for a while.
One option offered by the Echo built-in is echo - e which tells Bash to interpret escaped characters or sequences which can be used to enhance text output.
Escape sequences can represent special characters like tab, new line or the bell and they can be used to change the color of text in the terminal as well.
When we write script, we'll often want to format text to line it up in columns or indent it and the tab character plays a role there.

danish@danish-Ubuntu:~$ echo "Name\t\tNumber"; echo "Scott\t\t123"
Name\t\tNumber
Scott\t\t123
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo -e "Name\t\tNumber"; echo -e "Scott\t\t123"
Name        Number
Scott       123
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ echo -e "This text\nbreak in\nthree lines"
This text
break in
three lines
danish@danish-Ubuntu:~$

Formatting output with `printf`

As we've seen with echo composing strings that include variables or other data can get a little bit tedious, and so I want to show you another way to do it.
The printf builtin gives us the ability to use placeholders when we compose strings and format values in ways that can make our script and our output look cleaner and more organized.
Let's compare getting the same output result with both echo and printf.

danish@danish-Ubuntu:~$ echo "The results are: $(( 2 + 2 )) and $(( 3 / 1 ))"
The results are: 4 and 3
danish@danish-Ubuntu:~$ printf "The results are: %d and %d\n" $(( 2 + 2 )) $(( 3 / 1 ))
The results are: 4 and 3
danish@danish-Ubuntu:~$

The first %d prints the first value and the 2nd printf the next subsequent value.
Using printf, we'll write the literal parts of our string, the parts that don't change, and instead of putting Bash code mixed in with the string we use a placeholder wherever a dynamic value, one that will be the result of something our script runs will go and then at the end of the line we'll put whatever code we'll provide those values in the same sequence we used for the placeholders.
If we provide more values than there are placeholders we discover another useful feature of printf. It will repeat the output for each input it's given. The placeholder %d here represents a digit and a placeholder of %s represents a string value in this comparison below.

danish@danish-Ubuntu:~$ printf "The results are: %d\n" $(( 2 + 2 )) $(( 3 / 1 ))
The results are: 4
The results are: 3
danish@danish-Ubuntu:~$

The printf statement is longer than the echo statement that prints the same text but it can be easier to read and easier to work with.
In simpliar terms :-

Operator	Meaning
`%d`	Digit
`%s`	Strings

For more info check man 3 printf
A benefit of printf is that it lets us format output a little bit more easily than with echo.

Working with Arrays in Bash

When writing scripts, we'll sometimes need to store pieces of information that contain more than one variable. While we can always create a new variable for any piece of information we need to store, that can become difficult to track and maintain.
Arrays allow us to store related pieces of information and to refer to them more easily.
Bash supports two kinds of array:
- Indexed arrays i.e. (Values with positioned Index)
- Associative arrays i.e. (Key and Value), used by declare -A XXX
In an indexed array, we set or read pieces of information by referring to their position in the list, or their index. We can define an indexed array implicitly by providing a list of values within parenthesis and assigning a name.
Each term, double quotes separated by a space, and then a closing parenthesis, or we can do this explicitly using declare dash a. There's an array named snacks with three elements inside it. There aren't commas between elements like there are in other languages.

danish@danish-Ubuntu:~$ snacks=("apple" "banana" "orange")
danish@danish-Ubuntu:~$ echo $snacks
apple
danish@danish-Ubuntu:~$ echo ${snacks[@]}
apple banana orange
danish@danish-Ubuntu:~$

danish@danish-Ubuntu:~$ declare -a snacks=("apple" "banana" "orange")
danish@danish-Ubuntu:~$ echo ${snacks[@]}
apple banana orange
danish@danish-Ubuntu:~$

To retrieve an element inside an array, will use its zero based notation along with a variable name inside of curly braces. In this case, orange, since it's at index two, the third element. Apple is at index zero, and banana is at index one.

danish@danish-Ubuntu:~$ echo ${snacks[0]}
apple
danish@danish-Ubuntu:~$ echo ${snacks[1]}
banana
danish@danish-Ubuntu:~$ echo ${snacks[2]}
orange

We can also set array values by index number. We can write snacks, and then in square brackets I'll provide the index five, and set that equal to grapes. And we don't have to populate every element. These sparse arrays can be very useful when you need to record data with a particular position in a list instead of just adding on to the end of the array.

danish@danish-Ubuntu:~$ snacks[5]="grapes"
danish@danish-Ubuntu:~$ snacks+=("mango")
danish@danish-Ubuntu:~$ echo ${snacks[@]}
apple banana orange grapes mango

But when we print the same we are not having the empty spaces printed thus to print that we can do the below:-

danish@danish-Ubuntu:~$ for i in {0..6}; do echo "$i: ${snacks[$i]}"; done
0: apple
1: banana
2: orange
3: 
4: 
5: grapes
6: mango

If you're using Bash version four, and above, we can also make associative arrays. That is, we can specify a key and a value instead of just a list of values with positional index values. To do this, we'll use the below example:-

danish@danish-Ubuntu:~$ declare -A office
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ office[city]="San Francisco"
danish@danish-Ubuntu:~$ 
danish@danish-Ubuntu:~$ office["building name"]="HQ West"
danish@danish-Ubuntu:~$ echo ${office["building name"]} is in ${office[city]}
HQ West is in San Francisco
danish@danish-Ubuntu:~$

If you're using Bash3 or below, you won't have support for associative arrays, so keep that in mind when you're building your script.
Associative arrays and indexed arrays are both limited to one level or layer, so we can't create an array with an array inside of it, or a nested array, in Bash.
If you find yourself needing to represent data in a nested array, that's usually a good sign that it's time to move on from Bash and start writing your program in another, more suitable language.
Your script may benefit from using arrays, or you may not use them at all.
In many situations, the same information can be stored as single variables in an indexed array or in an associative array. What you choose to use comes down to how you want to model, think about, and use whatever data your program needs.

Control structures in Bash

Conditional statements with the "if" keyword

When we write scripts, we'll often use control structures, which instead of just running script commands from the top down one after the other, give us the opportunity to control and change how script commands are executed based on conditions we specify.
One of these control structures is an if statement. And an if statement runs or executes code based on the truth value of a given expression.
In Bash, that takes the form of if, followed by an expression from which we'll get a true or false value followed with a keyword then.
That's followed by script lines to run, if the condition evaluates as true. We can end the if statement there with fi, the word if spelled backwards, or we can add an else statement and provide script to run, if the condition evaluates as false.
The condition is something that returns a truth value, often the extended test with two brackets, but it can also be the older single bracket or an arithmetic evaluation.
It can also be a command, because remember, commands always finish with a zero or non-zero status interpreted as true or false.
In fact, when you're experimenting, it can be useful to use the true and false commands in your if statements to guarantee a particular result.

Working with "while" and "until" loops

Loops are control structures that run a specific piece of code over and over until the loop is caused to end.
There's two kinds of loops that are dependent on conditions, called
- while loops
- until loops.
The while loop runs for the duration of time when its condition is true and stops when that condition becomes false.
The until loop runs for the duration of time when its condition is false and stops when that condition becomes true.
Both types of loops start with the appropriate keyword and the condition to use to determine whether the loop should keep running. That's followed by do and the code to loop through. And then the construct ends with done.

#!/usr/bin/env bash

echo "While Loop"

declare -i n=0
while (( n < 10 ))
do
    echo "n:$n"
    (( n++ ))
done

echo -e "\nUntil Loop"
declare -i m=0
until (( m == 10 )); do
    echo "m:$m"
    (( m++ ))
done

Introducing "for" loops

A for loop takes a list or range of values and runs code once for each value in that list. Each iteration of the loop, each time the loop runs, a variable within the loop represents the current item in the list.

#!/usr/bin/env bash

for i in 1 2 3
do
    echo $i
done

for i in 1 2 3; do echo $i; done

#!/usr/bin/env bash

for i in {1..100}
do
    echo $i
done

#!/usr/bin/env bash

for (( i=1; i<=100; i++ ))
do
    echo $i
done

#!/usr/bin/env bash

declare -a fruits=("apple" "banana" "cherry")
for i in ${fruits[@]}
do
    echo $i
done

Selecting behavior using "case"

The case statement is a control structure which lets us define code to run based on particular conditions.
We provide it a series of values and whichever one matches the given input is what Bash runs.
A case statement starts out with the word case followed by a variable that you're testing and the word in and ends with esac.

#!/usr/bin/env bash
animal="dog"
case $animal in
    cat) echo "Feline";;
    dog|wolf) echo "Canine";;
    *) echo "No match!
esac

danish@danish-Ubuntu:~$./case_statement.sh 
Canine

If we change the value then see the output

#!/usr/bin/env bash

animal="Monkey"
case $animal in
    cat) echo "Feline";;
    dog|wolf) echo "Canine";;
    *) echo "No match!"
esac

Output of script

danish@danish-Ubuntu:~$ ./case_statement.sh 
No match!
danish@danish-Ubuntu:~$

Using functions in Bash

During your script development, you may find yourself repeating the same set of commands, or writing very similar commands while changing just one or two parameters.
It's a good practice to not repeat yourself if you don't have to. And keeping scripts organized without duplication makes them a lot more maintainable too.
To help organize code, we can use functions, which let us write a set of commands once and then refer to that code by name whenever we need to run it.
To create a function, we'll give the function a name, in this case the name fname, and follow that with open and close parentheses. And then we'll add a set of braces to enclose commands that will make up the function.

#!/usr/bin/env bash

greet() {
    echo "Hi there."
}

echo "And now, a greeting..."
greet

I'll use the variable, $1, which represents the first argument passed into a function. I'll save my script and run it again. Here I see my name passed into the function, greet. $1 represents the first argument pass and other arguments can follow that pattern. $2 is the second argument, $3 is the third, and so on.

#!/usr/bin/env bash

greet() {
    echo "Hello, $1"

}

greet Danish
echo "Greetings, Please tell me How are you?"
danish@danish-Ubuntu:~$ ./functions_check.sh 
Hello, Danish
Greetings, Please tell me How are you?

#!/usr/bin/env bash

greet() {
    echo "Hello there $1. What a nice $2."
}

greet Danish Morning
greet Everybody Evening

danish@danish-Ubuntu:~$ ./functions_check.sh 
Hello there Danish. What a nice Morning.
Hello there Everybody. What a nice Evening.

There's a few more special variables that we can use inside a function too, and those are $@ and $FUNCNAME. @ represents all of the arguments given to the function, which can be useful for looping through the arguments if your script calls for that.

#!/usr/bin/env bash

numberthing() {
    declare -i i=1
    for f in $@; do
        echo "$i: $f"
        (( i += 1 ))
    done
    echo "This counting was brought to you by $FUNCNAME."
}

numberthing "$(ls /)"
echo
numberthing pine birch maple spruce

#!/usr/bin/env bash

var1="I'm variable 1"

myfunction() {
    var2="I'm variable 2"
    local var3="I'm variable 3"
}
myfunction

echo $var1
echo $var2
echo $var3

Understanding `kube-system` in Kubernetes: A City Analogy You’ll Never Forget

Danish Khan — Sun, 27 Jul 2025 01:28:34 +0000

🧱 The Analogy: Kubernetes as a Smart City

Imagine Kubernetes as a modern smart city.

Now, within this city:

🏢 Pods = Buildings or Shops
🧑‍💼 Deployments = City Planners or Construction Managers
🛣️ Services = Roads & Signboards that help people reach buildings
🧱 Nodes = City Blocks where buildings stand
🧭 Ingress = City’s Gateways or Entry Checkpoints
📜 ConfigMaps/Secrets = City Rulebooks and Vaults
🛠️ kube-system = Municipal Headquarters

🏛️ What is `kube-system`?

The kube-system namespace is like the City’s Municipal Office.

You don’t see it on the news, but it's what keeps the entire city running.

You wouldn’t open a café or bank inside your city’s electrical grid control room, right?

Similarly, you shouldn’t deploy your app inside kube-system.

🧠 Memorize This for Life:

"If Kubernetes is a City, then kube-system is its Municipality – invisible to citizens but absolutely vital."

🛠️ What Runs Inside `kube-system`?

Let’s break down some key pods and their real-world equivalents:

Kubernetes Component	Real-World Analogy	Role in the City (Cluster)
`kube-dns` / `CoreDNS`	📇 Address Lookup Office	Resolves service names to IP addresses
`kube-proxy`	🚦 Traffic Police	Controls how traffic flows between buildings
`etcd`	🗃️ City’s Memory Archive	Stores all official records (K8s config/state data)
`kube-scheduler`	🏗️ Construction Allocator	Decides which block gets which new building (Pod)
`kube-controller-manager`	📋 Mayor's Secretariat	Ensures reality matches the city’s blueprints
`metrics-server`	🧑‍⚕️ Health Inspector	Monitors health across city buildings

🚫 Why You Shouldn’t Deploy Apps in `kube-system`

Because it’s not meant for public use.

It’s like trying to open your food truck inside the electricity board’s server room.

Your apps belong in dedicated districts like:

default (for general use)
dev, stage, prod (for environment segregation)
team-a, team-b (for multi-team setups)

Keep kube-system reserved for core city operations!

📸 Quick Commands to Peek into kube-system

kubectl get pods -n kube-system
kubectl describe pod <pod-name> -n kube-system

Use these to inspect what your Kubernetes "municipal office" is doing.

🧱 Brick-by-Brick Learning That Will Grow

In this “Kubernetes is a City” model, we’ll continue using this analogy for upcoming topics:

Deployments = City Builders
Services = Transport and Signage
Ingress = Entry Checkpoints
Volumes = Storage Units
Secrets = Vaults & Lockers
So whenever you get confused, just go back to the city map 🗺️.

🚀 Final Thought

You don’t need to memorize everything.Just understand the city, and the rest will make sense.
Stay tuned for the next brick — we’ll build this DevOps city together!

Written ✍️ for beginners stepping into the magical (and sometimes mystifying) world of Kubernetes. If you found this helpful, follow me for more ‘brick-by-brick’ DevOps analogies!
Let me know when you’re ready for the next brick, like explaining Deployments, Services, or Ingress, and I’ll extend the city analogy accordingly.

Demystifying Docker Multi-Platform Builds and Manifests: A Brick by Brick Guide

Danish Khan — Wed, 09 Jul 2025 17:58:25 +0000

🧱 "One image. Multiple platforms. Zero surprises."

This is what Docker's multi-platform build and manifest system enables — and it's more relevant than ever in today’s hybrid infrastructure world.

Whether you’re:

Working on an M1 Mac but deploying to an x86 GKE cluster,
Building in CI/CD on an ARM-based runner but targeting AMD64 servers,
Or managing both OpenShift on-prem and GKE on cloud...

You need to build and tag Docker images the right way — to ensure your containers run reliably, regardless of where they're pulled.

In this article, we’ll go brick by brick to explore:

✅ What is a multi-platform image build?
✅ Why does Docker pull different images on different machines?
✅ What exactly is a manifest and why does it matter?
✅ How to enforce platform consistency across builds?
✅ What breaks if you don’t use platform-aware builds?

And we’ll wrap it all up with:

🧠 Real-world DevOps gotchas
🛠️ Best practices for Jenkins and CI/CD pipelines
📦 Storage and runtime behavior of platform-specific layers

Let’s get started. 🔍

🧱 What is a Docker Multi-Platform Build?

A multi-platform build allows you to create Docker images that can run on multiple CPU architectures (like amd64, arm64) from a single machine — even if that machine only supports one architecture.

🧠 Why It Matters

Different environments use different processor types:

🖥️ Intel/AMD laptops and servers → amd64
🍏 Apple M1/M2 Macs → arm64
📦 Raspberry Pi → armhf, arm64
☁️ AWS Graviton or GCP Tau T2A → arm64

If you don’t build platform-aware images, your container might fail to run on systems it wasn’t built for.

🔍 How Docker Identifies Your Architecture Automatically

When you run:

docker pull busybox

Docker automatically knows your architecture using system calls like:

uname -m     # Returns x86_64 or arm64

It uses this info to pull the correct platform image by default — no need for you to explicitly specify the platform unless you're cross-building.

📦 Why SHA256 Digests Differ for Each Platform

Even if the tag is the same (busybox:latest), the actual SHA256 digest is different for amd64 vs arm64.

Why?

Each platform compiles binaries differently, so:

Different binaries = different layers
Different layers = different image digests

🤔 What Happens If I Pull Two Platform Variants?

Let’s say you do:

docker pull --platform=linux/arm64 busybox
docker tag busybox busybox:arm64

docker pull --platform=linux/amd64 busybox
docker tag busybox busybox:amd64

Now you have two tagged variants.

But if you just do:

docker run busybox

It will run the latest pulled busybox:latest, which may or may not match your expected platform.

✅ Best Practice:

Explicitly tag and run the desired platform image:

docker run busybox:arm64
docker run busybox:amd64

⚙️ How to Enforce Platform in Docker Build

If you're using a CI/CD pipeline (like Jenkins) or building locally, always fix the platform to avoid surprises:

docker buildx create --use   # once per agent
docker buildx build \
  --platform=linux/amd64 \
  -t myimage:1.0.0 \
  --push .

This ensures the image is always built for amd64, regardless of whether you're on M1, x86, or in the cloud.

📍 Real-World Use Case: Same Image for On-Prem & Cloud

Assume you’re running:

On-prem OpenShift (OCP) setup
Target deployment on GCP GKE

Even if both runtime platforms are amd64, your CI runner may be arm64 (e.g., running on M1 Mac or ARM-based VM). If you don’t specify --platform, your build may accidentally produce an arm64 image, which won’t work in production.

✅ Solution

In Jenkins or your build script:

docker buildx build \
  --platform=linux/amd64 \
  -t registry.io/myapp:1.0.0 \
  --push .

You can also verify the platform post-build:

docker image inspect myapp:1.0.0 \
  --format='{{.Architecture}}/{{.Os}}'

📈 Will Multi-Platform Build Increase Image Size?

Registry Size:

✅ Yes, total size increases because each architecture has its own full image:

amd64: 100MB
arm64: 90MB → Registry now stores ~190MB

Local Size:

❌ No, unless you --load all variants locally (default keeps only the last one).

Runtime Download:

✅ Efficient — Docker pulls only the matching platform image for the host.

📜 Deep Dive into Docker Manifests

🤖 What is a Manifest?

A Docker manifest is a metadata file that maps a Docker image tag to the correct image version for each platform.

Think of it as a:

📦 Smart label that says “if you’re on amd64, pull this; if arm64, pull that.”

It allows the same tag (myapp:latest) to point to multiple images, depending on who’s asking.

📖 Real-World Analogy

🍔 A restaurant has one menu item: "Burger".

If an Indian orders, they get a veg burger.

If an American orders, they get a beef burger.

If a Middle Easterner orders, they get a lamb burger.

They all just said “burger” — but they got what’s right for them.

Similarly:

docker pull myapp:latest

returns a platform-specific image — thanks to the manifest.

🔬 What’s Inside a Manifest?

{
  "manifests": [
    {
      "platform": { "architecture": "amd64", "os": "linux" },
      "digest": "sha256:aaa..."
    },
    {
      "platform": { "architecture": "arm64", "os": "linux" },
      "digest": "sha256:bbb..."
    }
  ]
}

🔍 How to Inspect a Manifest

docker buildx imagetools inspect myapp:latest

✅ What Problems It Solved

Problem	Solved By Manifest
One tag = one platform only	One tag supports many platforms
Manual selection of image	Automatic matching by Docker
Broken builds on ARM/M1	Seamless image pulls
Multiple tags to manage	Single tag, multi-arch supported

✅ Final Best Practices Summary

Task	Recommendation
CI builds on mixed-arch runners	Always use `--platform` in `docker buildx build`
Multi-platform support	Build with `--platform=linux/amd64,linux/arm64` and push manifest
Save registry space	Build only required platforms
Keep platform control	Tag each variant explicitly (e.g., `myapp:arm64`)
Validate architecture	Use `docker inspect` or CI assertions

🔚 Conclusion

Understanding Docker’s multi-platform builds and manifests is essential in today’s hybrid infrastructure world. Whether you're building locally, on Jenkins, deploying to GKE, or OpenShift — platform consistency matters.

One tag. Many platforms. No surprises.

That's the power of Docker Manifests.

Want to learn how to write your own manifest manually or automate this via Jenkins or GitHub Actions? Comment or connect — happy to help!

Understanding Docker Layer Caching: Why It Matters for DevOps Engineers

Danish Khan — Sat, 05 Jul 2025 23:48:54 +0000

As a DevOps engineer diving deep into containerization, I recently asked myself a seemingly simple question:

“When Docker builds an image, how exactly does it cache layers — and does understanding this really matter in real-world scenarios?”

Turns out, it does — a lot more than I thought.

Let’s explore how Docker caches image layers, why it does so, and how understanding this helps in real CI/CD pipelines and production builds.

🧱 Dockerfile: Every Instruction Is a Layer

Each line in your Dockerfile becomes a new layer in your image. Docker builds images step by step — and tries to cache each step to speed things up.

Here’s a simple Dockerfile to illustrate:

FROM ubuntu:20.04
RUN apt-get update
RUN apt-get install -y curl
COPY . /app
CMD ["./start.sh"]

Docker builds this from top to bottom. Each instruction like RUN or COPY is a layer, and Docker stores them in its local cache.

🔁 How Docker Layer Caching Works

When Docker runs a build:

It looks at each instruction in the Dockerfile.
If that instruction hasn't changed and all previous instructions are also cached, it reuses the existing cached layer.
If a change is detected, all instructions after that point are rebuilt — no cache reuse beyond that. You’ll see this during builds:

Step 2/5 : RUN apt-get update
 ---> Using cache
Step 3/5 : RUN apt-get install -y curl
 ---> Running in abc123...

The second instruction was cached. The third one was rebuilt — possibly because something changed or it wasn’t cached before.

🧠 Why This Matters in Real Life (Not Just Theory)

Understanding layer caching helps in practical DevOps scenarios. Here’s where it makes a difference:

✅ 1. Faster CI/CD Pipelines

CI pipelines that rebuild Docker images frequently can benefit hugely from layer caching.

Place slow-changing instructions early (e.g., apt-get update, tool installs).
Place frequently changing instructions later (e.g., COPY .). This lets Docker skip as much work as possible in repeated builds.

🐞 2. Troubleshooting Build Failures

Sometimes, caching hides problems. Maybe a stale file is being used due to cache.
To force a clean build, use:

docker build --no-cache -t myapp .

This is helpful when:

A file isn’t being copied as expected.
An install command fails only in CI but works locally.
Environment variables changed but not reflected.

📦 3. Reduce Image Size (and Security Surface)

Smaller images = faster pulls, better security.

Combine commands like this:

RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

This results in fewer layers, less bloat, and a tighter image.

🔐 4. Security & Audits

Understanding layer origins helps pinpoint:

When a vulnerable file was introduced.
Which layer to rebuild to remove it.
Which command added sensitive stuff.

🔍 Inspecting Layers: docker history

You can inspect image layers with:

docker history myapp

This shows each Dockerfile instruction and how much space it added:

IMAGE         CREATED         CREATED BY                     SIZE
abc123        10 seconds ago  RUN apt-get install curl       25MB
...

Want to go deeper? Try dive — it gives you a beautiful interactive CLI to explore each layer.

🏁 TL;DR

Docker caches each Dockerfile instruction (layer) if nothing has changed. As a DevOps engineer, understanding this:

Speeds up your CI/CD builds.
Helps troubleshoot weird Docker issues.
Keeps your image sizes optimized.
Gives you visibility into build and runtime behavior. Next time your Docker build seems “weirdly fast” or “unexpectedly broken,” caching might be the hero or the culprit.

Why I Started “DevOps Brick by Brick” — My Self-Taught DevOps/SRE/GitOps Journey

Danish Khan — Tue, 01 Jul 2025 21:59:09 +0000

Over the years, I realized that while working in tech, I was relying too much on prebuilt systems and quick fixes. I wanted to truly understand the tools I use daily — Docker, Kubernetes, Jenkins, GCP, GitOps, and beyond.

This blog is my public notebook — documenting what I learn, how I debug, and how I grow as a self-taught DevOps/SRE engineer.

I’ll be sharing bite-sized learnings, troubleshooting steps, and mini lab setups I build for myself. All with one motto in mind:

🐟 I will not give a fish, I'll teach you how to fish.

Welcome to "DevOps Brick by Brick" — let’s build this together.

Forem: Danish Khan

🚀 Python for SRE/DevOps: Building SDKs + Jenkins Automations

🧠 Big Picture Architecture

📚 Phase 1: Python Foundations (Automation-Focused)

What to Learn

Resources

🌐 Phase 2: APIs First (Before SDKs)

What to Learn

Resources

🧪 Practice

📦 Phase 3: Python Packaging (VERY IMPORTANT)

What to Learn

Resources

🧪 Practice

🏗️ Phase 4: Building SDK-Style Code

What to Learn

Typical SDK Structure

Resources

🧪 Practice

⚙️ Phase 5: Jenkins + Python Integration

What to Learn

Resources

🧪 Practice

🧪 Phase 6: Testing (CRITICAL for SRE)

What to Learn

Resources

⚡ Phase 7: Async Python (Advanced)

What to Learn

Resources

🧱 Real-World Pattern You Are Learning

🔥 Mini Project (Must Do)

Project: Monitoring Automation SDK

Structure

Features

📅 7-Day Fast-Track Plan

Day 1

Day 2

Day 3

Day 4

Day 5

Day 6

Day 7

🧠 Final Insight

🚀 Next Step : Coming Soon (WIP)

Bash Scripting for Non-Coders

A Practical, No-Fear Guide to Understanding the Shell

Piping & Re-direction

Re-direction in Bash

Re-direction in file

Commands and Built-In

Bash Expansions & Substitutions

Tilde Expansion ~

Brace Expansion {...}

Parameter expansion ${...}

Command Substitution $(...)

Arithmetic expansion $((...))

Programming/Scripting in Bash

Bash Script

Using echo in Bash

Working with Variables in Bash

Bash declare Options (Professional Cheat Sheet)

Variable Attributes

Arrays

Function Options (Rare)

Visibility / Declaration

Working with Numbers in Bash

Comparing Values with test or [ ... ]

Comparing values with extended test or [[ ... ]]

Formatting and styling text output

Formatting output with printf

Working with Arrays in Bash

Control structures in Bash

Conditional statements with the "if" keyword

Working with "while" and "until" loops

Introducing "for" loops

Selecting behavior using "case"

Using functions in Bash

Understanding `kube-system` in Kubernetes: A City Analogy You’ll Never Forget

🧱 The Analogy: Kubernetes as a Smart City

🏛️ What is kube-system?

Tilde Expansion `~`

Brace Expansion `{...}`

Parameter expansion `${...}`

Command Substitution `$(...)`

Arithmetic expansion `$((...))`

Bash `declare` Options (Professional Cheat Sheet)

Comparing Values with `test` or `[ ... ]`

Comparing values with extended `test` or `[[ ... ]]`

Formatting output with `printf`

🏛️ What is `kube-system`?

🛠️ What Runs Inside `kube-system`?

🚫 Why You Shouldn’t Deploy Apps in `kube-system`