Forem: Ridae HAMDANI

Master Disk I/O Metrics Monitoring

Ridae HAMDANI — Mon, 30 Jun 2025 13:14:00 +0000

Disk IOPS (input/output per second) stats are metrics that gives you insights on how your system interact with your storage devices. These metrics measure read and write operations.

Fundamental Disk IOPS metrics

IOPS (Input/Output Per Second)

IOPS stats measures how many Input(write)/Output(read) transaction your storage devices perform per second.

IOPS metrics are exported by node exporter.

irate(node_disk_reads_completed_total[$__rate_interval])
irate(node_disk_writes_completed_total[$__rate_interval])

IOPS depends on your disk but a higher number generally mean better performance, but a context matter , IOPS metrics is meaningless without a latencey figure:

High IOPS but low latency = healthy
Low IOPS + high latency = disk contention

Latency (Average time per operation)

Latency represents the time it takes for the I/O request to be completed.

Latency = Queue time + processing (READ or Write) operation.

Latency is the most important metrics to consider for a storage performance. Lower latency means better performance.

irate(node_disk_write_time_seconds_total[$__rate_interval])/irate(node_disk_writes_completed_total[$__rate_interval]) # Read Latency
irate(node_disk_write_time_seconds_total[$__rate_interval])/irate(node_disk_writes_completed_total[$__rate_interval]) # Write Latency

Utilisation

Disk Utilisation is the percentage of time the disk busy processing I/O operations.

High disk I/O time utilisation (e.g: > 80%) means te disk is busy almost all the time handling I/O requests. Technically, it means:

Slow or failing hardware
Concurrent access bottlenecks
High I/O workload
Insefficient access patterns

If latency is also high, this means the disk can't keep up , which leads to performance degradation.

rate(node_disk_io_time_seconds_total[1m]) * 100

High utilization + high latency = saturated disk ### Throughput or Bandwidth

Troughput or Bandwidth represents the amount of data in Megabytes per second (MB/s) transferred to or from the storage device.

irate(node_disk_written_bytes_tota[$__rate_interval]) 
irate(node_disk_read_bytes_total[$__rate_interval])

Here is how to interpret the bandwidth metrics:

If your read/write MB/s is close to the specs of your disk, it’s OK.
If you see very low bandwidth but high I/O wait or high disk utilization, the disk might be overloaded on small random reads (IOPS problem, not bandwidth).
If both bandwidth and IOPS are high, the disk is at full load.

Optimizing GitLab CI for Readability and Maintainability: From 1K to 600 Lines!

Ridae HAMDANI — Fri, 26 Jan 2024 10:36:11 +0000

This short article aims to share some tips I learnt from optimizing our GitLab CI file (.gitlab-ci.yaml) for building multiple Docker images.

By implementing these techniques, you’ll not only make your GitLab CI files more readable but also set the stage for a more agile and adaptable CI/CD environment.
Let’s delve into the practical insights that can elevate your DevOps experience and contribute to a more efficient and maintainable codebase.

1- Use default runner tag and override it when needed:

In case you are using a standard or a generic runner for the majority of your jobs and a a different runner for a few jobs, consider defining a global runner tag instead of defining one for each job.
You can then override the global tag when necessary.

❌ Don’t do:

compile:
  script:
    - ...
  tags:
    - myruuner
test:
  script:
    - ...
  tags:
    - testrunner  
deploy:
  script:
    - ...
  tags:
    - myrunner

✅ Do:

default:
  tags:
    - myrunner

compile:
  script:
    - ...

test:
  script:
    - ...
  tags:
    - testrunner  
deploy:
  script:
    - ...

2- Use `parallel:matrix`

GitLab has a powerful CI feature to run a matrix of jobs in parallel.
In our case we had multiple jobs to test every customized Docker JDK image and ensure that it does not break a standard maven build.
Taking advantage of the keyword parallel:matrix, can save you multiple lines of code and make your CI files more readable and easier to maintain.

Here is an example of refactoring our test jobs:

❌ Instead of configuring multiple job with the same script part, for example:

test-mvn-jdk-11:
  image: openjdk-11
  stage: test-image
  script:
    - mvn -V compile ...

test-mvn-jdk-17:
  image: openjdk-17
  stage: test-image
  script:
    - mvn -V compile ...

test-mvn-jdk-21:
  image: openjdk-21
  stage: test-image
  script:
    - mvn -V compile ...
....

✅ You can easily replace them with:

test-mvn:
  image: $JAVA_IMAGE
  stage: test-image
  script:
    - mvn -V compile ...
  parallel:
    matrix:
      - JAVA_IMAGE: [openjdk-11,openjdk-17,openjdk-21]

This will also make it easier to incorporate additional JDK versions in the future, requiring minimal adjustments and eliminating the need for extensive line changes or introducing new jobs.

3- Use rules to define variables:

GitLab rules provide a mechanism for specifying conditions to determine when CI/CD jobs run.
Additionally, the use of if statements within these rules allows for the dynamic definition of variables based on conditions such as branches or releases, offering flexibility in variable assignment.

For our repository's CI/CD workflow, we start by building docker images and test them if there's a Merge Request.
Only when everything checks out, we create a final version TAG for release.
Using rules reduced our build jobs by a half.

❌ Instead of defining two jobs, for example:

build-mr-jdk-11:
  stage: build
  script:
    - docker build -t jdk-11:mr-validation
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

build-jdk-11:
  image: openjdk-11
  stage: build
  script:
    - docker build -t jdk-11:$CI_COMMIT_TAG
  rules:
    - if: $CI_COMMIT_TAG

✅ Use the same job and override the variables depending on the rules conditions:

build-jdk-11:
  stage: build
  script:
    - docker build -t jdk-11:$TAG
rules:
    - if: $CI_COMMIT_TAG
      variables:
        TAG: "$CI_COMMIT_TAG"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      variables:
        TAG: "mr-validation"

Conclusion: Lean back and let the magic happen 🧙‍♀

In summary, applying these tips reduced our GitLab CI file from 1176 to 667 lines, significantly improving efficiency, manageability, and overall workflow.

⭐⭐⭐ Enjoy your learning!!! ⭐⭐⭐

Seamless GPG Key Migration: Moving Your Keys Across Machines

Ridae HAMDANI — Fri, 29 Dec 2023 10:38:43 +0000

I am using gpg key for a while to encrypt my files as well as sign my git commit, recently I migrated to a new virtual machine, but I come accross this challenge, how can I migrate my gpg keys to my new beast machine ?

In the following article , I will share with you how I migrated my gpg keys.

There is a number of ways to backup or migrate your gpg keys :

Easiest way - Migrate all

The first and the easiest way is to zip and copy all the gpg folder from one machine to an other, this will move all your configuration/keys, it is a good idea when the target machine is a newly fresh machine with no gpg keys yet.

Usually the gpg configuration files are located in the directory ~/.gnupg

Migrate only specific keys

The second way is to move only a specific key.
1- Start by finding the key(s) id you want to migrate by using this command:

gpg --list-secret-keys --keyid-format LONG

It should returns something like:

sec   rsa4096/[**YOUR KEY ID**] 2024-03-30 [SC]
      ABCDEFGHIJKLMNOPQRSTUVWXYZ
uid                 [ unknown] username (KEY NAME) <user@domain>
ssb   rsa4096/ABCDEFGHIJKL 2024-03-30 [E]

After the key size rsa4096/ is your key ID.

2- Export the key in preparation to move it

gpg --export -a [your key id] > gpg-pub.asc

3- Prepare the secret key for migration (if password protected, you’ll be prompted to enter it)

gpg --export-secret-keys -a [your key] > gpg-sc.asc

4- Drag the key pair from the current directory to your USB stick or however else you move them (you can copy/past the first as they are just a text files).

5- Once on the new machine, import them

$ gpg --import gpg-pub.asc
$ gpg --import gpg-sc.asc

If password protected, you’ll be prompted to enter it

⭐⭐⭐ Enjoy your learning….!!! ⭐⭐⭐

Support My Blog ☕️
If you enjoy my technical blog posts and find them valuable, please consider buying me a coffee at here. Your support goes a long way in helping me produce more quality articles and content. If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.

Understanding Allocatable Memory and CPU in Kubernetes Nodes

Ridae HAMDANI — Wed, 04 Oct 2023 07:55:20 +0000

In Kubernetes, allocatable resources refer to the portion of a node’s total resources that can be allocated to running containers or pods. These resources are distinct from the node’s total capacity, which includes all available CPU and memory resources. The key takeaway here is that not all resources on a node are available for use by workloads.

How to find allocatable resources capacity ?



kubectl describe node MY_NODE_1

This command describe your node configuration, the Capacity & Allocatable infomation :



...
Capacity:
  cpu:                12
  ephemeral-storage:  31436544Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             49156248Ki
  pods:               110
Allocatable:
  cpu:                10500m
  ephemeral-storage:  28447630903
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             45908120Ki
  pods:               110
...

As we can see , my node has 12 CPU and only 10.5 CPU is allocatable , 1.5 CPU is reserved for my system + kubelet which is a lot . For the memory, the node has 49Gb and 4Gb are reserved for the OS & kube system , the remain 45Gb is available for the containers.

Understanding Node Capacity and Allocatable Resources

Let’s break down how Kubernetes calculates allocatable resources:

Total Node Capacity: This is the sum of all the physical CPU and memory resources available on the node. It represents the maximum capacity the node can provide for running containers.
*Kubernetes system daemons Reserved resources *: Kubernetes sets aside a portion of the node’s resources for system daemons, such as the kubelet, container runtime, and system monitoring tools. These resources are reserved to ensure that critical cluster components can function effectively.
Node OS Overhead: Some resources are consumed by the operating system itself and are not available for workloads. This includes the kernel, system processes, and filesystem caches. The formula to calculate allocatable resources is: ```

Allocatable = Total Node Capacity - System Reserved Resources - Kube system daemons Reserved resources


## Why Allocatable Resources Matter ?
For several reasons:

- **Resource Guarantees:** Kubernetes ensures that pods receive the resources they request, and allocatable resources are what Kubernetes uses to make these guarantees. If there are not enough allocatable resources on a node to meet a pod’s requirements, scheduling that pod will fail.
- **Efficient Resource Utilization:** Efficiently allocating resources is essential for maximizing node utilization and optimizing costs. By managing allocatable resources effectively, you can prevent resource waste and overcommitting nodes.
- **Node Stability:** If a node runs out of allocatable resources, it can become unstable or unresponsive. Properly managing allocatable resources helps maintain node stability and prevents cluster disruptions.

## What quota should you use ?
The following article highlight the quota used by the GCP,AWS & Azure.


      
        
          
        
      
    
      
        
          Allocatable memory and CPU in Kubernetes Nodes
        
      
        
          Pods deployed in your Kubernetes cluster consume resources such as memory, CPU and storage. However, not all resources in a Node can be used to run Pods.
        
      
          
        learnk8s.io
      
    


  
  
  How to configure the Node Allocatable resources ?


1- Add the --kube-reserved flag to the kubelet command line with the desired CPU memory reservation value. For example, to reserve 500m CPU memory (0.5 CPU cores), add the following line to the [Service] section of the file:

Environment="KUBELET_EXTRA_ARGS=--kube-reserved=cpu=500m,memory=100Mi"

Make sure to specify the desired value according to your requirements.

2- Save the file and exit the text editor.

3- Reload the systemd service configuration to apply the changes:

sudo systemctl daemon-reload

4- Restart the kubelet service to apply the new configuration:

sudo systemctl restart kubelet

5- Verify that the kubelet has started successfully:

sudo systemctl status kubelet



The kubelet should now be running with the updated CPU memory reservation. Repeat these steps on each node where you want to change the CPU memory reservation.

Keep in mind that modifying kubelet configuration on running production clusters can impact node performance, so be cautious and test changes in a controlled environment first. Additionally, ensure that you have backups and a rollback plan in case the changes cause any issues.

What are the top 3 soft skills every software engineer should master?

Ridae HAMDANI — Tue, 07 Jul 2020 10:00:02 +0000

As a software engineer Hard skills are so important for our career. However, a lot of software engineers don't focus on Soft skills.

So What are the top 3 soft skills from your point of view?

Do you use gitbook, try this awesome plugin I wrote

Ridae HAMDANI — Sat, 20 Jun 2020 11:19:45 +0000

It was a time, I was looking for a Gitbook plugin to documente my commands by showing a nice terminal in my page . However I haven't found any existing plugins that satisfy my needs so, as a developer, I decided to write my own plugin and share it with the community.

Say hello to Terminull:

Terminull is a Gitbook plugin allows you to create a modern terminal for your Gitbook pages in order to documente your commands.

Features:

Specify the context directory
Add comment to the command
Copy command by clicking on a button
Show command output

How to use it ?

To use Terminull plugin in your Gitbook project, add the terminull plugin to the book.json file of your project, then install plugins using gitbook install.

{
    "plugins": ["terminull"]
}

Now you are ready to write your hello world terminal.
To create a terminal you can use Code markdown with term as language.
e.g :



```term
gitbook-plugin-terminull$ echo 'hello terminull' # This will print hello terminull
hello terminull
```

As a result you will see this beautiful terminal in your page.

You like my work ❗

If you like this plugin you are welcome to made a pull requests to add more feature

https://github.com/ridaeh/gitbook-plugin-terminull

☸️ Some changes between Helm v2 and Helm v3 that you should know

Ridae HAMDANI — Wed, 29 Jan 2020 13:04:47 +0000

I was working on deploying an application on Kubernetes with Helm v3, then I decided to do the same while using Helm v2 this time to validate the possibility of migration from helm2 to helm3.

In this article, I want to share with you some of the changes between v2 & v3, that I faced in my task when I verified the validation of my Chart and when I deployed my application with these two helm versions.

Helm v3 introduces some changes in:

Adios Tiller
Helm v2 vs v3 commands.
Chart apiVersion.
Chart dependencies.
Helm package command.
Route object in chart.
helm search command.
How to migrate from helm2 to helm3?

1. Adios Tiller:

In Helm v3 tiller is gone, and there is only Helm client 😊.

2. Helm v2 vs v3 commands:

Some commands are not supported or renamed in Helm v3:

Command	Helm2	Helm3
Initialize Helm client/server	init
Download a chart to your local directory	fetch	pull
Given a release name, delete the release from Kubernetes	delete	uninstall
Helm client environment information		env
Displays the location of HELM_HOME	home
Inspect a chart	inspect	show
Uninstalls Tiller from a cluster	reset

3. Chart apiVersion:

Helm decides to increment the chart API version to v2 in Helm3:

# Chart.yaml
-apiVersion: v1 # Helm2
+apiVersion: v2 # Helm3
...

4. Chart dependencies:

Helm v2 chart has a specific file called 'requirements.yaml' where dependencies are added under the dependencies section. With Helm v3 this section are moved from requirements.yaml to Chart.yaml.

More information about this change can be found in the official documentation.

5. Helm package command:

Running helm package command with Helm v2 will raise a directory name (foo) and Chart.yaml name (bar) must match error if the Chart name does not match the Chart root folder name. With Helm v3 this constraint is not compulsory.

6. Route object in Chart:

Unlike the Helm v2, route object in Helm v3 chart requires host and status fields.

apiVersion: v1
kind: Route
metadata:
  name: {{ include "toto.name" . }}
spec:
+  host: {{ .Values.host }}
  to:
    kind: Service
    name: {{ include "toto.name" . }}
    weight: 100
  port:
    targetPort: 'http'
  wildcardPolicy: None
+status: 
+  ingress: []
+  wildcardPolicy: None

7. helm search command:

With helm v2 you can use helm search [CHART_NAME] command to search for a chart in both your repo list and in helm hub.

With Helm v3 you have to specifie where to look for your chart .

Usage:
  helm search [command]

Available Commands:
  hub         search for charts in the Helm Hub or an instance of Monocular
  repo        search repositories for a keyword in charts

8. How to migrate from helm2 to helm3:

Helm team made a good helm plugin called helm-2to3 that facilitates :

Migration of Helm v2 configuration, data, plugin and releases to Helm v3.
Clean up Helm v2 configuration, release data, and Tiller deployment.

Here is full documentation about this plugin: https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/

Happy Helming folks 😊❤️️ !!!

Why do you need to set limits for container resource usage?

Ridae HAMDANI — Tue, 07 Jan 2020 15:08:10 +0000

Why do we need to set limits for container resource usage?

Enforcing resource limits is a critical best practice for running containers on a host or shared platforms like Kubernetes, Openshift...

If a container runs without a resource limitation, it may use all the resources available on the host, producing a disaster in production on a shared platform. This lack of deterministic resource usage could be a big problem for other applications on the host or container orchestrators.

Setting resources limits will:

prevent applications from consuming more than their expected resources on the host.
provide autoscaling controllers critical information needed to add and remove instances of a containerized service based on resource usage.

Setting resources limits for a Docker container is not required by default.
However, every Docker container gets its own Cgroup by default in order to permit you to set resources limits.

Cgroups(Control groups) is a Linux kernel feature for managing and monitoring system resource like CPU, disk I/O, memory and bandwidth usage.
it provides:

Resource limiting: a group can be configured not to exceed a specified resource limit.
Prioritization: one or more groups may be configured to utilize fewer or more CPUs or disk I/O throughput.
Accounting: a group's resource usage is monitored and measured.
Control: groups of processes can be frozen or stopped and restarted.

Cgroups partition those resources into groups then assigning tasks to those groups.

Conclusion:

Providing a stable product in production is a journey, in this brief article we learned the importance of limiting resources usage, the big challenges are to determine the container requirement resources and to configure containerized application runtimes ( e.g: JDK) to stick to the configured resources limits.