Forem: Ali Mehraji

Mattermost Receiver For Alertmanager in Prometheus-Stack in K8S

Ali Mehraji — Thu, 26 Feb 2026 20:18:46 +0000

Overview

Since Prometheus-Stack Alertmanager doesn’t include Mattermost as a built-in receiver, we will integrate it using a webhook via creating a Python relay application that accepts Alertmanager webhook notifications, formats them into a Mattermost-compatible payload, and sends them to a Mattermost incoming webhook URL.

Then we will deploy this relay inside Kubernetes using a Deployment and expose it with a Service so Alertmanager can reliably reach it.

Finally, we will configure Alertmanager using an AlertmanagerConfig resource to define a new receiver and route, so alerts matching our rules (severity/namespace/etc.) get delivered to Mattermost through the relay.

How

Prometheus Alert Rules trigger alerts.
Alertmanager processes and routes the alerts.
Alertmanager sends webhook notifications to the Python Relay Service.
The relay formats the alert payload.
The relay forwards the formatted message to Mattermost Incoming Webhook.

Steps

Create an Incoming Webhook in Mattermost. This webhook URL will be used by the Python relay to deliver alert messages to the desired channel.

Ensure you securely store the generated webhook URL, as it will be injected into the relay application via environment variables or Kubernetes secrets.
Python Relay Application

A lightweight webhook server that receives Alertmanager alerts and forwards them to Mattermost.
Build the Docker Image Dockerfile

Once the relay application is complete, build the image and push it to a container registry accessible by your Kubernetes cluster.
Deploy in Kubernetes: K8S Manifests
Configure Alertmanager with AlertmanagerConfig

Ensure having a label in AlertmanagerConfig to select via prometheus operator

Add the Alertmanagerconfig to prometheus values.yaml

⋮
alertmanager:
  alertmanagerSpec:
    alertmanagerConfigSelector:
      matchLabels:
        alertmanagerConfig: alertmanager-mattermost
    alertmanagerConfigMatcherStrategy:
      type: None
⋮

And then do helm upgrade to prometheus operator selects the alertmanagerConfig and route alerts.

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack -f values.yaml

Docker Remote Context via SSH over Proxy

Ali Mehraji — Tue, 23 Dec 2025 22:26:32 +0000

Securely manage a remote Docker daemon over SSH using Docker contexts, with SSH traffic routed through a SOCKS5 proxy.

If you want to connect to and manage a remote Docker daemon without exposing the Docker API over the network, you can use Docker contexts over SSH. In environments where direct access to the remote server is restricted or must pass through a controlled network path, the SSH connection can be routed through a SOCKS5 proxy.

By configuring SSH to use a proxy and defining a Docker context that targets the remote host via SSH, Docker commands executed locally are transparently forwarded to the remote Docker daemon. This allows you to use standard Docker workflows (docker ps, docker compose, docker build) as if the daemon were local, while still benefiting from SSH key authentication, proxy enforcement, and connection keepalive mechanisms.

SSH Configuration

To connect to a remote server via SSH through a proxy, ProxyCommand is used together with nc (netcat). All SSH connections to the remote host are routed through the specified SOCKS5 (it could be HTTP) proxy.

The wildcard host configuration (vps-*) ensures that any matching host uses the proxy and keepalive settings automatically.

~/.ssh/config

HOST vps-*
  ServerAliveInterval   10
  ProxyCommand nc --proxy-type socks5 --proxy 127.0.0.1:10808 %h %p

Host vps-docker
  HostName <HOST_IP>
  User <USER_NAME>
  Port <HOST_SSH_PORT>
  PreferredAuthentications publickey

ServerAliveInterval: keeps the SSH connection alive by sending periodic packets, preventing timeouts during long-running operations.
ProxyCommand: forces SSH traffic to pass through a local SOCKS5 proxy (e.g. a VPN tunnel).
PreferredAuthentications publickey: Specifies that SSH should prefer (and effectively enforce) public key–based authentication instead of password-based authentication when connecting to the host.

Docker Context

Docker can connect to the remote Docker daemon over SSH.
The SSH connection itself is routed through the configured SOCKS5 proxy , ensuring all Docker traffic to the remote host passes through the proxy.

To define a Docker context:

docker context create vps-docker-host --docker host=ssh://<vps-docker>

To switch and use created context:

docker context use vps-docker-host

Once this context is created and activated, all Docker CLI commands (including docker ps, docker compose, and docker build) will operate against the remote Docker daemon instead of the local one.

Resources

runcmd and package order in Cloud-Init

Ali Mehraji — Mon, 08 Dec 2025 00:45:58 +0000

During boot, cloud-init identifies the cloud it is running on and initializes the system accordingly. Cloud instances will automatically be provisioned during first boot with networking, storage, SSH keys, packages and various other system aspects already configured.

running Cloud-Init is the job of the cloud-init-local, cloud-init, cloud-config and cloud-final init scripts in /etc/init.d. Cloud-Init can be configured to carry out a wide range of tasks such as adding yum or apt repositories, writing files, creating users and groups, and bootstrapping configuration management.

How is it possible to run a command via runcmd before installing its related package via packages in Cloud-Init? Which one runs first ?

In a Cloud-Init configuration, it may look strange that a command referencing a service can run before the package providing that service is installed. For example:

#cloud-config
︙
package_update: true
package_upgrade: true

runcmd:
  - systemctl enable --now qemu-guest-agent.service

packages:
  - qemu-guest-agent
︙

In this setup, commonly used when provisioning VMs with Terraform’s libvirt provider on QEMU/KVM, runcmd attempts to enable and start qemu-guest-agent.service even though the package providing it (qemu-guest-agent) hasn’t been installed yet. The guest agent is often required early so tools like Ansible’s dynamic inventory can query host information.

So how can Cloud-Init run systemctl enable --now qemu-guest-agent.service before the package exists?

The sentence “The runcmd module runs very late in the boot process” can be confusing at first glance, because in Cloud-Init’s documented module stages, runcmd appears in the Config stage while package installation via packages happens in the Final stage.

It looks like runcmd runs before packages are installed and it wont work.

However, The runcmd module does not execute commands directly:

The runcmd module only writes the script to be run later. The module that actually runs the script is scripts_user in the Final boot stage.

So even though runcmd appears in the Config stage, the commands you place under runcmd are not executed immediately. They are scheduled to run later, after packages have been installed and other finalization tasks are complete.

By the time the scripts_user module runs (in the Final stage), the qemu-guest-agent is installed, qemu-guest-agent.service exists and systemctl enable --now will be succeed.

The runcmd scripts will be written in /var/lib/cloud/instance/scripts/runcmd in the instance.

cat /var/lib/cloud/instance/scripts/runcmd

#!/bin/sh
systemctl enable --now qemu-guest-agent

cloud-config Example:

#cloud-config
users:
  - name: USERNAME
    groups: users, admin
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    gecos: "DevOps Engineer"
    passwd: "SHA-512 hashed password" # mkpasswd --method=SHA-512
    shell: /bin/bash
    ssh_authorized_keys:
      - "ssh-ed25519 ..."
      - "ssh-rsa ..."

ssh_pwauth: false

package_update: true
package_upgrade: true

apt:
  # file /etc/apt/apt.conf.d/cloud-init-aptproxy will be written
  http_proxy: http://<proxy-server>:port
  https_proxy: http://<proxy-server>:port

write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      Port 2200
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 2
      AllowTcpForwarding no
      X11Forwarding no
      AllowAgentForwarding no
      AuthorizedKeysFile .ssh/authorized_keys

runcmd:
  - systemctl enable --now qemu-guest-agent

packages:
  - qemu-guest-agent
  - htop
  - tree

Resources

OOMKill in Kubernetes and Linux (Exit Code 137)

Ali Mehraji — Mon, 22 Sep 2025 13:23:25 +0000

When a container(process) is terminated due to an Out Of Memory (OOM) manager condition, Kubernetes marks it as OOMKilled. In this case , the Linux kernel’s OOM Killer terminates the process and the container exits with exit code 137. This exit code is an important troubleshooting signal in Kubernetes.

Why OOMKill Happens

Linux uses an over-commitment policy for memory management. Applications can request more memory than is physically available. When the actual demand exceeds system capacity, the kernel must decide which process to kill in order to reclaim memory. How OOM Kill works ?

The OOM Killer selects and terminates processes based on their OOM score.
In Kubernetes, this usually happens when a container exceeds its memory limit.
Kubernetes then records the container’s state as OOMKilled, and logs exit code 137. OOM Killer Sends What Signal to the Process ?

Exit codes 1 - 2, 126 - 165, and 255 have special meanings, and should therefore be avoided for user-specified exit parameters. Ending a script with exit 127 would certainly cause confusion when troubleshooting (is the error code a "command not found" or a user-defined one?).

Resources

Kubernetes Ephemeral Containers and kubectl debug Command

Ali Mehraji — Sat, 23 Aug 2025 22:29:01 +0000

This post is only a brief overview, with references collected at the end. If you’d like to dive deeper, I recommend starting with Deep Diving Kubernetes Ephemeral Containers and kubectl debug Command and then following the related Task: Copy Files To/From a Distroless Kubernetes Pod.

Ephemeral containers

Ephemeral containers provide a powerful way to debug applications running in Kubernetes. Unlike regular containers, they are not part of the pod’s original specification but can be injected into a running pod when needed. This makes them especially valuable for interactive troubleshooting, particularly when kubectl exec is insufficient—for example, when a container has already crashed or when the original image lacks essential debugging tools.

Modern container practices often favor minimal or distroless images to improve security and performance. In many cases, base images are stripped down to the essential. sometimes even using scratch with nothing but the application binary. While this approach:

have a smaller attack vector area.
have faster-scanning performance.
Reduced image size.
have a faster build and CD/CI cycle.
have fewer dependencies.

It also means that traditional debugging utilities (like a shell or package manager) are unavailable.

Ephemeral containers allow engineers to inject a temporary container image that includes the required debugging tools, without altering the original pod definition. This is particularly useful for inspecting the live state of an application, reproducing elusive issues, and running arbitrary commands inside a pod—giving teams the best of both worlds: lean production images with flexible on-demand debugging capabilities.

Share Process

While troubleshooting a Pod, I'd typically want to see the processes of all pods containers, as well as I'd be interested in exploring their filesystems. Can an ephemeral container be a little more unraveling?

However, if you explore the filesystem with ls, you'll notice that it's a filesystem of the ephemeral container itself (busybox in our case), and not of any other container in the Pod. Well, it makes sense - containers in a Pod (typically) share net, ipc, and uts namespaces, they can share the pid namespace, but they never share the mnt namespace. Also, filesystems of different containers always stay independent. Otherwise, all sorts of collisions would start happening.

Understanding process namespace sharing:

The container process no longer has PID 1. Some containers refuse to start without PID 1 (for example, containers using systemd) or run commands like kill -HUP 1 to signal the container process. In pods with a shared process namespace, kill -HUP 1 will signal the pod sandbox (/pause).
Processes are visible to other containers in the pod. This includes all information visible in /proc, such as passwords that were passed as arguments or environment variables. These are protected only by regular Unix permissions.
Container filesystems are visible to other containers. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions.Thus, if you know a PID of a process from the container you want to explore, you can find its filesystem at /proc/<PID>/root from inside the debugging container.

Lets get hands dirty

Create pod with a distroless image.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: distroless-nginx
  labels:
    app: distroless-nginx
spec:
  selector:
    matchLabels:
      app: distroless-nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: distroless-nginx
    spec:
      containers:
        - name: distroless-nginx-ctr
          image: cgr.dev/chainguard/nginx

After the pod is created, try to get shell via kubectl exec (try with sh, ash , ...):

k exec -it -n default distroless-nginx-cbbdbd698-49fjr -- bash

It wont be able to give you typical TTY shell.

error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "c336401c1da1996fc382a2c8aefd82552a60b6de52da1d12aec0d3b5f9b3b65a" : OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH

`kubectl debug`

POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --image=busybox ${POD_NAME}

When you inject an ephemeralContainer into a pod, it behaves like any other container, unless you explicitly grant it additional privileges or enable process and filesystem sharing with the target container.

So, unless your workloads run with shareProcessNamespace enabled by default (which is unlikely a good idea since it reduces the isolation between containers), we still need a better solution.

POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --target=distroless-nginx-ctr --image=busybox ${POD_NAME}

Targeting container "distroless-nginx-ctr". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
If you don't see a command prompt, try pressing enter.

/ # ps aux
PID   USER     TIME  COMMAND
    1 65532     0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf -e /dev/stderr -g daemon off;
    7 65532     0:00 nginx: worker process
    8 65532     0:00 nginx: worker process
    9 65532     0:00 nginx: worker process
   10 65532     0:00 nginx: worker process
   17 root      0:00 sh
   22 root      0:00 ps aux
/ #

In this case the debugger container can access to other processes. what does the --target do ?

k debug --help

--target='':
  When using an ephemeral container, target processes in this container name.

Note:

The --target parameter must be supported by the Container Runtime. When not supported, the Ephemeral Container may not be started, or it may be started with an isolated process namespace so that ps does not reveal processes in other containers.

Now, let’s say we want to modify the distroless-nginx-ctr container’s /etc/nginx/nginx.conf. Immediately we run into a problem: we don’t even have permission to change directories with cd, nor can we list the root directory of the target container.

~ # ls /proc/1/root
ls: /proc/1/root: Permission denied
~ # cd /proc/1/root
sh: cd: can't cd to /proc/1/root: Permission denied
~ #

This highlights a key limitation—process sharing alone isn’t enough. To properly inspect and interact with the target container’s filesystem, we also need elevated privileges. That’s where the --profile flag comes in, as it can grant the necessary access for debugging and modification.

POD_NAME=$(kubectl get pods -l app=distroless-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl debug -it -c debugger --target=distroless-nginx-ctr --profile=sysadmin --image=busybox ${POD_NAME}

Note:

kubectl debug automatically generates a container name if you don't choose one using the --container flag.
The -i flag causes kubectl debug to attach to the new container by default. You can prevent this by specifying --attach=false. If your session becomes disconnected you can reattach using kubectl attach.
The --share-processes allows the containers in this Pod to see processes from the other containers in the Pod. For more information about how this works, see Share Process Namespace between Containers in a Pod
Don't forget to clean up the debugging Pod when you're finished with it.
Install a Kubernetes policy engine (OPA/Gatekeeper) to prevent exposing your cluster:
- Block privileged debug containers
- Require automountServiceAccountToken: false
- Implement just-in-time access for debugging
- Use managed debug tools like Teleport instead of raw kubectl debug
Debug Session Monitoring Now we alert on:
- Added Debug Session Monitoring
- Any ephemeral container creation
- Service account token access from unexpected pods
- Debug sessions lasting >5 minutes

`kubectl debug node`

k debug node/debian-11 -it --image-pull-policy='IfNotPresent' --image=ubuntu --profile=sysadmin

When creating a debugging session on a node, keep in mind that:

kubectl debug automatically generates the name of the new Pod based on the name of the Node.
The root filesystem of the Node will be mounted at /host.
The container runs in the host IPC, Network, and PID namespaces, although the pod isn't privileged, so reading some process information may fail, and chroot /host may fail.
If you need a privileged pod, create it manually or use the --profile=sysadmin flag.

To debug node like an SSH session, just invoke chroot /host.

Creating debugging pod node-debugger-debian-11-wrjhs with container debugger on node debian-11.
If you don't see a command prompt, try pressing enter.
/ # chroot /host

Don't forget to clean up the debugging Pod when you're finished with it:

kubectl delete pod node-debugger-debian-11-wrjhs

Sidecar container via Docker

How docker exec and cdebug works ?

How to reproduce the docker exec-like functionality with the docker run command by starting a new container in the target container's namespaces:

docker run -itd --name sidecar --pid container:target --ipc container:target --network container:target alpine sh

Docker Debug

Resources

Python Env Variables in Dockerfile

Ali Mehraji — Fri, 25 Jul 2025 17:45:36 +0000

It is necessary to set Python-specific environment variables in Dockerfile. Some of these variables are optional, while others are essential.

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_TIMEOUT=60 \
    PIP_INDEX_URL=https://${JFROG}/artifactory/api/pypi/python/simple/

PYTHONUNBUFFERED

Force the stdout and stderr streams to be unbuffered. This option has no effect on the stdin stream.

Ensure logs appear immediately in the container logs, Disables output buffering for stdout and stderr.
Crucial for:
- Debugging or monitoring logs (e.g., in docker logs or CI pipelines).
- Streaming logs to systems like ELK, Loki, etc.

PYTHONDONTWRITEBYTECODE

If this is set to a non-empty string, Python won’t try to write .pyc files on the import of source modules. This is equivalent to specifying the -B option.

Disables writing .pyc files (compiled bytecode) to disk.
Prevents Python from creating:
- __pycache__/
- .pyc files
Keeps the container clean and lightweight.
Avoids unnecessary file writes (especially important on container read-only filesystems or volume mounts).

PIP_INDEX_URL

This environment variable is used for on-premises/private PYPI artifactory.

pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME>.

Dashes (-) have to be replaced with underscores (_).

Example:

--disable-pip-version-check --> PIP_DISABLE_PIP_VERSION_CHECK
--timeout --> PIP_TIMEOUT
--no-color --> PIP_NO_COLOR

invoke pip --help to get the cli options (flags)

  General Options:
    --proxy <proxy>             Specify a proxy in the form scheme://[user:passwd@]proxy.server:port.
    --retries <retries>         Maximum attempts to establish a new HTTP connection. (default: 5)
    --timeout <sec>             Set the socket timeout (default 15 seconds).
    --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
    --trusted-host <hostname>   Mark this host or host:port pair as trusted, even though it does not have valid or any HTTPS.
    --cert <path>               Path to PEM-encoded CA certificate bundle. If provided, overrides the default. See 'SSL Certificate
                                Verification' in pip documentation for more information.
    --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM
                                format.
    --cache-dir <dir>           Store the cache data in <dir>.
    --no-cache-dir              Disable the cache.
    --disable-pip-version-check Dont periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.
    --no-color                  Suppress colored output.
    --use-feature <feature>     Enable new functionality, that may be backward incompatible.
    --use-deprecated <feature>  Enable deprecated functionality, that will be removed in the future.
    --resume-retries <resume_retries>
                                Maximum attempts to resume or restart an incomplete download. (default: 0)

Notice

Security best practices in K8S

Running processes as a non-root user using USER nonroot:nonroot in the Dockerfile allows Kubernetes SecurityContext settings such as fsGroup, runAsGroup, and runAsUser to take effect in the pod template. Additionally, because the filesystem is set to read-only, PYTHONDONTWRITEBYTECODE=1 must be set to prevent Python from writing .pyc files.

Resources

TIL Kubectl CSA/SSA (Client/Server Side Apply)

Ali Mehraji — Thu, 05 Jun 2025 14:39:43 +0000

Kubectl CSA(Client Side Apply) and SSA(Server Side Apply).

The need to read and dig into the SSA and CSA came from command Annotation in Installation Requirements manifests ? and which was validating with dry-run that Question Came from a TIL about nsenter.

Field management

The Kubernetes API server tracks managed fields for all newly created objects. When trying to apply an object, fields that have a different value AND are owned by another manager will result in a conflict. This is done in order to signal that the operation might undo another collaborator's changes. Writes to objects with managed fields can be forced, in which case the value of any conflicted field will be overridden, and the ownership will be transferred.

Whenever a field's value does change, ownership moves from its current manager to the manager making the change.

Compared to the (legacy) kubectl.kubernetes.io/last-applied-configuration annotation managed by kubectl, Server-Side Apply uses a more declarative approach, that tracks a user's (or client's) field management, rather than a user's last applied state. As a side effect of using Server-Side Apply, information about which field manager manages each field in an object also becomes available.

Field management details are stored in a managedFields field that is part of an object's metadata.
The field management record consists of basic information about the managing entity itself, plus details about the fields being managed and the relevant operation (Apply or Update). If the request that last changed that field was a Server-Side Apply patch then the value of operation is Apply; otherwise, it is Update.

kubectl get omits managed fields by default. Add --show-managed-fields to show managedFields when the output format is either json or yaml.

Apply

Lets apply(create) a pod then check those fields.

netshoot.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: netshoot
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot:latest
      command: ["sleep", "infinity"]
      securityContext:
        capabilities:
          add: ["NET_ADMIN", "SYS_MODULE"]
  restartPolicy: OnFailure

kubectl apply --server-side --dry-run=server -f netshoot.yaml -o yaml -n test-tool --show-managed-fields | yq .metadata.managedFields

managedFields

managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:containers:
          k:{"name":"netshoot"}:
            .: {}
            f:command: {}
            f:image: {}
            f:name: {}
            f:securityContext:
              f:capabilities:
                f:add: {}
        f:restartPolicy: {}
    manager: kubectl
    operation: Apply
    time: "2025-06-04T20:43:41Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cni.projectcalico.org/containerID: {}
          f:cni.projectcalico.org/podIP: {}
          f:cni.projectcalico.org/podIPs: {}
    manager: calico
    operation: Update
    subresource: status
    time: "2025-06-04T20:43:42Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:ambient.istio.io/redirection: {}
    manager: install-cni
    operation: Update
    subresource: status
    time: "2025-06-04T20:43:42Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.237.81.196"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    subresource: status
    time: "2025-06-04T20:43:43Z"

Change some fields in `magaedFields`

A Server-Side Apply patch request requires the client to provide its identity as a field manager. When using Server-Side Apply, trying to change a field that is controlled by a different manager results in a rejected request unless the client forces an override. For details of overrides, see Conflicts.

Invoke below command to save the current(live) state of the pod in a file, netshoot-current.yaml.

k get -n test-tool pod netshoot --show-managed-fields -o yaml | tee netshoot-current.yaml

Change the podIP and podIPs, and apply it via:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: CHANGE_LAST_OCTET
    cni.projectcalico.org/podIPs: CHANGE_LAST_OCTET

kubectl apply --server-side -f netshoot-current.yaml -n test-tool

As we expected the Conflict happened, because

The owner of those ips fields is the CNI, in this case is calico.
AND
The Value of those IP Fields are different.

Error from server (BadRequest): metadata.managedFields must be nil

- apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cni.projectcalico.org/containerID: {}
          f:cni.projectcalico.org/podIP: {}
          f:cni.projectcalico.org/podIPs: {}
    manager: calico
    operation: Update
    subresource: status
    time: "2025-06-04T20:43:42Z"

Lets do the apply command verbose with -v=6 to see what exactly happening with imperative endpoints under the hood.

kubectl apply --server-side -f netshoot-current.yaml-n test-tool -v=6

"Response" verb="PATCH" url="https://rancher.example.com/k8s/clusters/<CLUSTER_ID>/api/v1/namespaces/test-tool/pods/netshoot?fieldManager=kubectl&fieldValidation=Strict&force=false" status="400 Bad Request" milliseconds=80

`kubectl apply --server-side` (rejected with Bad Request)

Respects field ownership strictly
- fieldValidation=Strict in query params in PATCH request.
Calico owns the PodIP/PodIPs fields
- apply is going to change the calico field manager to fieldManager=kubectl query param in PATCH request.
- For other updates, the API server infers a Field manager identity from the "User-Agent:" HTTP header (if present).
- The force is false in the server-side apply to respect the ownership.

Kubernetes server rejects the conflict because you don't own those fields

status="400 Bad Request in response, This is the correct protective behavior.

Bad Request" milliseconds=80
I0605 00:40:45.502024   24714 helpers.go:246] server response object: %s[{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "metadata.managedFields must be nil",
  "reason": "BadRequest",
  "code": 400
}]
Error from server (BadRequest): metadata.managedFields must be nil

`kubectl apply` or `kubectl edit` (succeeded)

If we don't use the --server-side which is going to be the default apply (CSA), in that case the apply wil use the default force=true in CSA.

It is however possible to change .metadata.managedFields through an update, or through a patch operation that does not use Server-Side Apply. Doing so is highly discouraged, but might be a reasonable option to try if, for example, the .metadata.managedFields get into an inconsistent state (which should not happen in normal operations).

The format of managedFields is described in the Kubernetes API reference.

Caution:

The .metadata.managedFields field is managed by the API server. You should avoid updating it manually.

"Response" verb="PATCH" url="https://rancher.example.com/k8s/clusters/<CLUSTER_ID>/api/v1/namespaces/test-tool/pods/netshoot?fieldManager=kubectl-client-side-apply&fieldValidation=Strict" status="200 OK" milliseconds=97
pod/netshoot configured

Uses Client-Side Apply (strategic merge patch)
- fieldManager=kubectl-client-side-apply in query params in PATCH request.
Calculates the patch on the client side
Sends a PATCH request that can override existing fields
Less strict about field ownership conflicts
- It does validate with fieldValidation=Strict in query param in PATCH request, but the force is force.
Essentially "forces" the change through

As you can see, the manager(owner) of the podIP and podIPs fields changed to kubectl-client-side-apply and the operation is Update.

- apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
    f:metadata:
      f:annotations:
        f:cni.projectcalico.org/podIP: {}
        f:cni.projectcalico.org/podIPs: {}
        f:kubectl.kubernetes.io/last-applied-configuration: {}
  manager: kubectl-client-side-apply
  operation: Update
  time: "2025-06-04T21:26:09Z"

Conflict Example

Scenario:

Apply netshoot-deployment.yaml deployment with 1 replicas, no server-side, just as normal (CSA)
- Sets the Owner(manager) and operation of the replicas field to
```
manager: kubectl-client-side-apply
operation: Update
```
- The kubectl.kubernetes.io/last-applied-configuration annotation exists.
- For the first time apply the manager for the whole fields is kubectl-client-side-apply.
Scale the pods with

  kubectl scale deployment netshoot -n test-tool --replicas 3

ad-hoc scale does not update the last-applied-configuration , just sets a new owner(manager) to the replicas field.
```
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
  f:spec:
    f:replicas: {}
manager: kubectl
operation: Update
subresource: scale
```
- Another scale with changing the replicas in manifest and apply with --server-side, for example replicas: 5

  kubectl apply --server-side -f netshoot-deployment.yaml -n test-tool -v=6

Conflict status="409 Conflict"

"Response" verb="PATCH" url="https://rancher.example.com/k8s/clusters/<CLUSTER_ID>/apis/apps/v1/namespaces/test-tool/deployments/netshoot?fieldManager=kubectl&fieldValidation=Strict&force=false" status="409 Conflict" milliseconds=75

```bash
error: Apply failed with 1 conflict: conflict with "kubectl" with subresource "scale" using apps/v1: .spec.replicas
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts
```

Why we expect a conflict? > A conflict is a special status error that occurs when an Apply operation tries to change a field that another manager also claims to manage. This prevents an applier from unintentionally overwriting the value set by another user.

ArgoCD

This option enables Kubernetes Server-Side Apply.

By default, Argo CD executes kubectl apply operation to apply the configuration stored in Git. This is a client side operation that relies on kubectl.kubernetes.io/last-applied-configuration annotation to store the previous resource state.

However, there are some cases where you want to use kubectl apply --server-side over kubectl apply:

Resource is too big to fit in 262144 bytes allowed annotation size.
Patching of existing resources on the cluster that are not fully managed by Argo CD.
Use a more declarative approach, which tracks a user's field management, rather than a user's last applied state.
- If ServerSideApply=true sync option is set, Argo CD will use kubectl apply --server-side command to apply changes.

Argocd and HPA, The Question is how to control ArgoCD from interfering a Deployment Replicas when the HPA is Enabled for ?

Environment

Kubernetes cluster deployed with Kubespray and added to a rancher cluster manually.

Resources

TIL `nsenter`

Ali Mehraji — Wed, 28 May 2025 13:05:36 +0000

Host Level operations on a K8S node with privileged container.

Annotations

Set an annotation named command and use yaml anchor &cmd to use it later.

In YAML, &cmd creates an anchor named cmd. You can later reference it using *cmd.

Kubernetes Annotations Document

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.
You can use either labels or annotations to attach metadata to Kubernetes objects. Labels can be used to select objects and to find collections of objects that satisfy certain conditions. In contrast, annotations are not used to identify and select objects. The metadata in an annotation can be small or large, structured or unstructured, and can include characters not permitted by labels. It is possible to use labels as well as annotations in the metadata of the same object.

But wait, the &cmd yaml anchor does nothing to do with the annotation, so im waiting for the discussion.

[QUESTION] command Annotation in Installation Requirements manifests ?

[Update]: Discussion Answer

Longhorn NFS Installation

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: longhorn-nfs-installation
  namespace: longhorn-system
  labels:
    app: longhorn-nfs-installation
  annotations:
    command: &cmd OS=$(grep -E "^ID_LIKE=" /etc/os-release | cut -d '=' -f 2); if [[ -z "${OS}" ]]; then OS=$(grep -E "^ID=" /etc/os-release | cut -d '=' -f 2); fi; if [[ "${OS}" == *"debian"* ]]; then sudo apt-get update -q -y && sudo apt-get install -q -y nfs-common && sudo modprobe nfs; elif [[ "${OS}" == *"suse"* ]]; then sudo zypper --gpg-auto-import-keys -q refresh && sudo zypper --gpg-auto-import-keys -q install -y nfs-client && sudo modprobe nfs; else sudo yum makecache -q -y && sudo yum --setopt=tsflags=noscripts install -q -y nfs-utils && sudo modprobe nfs; fi && if [ $? -eq 0 ]; then echo "nfs install successfully"; else echo "nfs install failed error code $?"; fi
spec:
  selector:
    matchLabels:
      app: longhorn-nfs-installation
  template:
    metadata:
      labels:
        app: longhorn-nfs-installation
    spec:
      hostNetwork: true
      hostPID: true
      initContainers:
      - name: nfs-installation
        command:
          - nsenter
          - --mount=/proc/1/ns/mnt
          - --
          - bash
          - -c
          - *cmd
        image: alpine:3.12
        securityContext:
          privileged: true
      containers:
      - name: sleep
        image: registry.k8s.io/pause:3.1
  updateStrategy:
    type: RollingUpdate

Host Level Operation on K8S Node within a container

Was wondering how a container can install a package or do host level commands!?

spec:
      hostNetwork: true
      hostPID: true
      initContainers:
      - name: nfs-installation
        command:
          - nsenter
          - --mount=/proc/1/ns/mnt
          - --
          - bash
          - -c
          - *cmd
        image: alpine:3.12
        securityContext:
          privileged: true

`command` breakdown

nsenter

nsenter is a Linux utility used to enter different namespaces of other processes — like the network, mount, PID, UTS, etc. This is useful when you want to operate inside another process's context (like the host) from within a container.

--mount=/proc/1/ns/mnt

enter the mount namespace of process 1 (usually the host system’s init process inside the node).
/proc/1/ns/mnt special file that represents the mount namespace of PID 1 (host).

-- bash -c *cmd

Separator; everything after the -- is passed to the new shell.
new shell runs the cmd, which is defined in the annotation of Daemonset, and as described in the K8S Annotaion document, the container can access to command annotation.

For the host(node) level operations, the nsenter needs privileges

hostNetwork: true

What it does:
- Makes the Pod (and all containers in it) use the host’s network namespace.
- That means it shares the host’s network stack — same IP address, routing table, etc.
Why it’s needed:
- If the command inside the init container needs to:
- Access network services that are only available on the host network (e.g., certain NFS servers or internal routes),
- Or resolve hostnames as the host does, this gives it direct access.

  kubectl explain ds.spec.template.spec.hostNetwork

  hostNetwork   <boolean>
    Host networking requested for this pod. Use the host's network namespace. If
    this option is set, the ports that will be used must be specified. Default
    to false.

hostPID: true

What it does:
- Makes the Pod share the host’s PID namespace.
- So the container can see and interact with host processes, including PID 1.
Why it’s needed:
- Using --mount=/proc/1/ns/mnt, path points to PID 1’s mount namespace — which is usually the host’s root process (init/systemd).
- Without hostPID: true, PID 1 inside the container is not the host’s PID 1 — and nsenter would fail because /proc/1/ns/mnt would refer to the container’s own namespace, not the host’s.

  kubectl explain ds.spec.template.spec.hostPID

    hostPID <boolean>
    Use the host's pid namespace. Optional: Default to false.

initContainers.securityContext.privileged: true

  kubectl explain ds.spec.template.spec.initContainers.securityContext.privileged

  Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host.

Additional notes from ChatGPT

🧠 What Happens When You Use It?

With nsenter --mount=/proc/1/ns/mnt,You are telling your container

"Hey, stop using your own isolated mount namespace. Enter the host's mount namespace."
This lets your process (inside a container) perform filesystem operations as if it were on the host.

For example:

nsenter --mount=/proc/1/ns/mnt -- bash -c 'mount -t nfs ...'

🔒 Permissions Required

To access /proc/1/ns/mnt:

You must have CAP_SYS_ADMIN.
In Kubernetes, that typically means the container must be run as privileged: true.
You also need hostPID: true to ensure PID 1 is the host PID 1, not the container's own init.

Resources

Offline, Multistage Python Dockerfile

Ali Mehraji — Mon, 26 May 2025 21:00:16 +0000

There Was a need to reduce a python application docker image size, also had to have offline installation either in the PYPI packages installation or in APT update and installing packages.

Lets get to the Optimization Line by Line.

Docker build `buildx` `syntax`

First of all make sure docker uses the docker buildx as its default docker build.

BuildKit

If you have installed Docker Desktop, you don't need to enable BuildKit. If you are running a version of Docker Engine version earlier than 23.0, you can enable BuildKit either by setting an environment variable, or by making BuildKit the default setting in the daemon configuration.

DOCKER_BUILDKIT=1 docker build --file /path/to/dockerfile -t docker_image_name:tag

# syntax=docker/dockerfile:1.4 for the heredoc in Dockerfile.

# syntax=docker/dockerfile:1.4 # Required for heredocs [3, 4]

Project Directory Tree

├── main.py
├── requirements.txt
└── src
    ├── log.py
    └── prometheus.py

Multistage Dockerfile

as mentioned before, at first provisioning the base stage to be used in the next build and runtime stages.

`base` stage

base image

ARG JFROG=jfrog.example.com

FROM ${JFROG}/docker/python:3.13-slim AS base

Change the default SHELL
- A safe way with custom shell with pipefail and errexit options, its very useful in the Heredoc in the Debian Private repo setup section.

SHELL ["/bin/bash", "-c", "-o", "pipefail", "-o", "errexit"]

Environments, see Python Env Variables in Dockerfile

ARG JFROG=jfrog.example.com

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_TIMEOUT=60 \
    PIP_INDEX_URL=https://${JFROG}/artifactory/api/pypi/python/simple/

Private Debian Repository (Offline Installation)

Used heredoc in Docker to change the base image apt sources to update and install packages from private Debian repository. heredoc needs the dockerfile syntax mentioned before.
If the structure of the Debian repo is different in a private repo , please change the URIs.

DEB822 format (apt .sources files)

# Using DEB822 format (.sources files) - for newer systems
RUN <<EOF

CODENAME=$(grep VERSION_CODENAME /etc/os-release | cut -d'=' -f2)
DISTRO=$(grep '^ID=' /etc/os-release | cut -d'=' -f2)

cat > /etc/apt/sources.list.d/debian.sources <<SOURCE_FILE_CONTENT
Types: deb
URIs: https://${JFROG}/artifactory/debian/debian/
Suites: ${CODENAME} ${CODENAME}-updates
Components: main
Trusted: true

Types: deb
URIs: https://${JFROG}/artifactory/debian/debian-security/
Suites: ${CODENAME}-security
Components: main
Trusted: true
SOURCE_FILE_CONTENT
EOF

Install Shared and common packages in all stages.
- In the package installation there is no need to install recommended packages to reduces the image size.
- After installation, for the sake of size image there is need to remove packages downloads.

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    gnupg \
    lsb-release \
    && rm -rf /var/lib/apt/lists/*

`build` stage

Use the prepared base image as build image

FROM base AS build

There was no need for build specific packages in all stages, so just install them in build stage.

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential && \
    rm -rf /var/lib/apt/lists/*

Install requirements
- Change directory to app.
- Create virtualenv, in the runtime stage, the virtualenv will be copied in image.
- use the cache mount for faster build.
- For the sake of image size install requirements with disabling pip cache with --no-cache-dir flag.

WORKDIR /app

RUN python -m venv .venv
ENV PATH="/app/.venv/bin:$PATH"

COPY requirements.txt .

RUN --mount=type=cache,target=/root/.cache/pip \
pip --timeout 100 install --no-cache-dir -r requirements.txt

`runtime` stage

Use the prepared base image as build image

FROM base AS build

WORKDIR /app

Security best practices
- Create group and user to leverage the kubernetes runAsUser, runAsGroup and fsGroup securityContext

RUN addgroup --gid 1001 --system nonroot && \
    adduser --no-create-home --shell /bin/false \
    --disabled-password --uid 1001 --system --group nonroot

USER nonroot:nonroot

VirtualEnv
- Add the /app/.venv/bin into PATH.
- Copy the virtualenv from build stage.

ENV VIRTUAL_ENV=/app/.venv \
    PATH="/app/.venv/bin:$PATH"

COPY --from=build --chown=nonroot:nonroot /app/.venv /app/.venv

Copy src directory.

COPY --chown=nonroot:nonroot src /app/src
COPY --chown=nonroot:nonroot main.py .

CMD to run container from image.

CMD ["python", "/app/main.py"]

Before And After The optimization

Before The Optimization

The Dockerfile was:

FROM jfrog.example.com/docker/python:latest

WORKDIR /app
ADD src/ .

RUN pip config set global.index-url https://jfrog.example.com/artifactory/api/pypi/python/simple/ &&  \
    pip --timeout 100 install -r requirements.txt

CMD ["python","-u","main.py"]

After build its size was 1.02GB.

Final Dockerfile After Optimization

After all Optimization and multistage Dockerfile its size reduced to 242MB.

# syntax=docker/dockerfile:1.4 # Required for heredocs [3, 4]

ARG PYTHON_VERSION=3.12.3
ARG JFROG=jfrog.example.com

FROM ${JFROG}/docker/python:${PYTHON_VERSION}-slim AS base
SHELL ["/bin/bash", "-c", "-o", "pipefail", "-o", "errexit"]

ARG JFROG=jfrog.example.com

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_TIMEOUT=60 \
    PIP_INDEX_URL=https://${JFROG}/artifactory/api/pypi/python/simple/

# Using DEB822 format (.sources files) - for newer systems
RUN <<EOF

CODENAME=$(grep VERSION_CODENAME /etc/os-release | cut -d'=' -f2)
# DISTRO=$(grep '^ID=' /etc/os-release | cut -d'=' -f2)

cat > /etc/apt/sources.list.d/debian.sources <<SOURCE_FILE_CONTENT
Types: deb
URIs: https://${JFROG}/artifactory/debian/debian/
Suites: ${CODENAME} ${CODENAME}-updates
Components: main
Trusted: true

Types: deb
URIs: https://${JFROG}/artifactory/debian/debian-security/
Suites: ${CODENAME}-security
Components: main
Trusted: true
SOURCE_FILE_CONTENT
EOF

# securely copy .netrc using BuildKit secrets
RUN --mount=type=secret,id=netrc,target=/root/.netrc \
    apt-get update && apt-get install --no-install-recommends --no-install-suggests -y \
      ca-certificates \
      gnupg \
      curl \
    && apt-get clean \
    && apt-get remove --purge --auto-remove -y \
    && rm -rf /var/lib/apt/lists/*

FROM base AS build

# securely copy .netrc using BuildKit secrets
RUN --mount=type=secret,id=netrc,target=/root/.netrc \
    apt-get update && apt-get install --no-install-recommends --no-install-suggests -y \
      build-essential \
    && apt-get clean \
    && apt-get remove --purge --auto-remove -y \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

RUN python -m venv .venv
ENV PATH="/app/.venv/bin:$PATH"

COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --timeout 100 --no-cache-dir --upgrade pip && \
    pip install --timeout 100 --no-cache-dir -r requirements.txt

FROM base AS runtime

WORKDIR /app

RUN addgroup --gid 1001 --system nonroot && \
    adduser --no-create-home --shell /bin/false \
    --disabled-password --uid 1001 --system --group nonroot

USER nonroot:nonroot

ENV VIRTUAL_ENV=/app/.venv \
    PATH="/app/.venv/bin:$PATH"

COPY --from=build --chown=nonroot:nonroot /app/.venv /app/.venv
COPY --chown=nonroot:nonroot src /app/src
COPY --chown=nonroot:nonroot main.py .

CMD ["python", "/app/main.py"]

Notice

After a second thought, realized there is no need for dockerfile syntax version 1.4 to manipulate the apt sources, it could be done with sed.

The new dockerfile syntax was fun to learn, so i keep this guide as it is.

The Dockerfile with no new syntax and manipulating the apt sources via sed.

Updates

Mon Jul 14 2025

Add Credentials for Artifactories with authentication
handled via .netrc and mount it in build time via Buildkit
no credential exposed in image history or saved in image with Secret Mount
Added Line

RUN --mount=type=secret,id=netrc,target=/root/.netrc

After update, updating and installing packages steps

# securely copy .netrc using BuildKit secrets
RUN --mount=type=secret,id=netrc,target=/root/.netrc \
    apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

Resources

Alertmanager with Telegram for Prometheus-Stack in K8S

Ali Mehraji — Sun, 09 Mar 2025 12:06:12 +0000

How to create a Telegram bot

To create a Telegram bot, you'll need to have the Telegram app installed on your computer. If you don't have it already, you can download it from the Telegram website.
Connect to BotFather and hit the /newbot and follow the instruction

BotFather is a bot created by Telegram that allows you to create and manage your own bots. To connect to BotFather, search for "@botfather" in the Telegram app and click on the result to start a conversation.

In case you’d like a more detailed walkthrough, we suggest you take a look at this amazing tutorial prepared by Telegram.

Save the bot access token, it will be used in seting up telegram receiver in alertmanager.Also, join the bot to a channel/group to get the messages.

To get a info about the bot

The bot prefix has to be in the URL before the
bot.

BOT_ACCESS_TOKEN: it is provided by the "@BotFather".

curl https://api.telegram.org/bot<BOT_ACCESS_TOKEN>/getMe | jq .

{
    "ok": true,
  "result": {
      "id": <some_id>,
    "is_bot": true,
    "first_name": <"bot_name">,
    "username": <"bot_username">,
    "can_join_groups": true,
    "can_read_all_group_messages": false,
    "supports_inline_queries": false,
    "can_connect_to_business": false,
    "has_main_web_app": false
  }
}

CHAT_ID: After joining the bot to a channel/group , get the chat_id which you want to send message on behalf of the bot to that channel/group .

curl https://api.telegram.org/bot<BOT_ACCESS_TOKEN>/getUpdates | jq .

To Send a test message

curl -X POST "https://api.telegram.org/bot<BOT_ACCESS_TOKEN>/sendMessage" -d "chat_id=<CHAT_ID>" -d "text=Hello, World!"

Let's say the prometheus-stack is installed via helm or its operator in K8S cluster.
The release name will be used in many objects like AlertmanagerConfig, ServiceMonitor, PodMonitor, etc ... which the prometheus operator be able to select objects and add configs to the prometheus or alertmanager .

get release list in the prometheus namespace

helm list -n prometheus

NAME                NAMESPACE   REVISION    UPDATED                                     STATUS      CHART                           APP VERSION
prometheus-stack    prometheus  1       2025-03-08 23:34:47.082601345 +0330 +0330   deployed    kube-prometheus-stack-68.4.4    v0.79.2

`AlertmanagerConfig` to define the `Telegram` receiver

GROUP:      monitoring.coreos.com
KIND:       AlertmanagerConfig
VERSION:    v1alpha1

DESCRIPTION:
    AlertmanagerConfig configures the Prometheus Alertmanager,
    specifying how alerts should be grouped, inhibited and notified to external
    systems.

`telegramConfigs`

kubectl explain AlertmanagerConfig.spec.receivers.telegramConfigs

GROUP:      monitoring.coreos.com
KIND:       AlertmanagerConfig
VERSION:    v1alpha1

FIELD: telegramConfigs <[]Object>


DESCRIPTION:
    List of Telegram configurations.
    TelegramConfig configures notifications via Telegram.
    See
    https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

  botToken  <Object>
    Telegram bot token. It is mutually exclusive with `botTokenFile`.
    The secret needs to be in the same namespace as the AlertmanagerConfig
    object and accessible by the Prometheus Operator.

    Either `botToken` or `botTokenFile` is required.

  chatID    <integer> -required-
    The Telegram chat ID.

  parseMode <string>
  enum: MarkdownV2, Markdown, HTML
    Parse mode for telegram message

  sendResolved  <boolean>
    Whether to notify about resolved alerts.

The botToken has to be a Secret in the same namespace as the AlertmanagerConfig. lets say we name the secret file name telegram-bot-token-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: telegram-bot-token-secret # secret name will be referred in botToken in AlertmanagerConfig
  namespace: prometheus # same `namespace` as the `AlertmanagerConfig`.
type: Opaque
data:
  # bot-token will be used as key for botToken in AlermanagerConfig.
  bot-token: <Base64-encoded-BOT-TOKEN>
  # example
  # echo -n "<BOT_ACCESS_TOKEN>" | base64 (Token without bot prefix, exactly the token get from bot_father)

Create the Secret

kubectl apply -f telegram-bot-token-secret.yaml -n prometheus

`AlertmanagerConfig`

Create a new file named alertmanagerconfig-telegram.yaml with contents:

---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config-telegram
  namespace: prometheus
  labels:
    release: prometheus-stack # the release label should be the exact label of the prometheus helm release label
    alertmanagerConfig: alertmanager-telegram # custom label and also set this label to alertmanager in the prometheus-stack values.yaml
spec:
  route:
    groupBy: ["alertname","job","namespace"] # it could be other labels to be grouped by
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 1h
    receiver: "telegram"
    routes:
    - receiver: "telegram"
      matchers:
        - name: severity
          matchType: "=~"
          value: "warning|critical"
  receivers:
    - name: telegram
      telegramConfigs:
        - botToken:
            name: telegram-bot-token-secret # refer to the secret name has been created before
            key: bot-token # bot-token data in the secret
          chatID: <CHAT_ID>
          parseMode: 'MarkdownV2' # The template I provide is in Markdown Mode
          disableNotifications: false
          sendResolved: true

Create the AlertmanagerConfig

kubectl apply -f alertmanagerconfig-telegram.yaml -n prometheus

The `alertmanager` changes in the`prometheus-stack` `values.yaml`

Templates in alertmanager.templateFiles will be mounted in /etc/alertmanager/config/*.tmpl.

The telegram.tmpl will be mounted as /etc/alertmanager/config/telegram.tmpl, with defining the {{ define "telegram.default.message" }} there is no need to define message in the AlertmanagerConfig.spec.receivers.telegramConfigs. with that the default template for the telegram will be overwritten.

  alertmanager:
    templateFiles:
      telegram.tmpl: |-
        {{ define "telegram.default.message" }}
        {{- if eq .Status "firing" -}}
            {{- if eq .CommonLabels.severity "critical" -}}
                🔴 Alert: {{ .CommonLabels.alertname }}
            {{- else if eq .CommonLabels.severity "warning" -}}
                🟠 Alert: {{ .CommonLabels.alertname }}
            {{- else -}}
                ⚪️ Alert: {{ .CommonLabels.alertname }}
            {{- end }}
        Status: 🔥 FIRING
        Severity: {{ if eq .CommonLabels.severity "critical" }}🔴 {{ .CommonLabels.severity | title }}{{ else if eq .CommonLabels.severity "warning" }}🟠 {{ .CommonLabels.severity | title }}{{ else }}⚪️ {{ .CommonLabels.severity | title }}{{ end }}
        {{- else if eq .Status "resolved" -}}
            ⚪️ Alert: {{ .CommonLabels.alertname }}
        Status: ✅ RESOLVED
        Severity: {{ if eq .CommonLabels.severity "critical" }}🟢 {{ .CommonLabels.severity | title }}{{ else if eq .CommonLabels.severity "warning" }}🟢 {{ .CommonLabels.severity | title }}{{ else }}⚪️ {{ .CommonLabels.severity | title }}{{ end }}
        {{- end }}

        {{- range .Alerts -}}

        {{- if .Labels.job }}
        Job: `{{ .Labels.job }}`
        {{- end }}

        {{- if .Labels.namespace }}
        Namespace: `{{ .Labels.namespace }}`
        {{- end }}

        {{- if .Labels.instance }}
        Instance: `{{ .Labels.instance }}`
        {{- end }}

        {{- if .Annotations.runbook_url }}
        [RunbookURL]({{ .Annotations.runbook_url }})

        {{- end }}
        {{- end }}
        {{ end }}

Example Message

🟠 Alert: TargetDown
Status: 🔥 FIRING
Severity: 🟠 Warning
Job: kube-proxy
Namespace: kube-system
RunbookURL

alertmanagerConfigSelector

  alertmanager:
    alertmanagerSpec:
      ## AlertmanagerConfigs to be selected to merge and configure Alertmanager with.
      ##
      alertmanagerConfigSelector:
        matchLabels:
          alertmanagerConfig: alertmanager-telegram # it is the same label in the AlertmanagerConfig.

alertmanagerConfigMatcherStrategy with the None Type the AlertmanagerConfig will match the all alerts.

  kubectl explain Alertmanager.spec.alertmanagerConfigMatcherStrategy

  GROUP:      monitoring.coreos.com
  KIND:       Alertmanager
  VERSION:    v1

  FIELD: alertmanagerConfigMatcherStrategy <Object>


  DESCRIPTION:
      AlertmanagerConfigMatcherStrategy defines how AlertmanagerConfig objects
      process incoming alerts.

  FIELDS:
  type  <string>
  enum: OnNamespace, None
    AlertmanagerConfigMatcherStrategyType defines the strategy used by
    AlertmanagerConfig objects to match alerts in the routes and inhibition
    rules.

    The default value is `OnNamespace`.

  alertmanager:
    alertmanagerSpec:
      ## Defines the strategy used by AlertmanagerConfig objects to match alerts. eg:
      ##
      alertmanagerConfigMatcherStrategy:
      type: None
      ## Example with use OnNamespace strategy
      # alertmanagerConfigMatcherStrategy:
      #   type: OnNamespace

After all changes are made in the values.yaml, apply changes , in the case you installed the prometheus-stack with helm

helm upgrade prometheus-stack . -n prometheus --values values.yaml

Example alerts via `prometheus-rule`

Create a file named targets-prometheus-rules.yaml with contents:

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  namespace: prometheus
  name: targets
  labels:
    release: prometheus-stack # this label has to be the release name prometheus-stack
spec:
  groups:
    - name: targets
      rules:
        - alert: PrometheusTargetMissing
          expr: up == 0
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "Prometheus target missing \\(instance {{ $labels.instance }}\\)"
            description: "A Prometheus target has disappeared. An exporter might be crashed. \n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

        # node_exporter related metrics for alert
        - alert: HostOutOfMemory
          expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < .10)
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: Host out of memory (instance {{ $labels.instance }})
            description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

kubectl apply -f targets-prometheus-rules.yaml -n prometheus

Example Message in Telegram

⚪️ Alert: PrometheusTargetMissing
Status: ✅ RESOLVED
Severity: 🟢 Warning
Job: kube-proxy
Namespace: kube-system
Instance: 192.168.173.192:10249
Job: kube-proxy
Namespace: kube-system
Instance: 192.168.173.27:10249
Job: kube-proxy
Namespace: kube-system
Instance: 192.168.173.31:10249

🔴 Alert: KubeControllerManagerDown
Status: 🔥 FIRING
Severity: 🔴 Critical
RunbookURL

🔴 Alert: KubeSchedulerDown
Status: 🔥 FIRING
Severity: 🔴 Critical
RunbookURL

Read about the RunbookURL

Script Template in vim

Ali Mehraji — Sat, 08 Mar 2025 23:05:25 +0000

Script templates in Vim

Every time I wrote Bash or Python scripts in Vim, I had to add the Shebang #! /bin/bash or #! /usr/bin/python in the scripts manually.

So how can we automate this process with Vim to catch *.sh or *.py extensions and create a new file with our desired template, Let’s go for it :

First, you need to create a template file at $HOME/.vim/sh_template.temp with the contents you want:

#! /bin/bash

# =================================================================

# Author: < Your Name >
# Email: <Your Email>

# Script Name:
# Date Created:
# Last Modified:

# Description:
# < Description of what is this script for >

# Usage:
# < How to use this script , flag usage or ... >

# =================================================================

# ======================== Start Of Code ==========================

set -xe

Next, you need to configure autocmd in Vim by editing $HOME/.vimrc and adding this line:

" ========= Shell Script Template ========================
au bufnewfile *.sh 0r $HOME/.vim/sh_template.temp

Note:

Comment lines in vim scripting that are applied in .vimrc too, begin with " character .
au represents autocmd. autocmd docs
bufnewfile event for opening a file that doesn’t exist for editing.
*.sh cath all files with .sh extension. For Python scripts, you can replace it with *.py .

Now its time to create a shell script with the desired template:

vi Shell_Script.sh

Conclusion:

You can follow the same steps for every script or language you need.
Start using vim, you’ll love it ;).
There’s even a game to learn, give a try Vim Adventures.

Special $@ and $* Parameters in bash

Ali Mehraji — Sat, 08 Mar 2025 22:48:26 +0000

There is a say no difference between $@ and $* in Shell! What ?!!

Have you ever thought if there is no difference, why there are two of them? it could be one $@ or $* , Do You Agree?

So Let’s see What the difference is . Create a script and try it one by one

First One $@ :

#! /bin/bash

MAIN()
{
        echo "First Parameter is $1"
}

MAIN $@

Execute it like below and give the script some arguments:

$ ./Diff.sh "Jhon Smith" "Marry Ann"
First Parameter is Jhon

Second One $* :

#! /bin/bash

MAIN()
{
        echo "First Parameter is $1"
}

MAIN $*

The result will be :

$ ./Diff.sh "Jhon Smith" "Marry Ann"
First Parameter is Jhon

Far Now There is no difference, maybe those people were right!

Let us go further and try them with double quotes "$@" :

#! /bin/bash

MAIN()
{
        echo "First Parameter is $1"
}

MAIN "$@"

$ ./Diff.sh "Jhon Smith" "Marry Ann"
First Parameter is Jhon Smith

The second one in double quotes "$*" :

#! /bin/bash

MAIN()
{
        echo "First Parameter is $1"
}

MAIN "$*"

$ ./Diff.sh "Jhon Smith" "Marry Ann"
First Parameter is Jhon Smith Marry Ann

So be careful when you use double quotes and tell those people there is a difference, explain to them shell is sensitive to double quotes.

Forem: Ali Mehraji

Mattermost Receiver For Alertmanager in Prometheus-Stack in K8S

Overview

How

Steps

Docker Remote Context via SSH over Proxy

SSH Configuration

Docker Context

Resources

runcmd and package order in Cloud-Init

Resources

OOMKill in Kubernetes and Linux (Exit Code 137)

Why OOMKill Happens

Resources

Kubernetes Ephemeral Containers and kubectl debug Command

Ephemeral containers

Share Process

Lets get hands dirty

kubectl debug

kubectl debug node

Sidecar container via Docker

Resources

Python Env Variables in Dockerfile

PYTHONUNBUFFERED

PYTHONDONTWRITEBYTECODE

PIP_INDEX_URL

Notice

Resources

TIL Kubectl CSA/SSA (Client/Server Side Apply)

Field management

Apply

Change some fields in magaedFields

kubectl apply --server-side (rejected with Bad Request)

kubectl apply or kubectl edit (succeeded)

Conflict Example

ArgoCD

Read More

Environment

Resources

TIL `nsenter`

Annotations

Host Level Operation on K8S Node within a container

command breakdown

Additional notes from ChatGPT

Resources

Offline, Multistage Python Dockerfile

Docker build buildx syntax

Project Directory Tree

Multistage Dockerfile

base stage

build stage

runtime stage

Before And After The optimization

Before The Optimization

Final Dockerfile After Optimization

Notice

Updates

Resources

Alertmanager with Telegram for Prometheus-Stack in K8S

How to create a Telegram bot

AlertmanagerConfig to define the Telegram receiver

telegramConfigs

AlertmanagerConfig

The alertmanager changes in theprometheus-stack values.yaml

Example alerts via prometheus-rule

Example Message in Telegram

Script Template in vim

Script templates in Vim

Special $@ and $* Parameters in bash

Resources

`kubectl debug`

`kubectl debug node`

Change some fields in `magaedFields`

`kubectl apply --server-side` (rejected with Bad Request)

`kubectl apply` or `kubectl edit` (succeeded)

`command` breakdown

Docker build `buildx` `syntax`

`base` stage

`build` stage

`runtime` stage

`AlertmanagerConfig` to define the `Telegram` receiver

`telegramConfigs`

`AlertmanagerConfig`

The `alertmanager` changes in the`prometheus-stack` `values.yaml`

Example alerts via `prometheus-rule`