Forem: Satyam Kumar

Linux: Signalling custom events with kill

Satyam Kumar — Mon, 15 Sep 2025 04:25:09 +0000

Often we need a daemon process to respond to some external events. We might want to notify a running process of some external event and make that process do some work corresponding to that event or we may simply want to modify the behavior of a process corresponding to some external events.

Now one way to achieve this will be to use IPC (inter process communication) using sockets or pipes or dbus. All of these requires the running process to constantly poll the IPC elements used like socket files, pipe files etc or expose some methods on dbus. The daemon and the notifier process are tied by a thin bridge of socket files or pipe files or the dbus methods. Is there a native OS way to achieve this?

Real time signals

Linux already has the functionality that allows us to send a variety of signals to our running processes and we use them on a daily basis to terminate, kill or HUP a process. But linux also has an additional set of signals called real time signals that allows the daemons to respond to arbitrary events. On its own, these signals are meaningless to the kernel or the process. The definition and behavior for these signals are left for the process/daemons to define.

There are a total of 33 real time signals that the linux kernel supports, numbered between 32 to 64. Internally, glibc threading implementation might use 2 or 3 real time signals. Due to this, the actual range available for the use might not begin with 32. Hence we should refrain from using these hard-coded numbers for the signals and instead the recommended way to use these signals is to use notation SIGRTMIN+n or SIGRTMAX-n, where n starts from 1. We should ensure that the value of these signals doesn't go outside SIGRTMIN - SIGRTMAX. Here SIGRTMIN and SIGRTMAX are defined by the kernel depending upon the glibc threading implementation and we don't need to worry about their exact values.

We can actually list all the available signals in bash for our system:

$ bash -c "kill -l"
1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

All the signals which starts with SIGRTMIN or SIGRTMAX are the real time signals and their corresponding numeric value is shown before them.

Using Real Time Signals in apps

Here I will demo the use of these signals using a simple bash script but the same approch can be taken with any language. First we need to decide which real time signals our app will be using and what function those signals would correspond to. Then we need to implement below two steps in our program for each of the real time signals we have decided to use:

depending upon the signal function we have decided, define separate handler function for each signal
assign these handler functions to their respective real time signals we have decided.

Sample program

#!/usr/bin/env bash

#### Section 1: define all handler functions ######################

# handler function to send email SIGRTMIN+1
send_email_notification() {
  echo "Sending email notification"
  # code for ending email goes here
}

# handler function to enable execution with set -x
enable_debug_logs() {
  set -x
  echo "enabling debug logs"
}

# handler function to disable -x
disable_debug_logs() {
  set +x
  echo "disabling debug logs"
}
######################################################33333

#### Section 2: adding handler to signals #########################

## Now we will lay down our traps to handle these signals
## In bash 'trap' is how we define the signal handler
trap send_email_notification 'SIGRTMIN+1'
trap enable_debug_logs 'SIGRTMIN+2'
trap disable_debug_logs 'SIGRTMIN+3'
########################################################

### the main loop which will do some work
while true; do
  echo "Doing some work"
  sleep 10
done

This is very simple bash program that I have written to demo the concept and the possible use case. Here corresponding to the steps mentioned previously, I have divided the program into two sections. The real time signals and their function that our program will be using are as follows:

SIGRTMIN+1 : to send email notification on demand
SIGRTMIN+2: to enable printing of debug output
SIGRTMIN+3: to disable printing of debug output

Its not mandatory to go serially or pick these real time signals from start. You can pick any real time signal in the range allowed by the OS . Let’s now look at what the two sections defined in the program do.

Section 1: defining the handlers

# handler function to send email SIGRTMIN+1
send_email_notification() {
  echo "Sending email notification"
  # code for ending email goes here
}

# handler function to enable execution with set -x
enable_debug_logs() {
  set -x
  echo "enabling debug logs"
}

# handler function to disable -x
disable_debug_logs() {
  set +x
  echo "disabling debug logs"
}

Here we are defining three handler functions:

send email notification handler (send_email_notification): This function is supposed to send some email notification to some defined set of email IDs when the signal is received.
enable debug logs (enable_debug_logs): This function will run set -x which will cause the bash script to print the lines it is executing
disable debug logs (disable_debug_logs): This function will run set +x which will cause the bash script to stop printing the lines it is executing

Section 2: assigning the handlers to the signals

trap send_email_notification 'SIGRTMIN+1'
trap enable_debug_logs 'SIGRTMIN+2'
trap disable_debug_logs 'SIGRTMIN+3'

This is where the actual magic happens. In bash we assign the handlers to the signal using trap command. The syntax is quite simple:

trap <handler function name> <signal name>

In the program above using trap command, we have assigned each of our handler function to their respective real time signals we had decided to use.

Seeing signals in action

Time to see this in action now. Lets run our bash script in one terminal and open a new terminal to run our command to send the real time signals. Using the ps command, note the PID of our bash script (in my case it is 692560). Now to send the signal to our program, we will be using the bash kill command. The syntax is like this:

kill -s <signal name> <pid>

Let's now see each of the three scenarios in action.

Note: below demo uses bash kill command. If you are using any other shell then please check the respective documentation on how to send the real time signals in that shell.

Experiment 1: tell our bash process to send email notification

Let's run the below command in the second terminal:

kill -s SIGRTMIN+1 692560

Now in the first terminal, after few seconds (utmost 10), you should see the the following sequence of lines (the delay will happen because sleep command will be running):

Doing some work
Doing some work
Sending email notification
Doing some work

This shows that our send_email_notification handler was executed successfully.

Experiment 2: enable printing of debug output

Run the below command in the second terminal now:

kill -s SIGRTMIN+2 692560

Observe the output in the first terminal for next 20 secs. You will see output like below. The program will start printing the lines it is executing.

Doing some work
++ echo 'enabling debug logs'
enabling debug logs
+ true
+ echo 'Doing some work'
Doing some work
+ sleep 10
+ true
+ echo 'Doing some work'
Doing some work
+ sleep 10

Thus we can see that our enable_debug_logs handler was also executed successfully.

Experiment 3: disable printing of debug logs

Now final experiment. Lets disable the debug logs printing.

 kill -s SIGRTMIN+3 692560

Again lets observe the output on the first terminal for next 20 secs. You will see output similar to below. The program will stop printing the lines it is executing and go back to normal output sequence.

Doing some work
+ sleep 10
++ disable_debug_logs
++ set +x
disabling debug logs
Doing some work
Doing some work

This show that our disable_debug_logs was also successfully executed.

Conclusion

Thus we saw how we can easily implement these real time signal handling in our program and notify our running process using native linux signalling. This is the simplest way of notifying a running process without requiring expensive socket, pipe or dbus setup. Of course this doesn't allows you to send some data along if your process requires that but it is quite good for scenarios where the process just needs to be notified of some event to do some work.

Bonus

As a bonus, I leave you with a simple python program that implements the above idea using python. Try to run it and send the same real time signals to it as above and see the output.

#!/usr/bin/env python

import signal
import logging
import sys
import time
import types

# setup logger
logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)  # Set the log level
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
stdout_handler.setFormatter(formatter)
logger.addHandler(stdout_handler)

def send_email_notification(signum: int, frame: types.FrameType | None) -> None:
    logger.info("Sending email notification")

def enable_debug_logs(signum: int, frame: types.FrameType | None) -> None:
    stdout_handler.setLevel(logging.DEBUG)
    logger.info("enabling debug log")

def disable_debug_log(signum: int, frame: types.FrameType | None) -> None:
    stdout_handler.setLevel(logging.INFO)
    logger.info("disabling debug log")

if __name__ == '__main__':
    signal.signal(signal.SIGRTMIN+1, send_email_notification)
    signal.signal(signal.SIGRTMIN+2, enable_debug_logs)
    signal.signal(signal.SIGRTMIN+3, disable_debug_log)

    while True:
        time.sleep(5)
        logger.info("Doing some work")
        logger.debug("this is debug log for that work")

References

For further reading:

Signal man page

Analyse and Optimise your container image size

Satyam Kumar — Wed, 27 Aug 2025 15:41:25 +0000

Whether you are a software developer, a devops engineer or a linux sys admin, everyone has seen how ubiquitous the containers have become. There is no denying the fact that they are quite convenient to run the applications. You have your application ready, all you need is write a neat little Dockerfile and run the docker build command and there you have your application containerized. Declarative nature of writing a Dockerfile is so easy that we often dont even think about optmising it and the end result is ridiculously larger image size. Although there are a ton of ways to analyse your dockerfile and optimise it but in this post today I would like to focus on one specific scenario that I recently faced in my organisation.

An example setup

As this blog post that I am writing is outside the organisation I am working with so I will try to imagine a situation which is close to the scenario I faced within my org and try to build up a solution from there.

Lets say we have the below dockerfile with us:

FROM ubuntu:latest
RUN apt update && apt install make
RUN mkdir /workdir
WORKDIR /workdir
COPY Makefile  .
RUN make
RUN chmod 755 mybinary
RUN mv mybinary /usr/bin/mybinary

What this will do is compile an application of size 2GB and then install it to the /usr/bin. Lets trigger the build of this image and see what the image size is after the end.

satyam@wakanda ~/demo $ docker build -t mydemo:latest .
[+] Building 9.1s (13/13) FINISHED                                                         docker:default
 => [internal] load build definition from Dockerfile                                                 0.0s
 => => transferring dockerfile: 215B                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                     0.0s
 => [internal] load .dockerignore                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => [1/8] FROM docker.io/library/ubuntu:latest                                                       0.0s
 => [internal] load build context                                                                    0.0s
 => => transferring context: 30B                                                                     0.0s
 => CACHED [2/8] RUN apt update && apt install make                                                  0.0s
 => CACHED [3/8] RUN mkdir /workdir                                                                  0.0s
 => CACHED [4/8] WORKDIR /workdir                                                                    0.0s
 => CACHED [5/8] COPY Makefile  .                                                                    0.0s
 => CACHED [6/8] RUN make                                                                            0.0s
 => CACHED [7/8] RUN chmod 755 mybinary                                                              0.0s
 => [8/8] RUN mv mybinary /usr/bin/mybinary                                                          2.0s
 => exporting to image                                                                               7.0s
 => => exporting layers                                                                              7.0s
 => => writing image sha256:0047f0b0f1bb5e2ffbf178f5f0cddbad64aaa71d37d28eb0c6c0aecf1def9963         0.0s
 => => naming to docker.io/library/mydemo:latest                                                     0.0s
satyam@wakanda ~/demo $ docker image ls mydemo:latest
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
mydemo       latest    0047f0b0f1bb   4 minutes ago   6.57GB

What!!!

How did the image end up being 6.5 GB when the application is just 2GB !!

Where it all went wrong ?

Now surely if my container image is ending up being 3 times my application size then I am not doing a very good job at containerizing it. Large containers take large disk space and are time taking to build and export.

So how can we analyse how our image size ballooned to this size?

We will use an open source tool called dive.

Let's pass our image tag as input to this command

satyam@wakanda ~demo $ dive mydemo:latest
Image Source: docker://mydemo:latest
Extracting image from docker-engine... (this can take a while for large images)
Analyzing image...

This will take some time to run and then you will be presented with a screen containing very detailed view of each layer of the image and also the filesystem view of the last layer. We will focus on the layer view. This is what you are expected to get:

In the output we can clearly see how much size each of our dockerfile commands have contributed to the overall image size. In Dockerfile, each dockerfile command you write corresponds to one layer of container image and each new layer is added upon the previous layer. Thus at the end the container image is the sum of all the layers that was created by each of the dockerfile commands.

In the image above, we can clearly see that the last three layers are the ones adding to the bulk of image size. So lets start by analysing each of them one by one.

1. The third last layer

   2.1 GB  RUN /bin/sh -c make # buildkit

In this output we can see that our RUN make command created our application of 2.1GB in size which is ok because we know that the app was supposed to be 2GB. So far so good !!

2. The second last layer - The first mistake

   2.1 GB  RUN /bin/sh -c chmod 755 mybinary # buildkit

Here we ran chmod 755 mybinary to modify the permission of our binary file. Now on its own this looks like a very innocent command so why is this creating an image layer equivalent to the previous layer in size?

The answer lies in the docker's copy on write approach of creating the image layer. When the container is being built, each of the layers that are created are immutable. They can never be changed by the commands of the next layer. So when we do RUN chmod ..., this will copy the binary onto the new layer and there it will modify its permissions. Hence this simple chmod ends up contributing a size equal to its previous layer. Now this make up total size of ~4GB. What about the rest 2GB?

3. The final layer - The second mistake

   2.1 GB  RUN /bin/sh -c mv mybinary /usr/bin/mybinary # buildkit

Here we ran mv mybinary /usr/bin/mybinary. Now again a very innocent command. Technically we are just moving files so this should not be adding to the size. But remember our observation for the previous layer. Every previously built layer is immutable. So even though we are using an mv commad, docker cannot perform this operation on the previous layer. What it will end up doing is again copy the file from previous layer to the new layer and then perform the mv operation here. Thus it will again end up adding 2GB.

Hence we have now 6Gb of the image...

How could this have been avoided ?

We can easily fix this problem by using docker's multi stage build. What this means is that we build out the application and fix its permission in one stage and copy the final application onto the next stage. The layers of one stage are not added to the layers of the other stage.

Here is our fixed dockerfile.

# Stage 1
FROM ubuntu:latest
RUN apt update && apt install make
RUN mkdir /workdir
WORKDIR /workdir
COPY Makefile  .
RUN make
RUN chmod 755 mybinary

# Stage 2
FROM ubuntu:latest
COPY  --from=0 /workdir/mybinary /usr/bin/mybinary

Each stage in a multi stage build is identified by a seperate FROM command. In the first stage, we build out the application and fix the permissions of the application and then in the second stage, we copy the final application from the first stage to the required destination in the final image.

Lets build the image and see the image size now.

satyam@wakanda ~demo $ docker build -t mydemo:fixed .
[+] Building 10.6s (13/13) FINISHED                                                                                                                                                                    docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 251B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => CACHED [stage-0 1/7] FROM docker.io/library/ubuntu:latest                                                                                                                                                    0.0s
 => [internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 30B                                                                                                                                                                                 0.0s
 => CACHED [stage-0 2/7] RUN apt update && apt install make                                                                                                                                                      0.0s
 => CACHED [stage-0 3/7] RUN mkdir /workdir                                                                                                                                                                      0.0s
 => CACHED [stage-0 4/7] WORKDIR /workdir                                                                                                                                                                        0.0s
 => CACHED [stage-0 5/7] COPY Makefile  .                                                                                                                                                                        0.0s
 => CACHED [stage-0 6/7] RUN make                                                                                                                                                                                0.0s
 => CACHED [stage-0 7/7] RUN chmod 755 mybinary                                                                                                                                                                  0.0s
 => [stage-1 2/2] COPY  --from=0 /workdir/mybinary /usr/bin/mybinary                                                                                                                                             2.1s
 => exporting to image                                                                                                                                                                                           7.0s
 => => exporting layers                                                                                                                                                                                          7.0s
 => => writing image sha256:ec007c5a4c19fd0481b7f41fff27024a3ac25ba0e9e41f25a7fa70e2cae98d46                                                                                                                     0.0s
 => => naming to docker.io/library/mydemo:fixed
satyam@wakanda ~demo $ docker image ls mydemo:fixed
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
mydemo       fixed     ec007c5a4c19   16 seconds ago   2.23GB

Behold!! we reduced our image size from 6GB to ~2GB. We can also visualize it with dive tool

Just two layers!!! Remember I mentioned that in multi stage builds, the layers from previous stage are not added to the final stage. Since in the second stage all we did was copy the application from first stage so we only have two layers and both of them are of expected size.

So this is how we used dive tool to analyse our image layers and used multi stage builds to reduce the image size.

Let me know your thoughts on this in the comments below and how you approached and optimised the image size in your projects.

For more reading:
Multi Stage Builds
Dive tool