Forem: Shubham Kumar

Working on mulitple braches at once using git worktree

Shubham Kumar — Sun, 02 Mar 2025 14:50:00 +0000

Working on multiple branches at once

Git worktree enables you to have multiple working directories linked to the same repository. Each directory can have its own branch checked out. And you can edit the code for each feature without changing or stashing the changes you did on each branch.

Worktree creation

The following command will create a new directory, debug with branch1 checked out.

You can browse the ../debug directory and change the code as per your development for branch1.

git worktree add ../debug branch1

pwd
echo somenewfeature on branch 1 > somenewfeature
cat somenewfeature

/home/cold/Projects/Personal/gitworktreedemo/debug
somenewfeature on branch 1

In the meanwhile you can keep editing the root directory with the main branch.

pwd
echo somenewfeature on main 1 > somenewfeatureonmain
cat somenewfeatureonmain

/home/cold/Projects/Personal/gitworktreedemo/main
somenewfeature on main 1

You can run the git commands going inside each directory.

pwd
git add .
git status

/home/cold/Projects/Personal/gitworktreedemo/main
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    new file: somenewfeatureonmain


pwd
git add .
git status

/home/cold/Projects/Personal/gitworktreedemo/debug
On branch branch1
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    new file: somenewfeature

List existing worktrees

You can list your worktrees from the main as well as the debug directories.

git worktree list

/home/cold/Projects/Personal/gitworktreedemo/main e0ccbca [main]
/home/cold/Projects/Personal/gitworktreedemo/debug 42ad28a [branch1]

Remove a worktree

When you are finally done with the feature branch, you can delete the worktree using remove command.

git worktree remove ../debug
ls ..
git worktree list

main
/home/cold/Projects/Personal/gitworktreedemo/main e0ccbca [main]

Any commit you did on the feature branch will automatically be reflected in the main codebase.

git checkout branch1
git log --oneline

A   somenewfeatureonmain
82cad2d done
42ad28a A commit on branch1
e0ccbca A commit on main

Advice on working with worktrees

As the new directory will not be ignored by default, it is best to create your worktree directories out of git root

The way I create my worktrees is I create a new directory with project name. Then I clone my actual repository inside this directory and keep my worktrees on the same level as project.

For example, my main project is cloned inside main directory and I created debug directory in the same level as main.

tree .

.
├── debug
│   └── somenewfeature
└── main
    └── somenewfeatureonmain

2 directories, 2 files

Displaying full content in rss.xml in Hugo

Shubham Kumar — Sun, 02 Mar 2025 13:50:00 +0000

Create a new file at layouts/_default/rss.xml. You can define your rss.xml as per your liking.

By default rss.xml only displays the summary of your articles. But I want to display the whole content. This helps me in syncing my posts with my dev.to site.

You can find the Hugo’s default rss.xml at Hugo’s Github repo. Let’s copy the contents to our rss.xml file that we just created.

Then change the description to show content instead of summary as follows.

-<description>{{ .Summary | transform.XMLEscape | safeHTML }}</description>
+<description>{{ .Content | transform.XMLEscape | safeHTML }}</description>

Before the changes there were only 504 characters in the characters.

curl localhost:1313/index.xml | grep "description" | head -n 3 | tail -n 1 | wc -c

: 504

After the changes the number of characters increased significantly as the whole blog is being rendered in the description.

curl localhost:1313/index.xml | grep "description" | head -n 3 | tail -n 1 | wc -c

: 14582

Partials in Hugo

Shubham Kumar — Tue, 25 Feb 2025 14:39:00 +0000

Partials and code reusability

Hugo provides a nice mechanism to separate your components and stitch them at different places. This means you can design your components like nav-bar or menu bar and stitch them in all of your layouts. Earlier, we wrote some code to load tailwind in our index page. But we do not want to write this boilerplate code for all our pages. Instead we should have defined the tailwind loader code in a component and call it from all the layouts. Actually, we should have called the boilerplate code from baseof.html from which all other layouts are inherited.

baseof.html

When we created the theme, a special file was created for us, baseof.html. Every other layout inherits baseof.html.

If you look at the contents of baseof.html which was generated for use. You will find 3 partial sections, head, header and footer. These are the default standards but you will go on creating partials as your code base becomes more complex.

The main block denoted by {{ block "main" . }}, is where your content from other pages will be inserted. You can define your layout as main in other layout files and that block will be inserted here.

<!DOCTYPE html>
<html>
    {{- partial "head.html" . -}}
    <body>
        {{- partial "header.html" . -}}
        <div id="content">
        {{- block "main" . }}{{- end }}
        </div>
        {{- partial "footer.html" . -}}
    </body>
</html>

You can ignore the dashed infront and at last of the decalrations. These are the guide for Hugo to trim spaces infront and back.

head.html

After html tag we have a partial head.html instead of a head tag. Hugo is smart enough to generate head attribute from the partial. Looking closely, it just process your partials and replaces the partial block with it.

We could have defined out tailwind loading code here as follows. Now we got the support for tailwind everywhere in our site. You can also define some meta tags or other libraries here.

<head>
  {{ with resources.Get "css/main.css" }}
  {{ with . | css.TailwindCSS }}
      <link rel="stylesheet" href="{{ .RelPermalink }}">
  {{ end }}
  {{ end }}
</head>

Defining custom partials

Defining partials is as easy as it can be. You just need to create a file under partials directory. And write whatever processing you want to do. And include it in you main layout using {{ partial "your-file.html" . }}. You can pass any data along with your partial definition. In the above partial definition, we are passing the current context to our partial.

Make CSS styling a partial

Let’s create a new partial by creating a file called css.html. This will contain the logic to process and import our tailwind libraray.

The contents of css.html will be as follows.

{{ with resources.Get "css/main.css" }}
{{ with . | css.TailwindCSS }}
    <link rel="stylesheet" href="{{ .RelPermalink }}">
{{ end }}
{{ end }}

And our head block will include this as a partial as follows. The below code goes inside baseof.html.

<head>
  {{ partial "css.html" . }}
</head>

Modify the pages to use baseof

You will need to define the main block in all your layouts. This will help Hugo to link the contents of layouts page and combine it with baseof to generate the rendered page.

Below is how, I modified my index.html page.

{{ define "main" }}
  <p class="text-red-500">This paragraph is red.</p>
  {{ .Content }}
{{ end }}

curl localhost:1313 | grep "main.css"

: <link rel="stylesheet" href="/css/main.css">

This looks as expected. We can see out main.css linked to our homepage.

Now let’s modify the list.html layout and define main here as well.

{{ define "main" }}
  <ul>
    {{ range .Pages }}
    <li>{{ $.Site.Title }} - {{ $.Title }} - {{ .Title }}</li>
    {{ end }}
  </ul>
{{ end }}

curl localhost:1313/posts/ | grep "main.css"

: <link rel="stylesheet" href="/css/main.css">

Awesome, we can now use tailwind from anywhere in our website.

Summary

In this post we explored partials which is basically a way of reusing the code in Hugo. We learned about baseof.html which is inherited by all the layouts. We combined our tailwind learning with partials and created a css.html partial which can now be used from anywhere.

Tailwind support in Hugo

Shubham Kumar — Sun, 23 Feb 2025 16:57:00 +0000

Expectations from this post

With the version 0.128.0, Hugo started supporting Tailwind internally. In this post, let’s configure Tailwind and display some text in red.

Tailwind and other supporting apps

We will install the below 4 packages.

tailwindcss - This is the core Tailwind library and is required for using tailwind
tailwind/cli - This is a CLI tool for Tailwind that helps in building tailwind styles
postcss - This is a library to process CSS with JavaScript plugins.
autoprefixer - This is a postcss library that helps in automatically generating the vendor prefixes like -webkit- and -moz-.

We will install these packages as dev as they play no role after building the site.

npm install --save-dev tailwindcss @tailwindcss/cli postcss autoprefixer

PostCSS config

You need to specify the plugins for postcss. This can be done by creating a file, postcss.config.js. You can add the plugins you want to use here as follows.

For now, we will add tailwindcss and autoprefixer as the plugins.

  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
};

Tailwind config

We can create a more advanced config for Tailwind later.

For now, let’s just create a file, tailwind.config without any contents in the theme’s root directory.

Loading Tailwind on the index page

Let’s first create a resource main.css in the assets directory of the theme. Here we will import Tailwind and then we will load this resource in our index file using some Hugo magic.

main.css

In the theme’s root directory, let’s create a directory assets/css and add a file main.css here.

The content of the file will just load the Tailwind library.

@import "tailwindcss";

index.html

Here, we just want to add main.css as our stylesheet.

This is done by using link tag.

First we want to load the resource then we want to process it using tailwind and add it to our page.

We can use Hugo’s with function to load the file and again using with, we can process the obtained file using tailwind. The below snippet will do the trick.

{{ with resources.Get "css/main.css" }} will try to get the main.css file. Hugo will look into the assets direcetry for css/main.css.{{ with . }} will only be processed if the file was found. And if found we can pipe the contents to be processed by css.TailwindCSS provided by Hugo.

{{ with resources.Get "css/main.css" }}
    {{ with . | css.TailwindCSS }}
        <link rel="stylesheet" href="{{ .RelPermalink }}">
    {{ end }}
{{ end }}

This would be enough to start using Tailwind functionalities.

To test if it is working of not, we can add a simple text and use some tailwind features to make it red.

Inside the index.html, body let’s add the following. Adding class as text-red-500, should turn the content of <p> to red.

<p class="text-red-500">This paragraph is red.</p>

Taking a peek

Let’s first curl the homepage and verify of link was created successfully.

curl localhost:1313

<html>
  <head>
    <meta name="generator" content="Hugo 0.144.2"><script src="/livereload.js?mindelay=10&amp;v=2&amp;port=1313&amp;path=livereload" data-no-instant defer></script>
    <title>
      Shubham&#39;s corner
    </title>
        <link rel="stylesheet" href="/css/main.css">
  </head>
  <body>
    <p class="text-red-500">This paragraph is red.</p>
  </body>
</html>

The file main.css here is the processed main.css and will look very different than what we have written.

This is good, next let’s see the actual site.

The rendered text is red which means we were finally able to load out Tailwind.

Summary

In this post, we saw how to enable the support of Tailwind CSS in out Hugo site. We added some text and turned it to red using Tailwind classes. Next, we will explore code re-usability in Hugo which is where partials come to play.

Upgrading Hugo to latest version on Debian based systems

Shubham Kumar — Sun, 23 Feb 2025 15:59:00 +0000

There is a bad news for Ubuntu (or any Debian based OS) user if you have installed Hugo using apt. The official apt repository is not being updated continuously.

Today, I wanted to experiment with the Tailwind support introduced in Hugo 0.128.0. But the apt package manager does not has this version. The latest version is 0.92.2 as of now.

apt list -a hugo

: Listing...
: hugo/jammy-updates,jammy-security,now 0.92.2-1ubuntu0.1 amd64 [installed]
: hugo/jammy 0.92.2-1 amd64

In such scenarios, you can install the .deb file directly from the published releases. The disadvantage of this method is, you will lose the ability to upgrade the version automatically. In the future, you will be restricted to download the deb file and install the updated version.

Anyways, here are the steps to do so.

First look for the latest (or the version you want) to install at the GoHugoIO release page - https://github.com/gohugoio/hugo/releases

At the time of writing this post, the latest version is 0.144.2. Any my current version is 0.92.2.

hugo version

: hugo v0.92.2+extended linux/amd64 BuildDate=2023-01-31T11:11:57Z VendorInfo=ubuntu:0.92.2-1ubuntu0.1

From the Downloads directory, I can run the below command to download the deb file that I want to install. You can use any directory to download this file. Generally people use /tmp, but I like all my downloads in my Downloads directory.

wget https://github.com/gohugoio/hugo/releases/download/v0.144.2/hugo_0.144.2_linux-amd64.deb

Next you can install the deb file as follows.

ls | grep hugo

: hugo_0.144.2_linux-amd64.deb

dpkg -i hugo_0.144.2_linux-amd64.deb

This should upgrade your installation to latest.

hugo version

: hugo v0.144.2-098c68fd18f48031a7145bedab30cbaede48858f linux/amd64 BuildDate=2025-02-19T12:17:04Z VendorInfo=gohugoio

Hope this will help you upgrading Hugo to latest version on Debian based systems.

Object pool design pattern in Java

Shubham Kumar — Sat, 07 Dec 2024 17:34:00 +0000

What is it?

The object pool design pattern exposes a manager to manage a pool of reusable objects. The idea is to keep a know number of reusable objects (with a hard limit to initialize some more lazily). Whenever someone need the object from the pool, it will ask the pool manager. If there are free objects, the manager will engage one for your. If there aren’t any free objects but the hard limit is not breached, then the manager will initialize a new object and provide you. Else if the hard limit is breached then you will return empty handed.

Why is this?

This is used when we have an resource object which takes some time to initialize. And once initialized, it can be reused over and over again without a performance hit.

Basically, creation is expensive and hence we want to reuse already created instances. Generally, we put a soft limit on the number of resources initially initialized. And we want to create more resources lazily if required. To prevent a huge number of resources from being created, we also put a hard limit.

Example

At GreyOrange, we use something called as IDC files. These are huge binary files (sometimes 100-200 GBs). They provide the time it takes to travel b/w 2 coordinates. We created an IDC Manager to parse these files and provide us the required information. The initialization takes a huge amount of time (sometimes 10s). Once initialized, it takes less than 1ms to provide the information.

Right now, we are good with just once instance of this manager, so it is a singleton class with one buffer linked to an IDC file. But if the demands for parallel calls increases, we might want to implement the manager as a Object Pool.

How to implement this?

There are 3 requirements.

An Object Pool Manager
The initial number of objects - m
The maximum number of objects - n

The Object Pool Manager will be a singleton class. We cannot allow multiple Object Pool Manager objects as they will create max, n objects each.

We will create 2 lists, availableResources and enagaedResources. Initially, we will populate the availableResources with m new resource objects.

Each getter call will check the availableResources list for available objects. If the objects are available then it will move the last object to engagedObjects. If the objects are not available then there are 2 choices. Check the hard limit, if not reached then create more objects and add to availableResources. Else return null.

A pseudo code for the manager is as follows.

class PoolManager {
    private static PoolManager instance;
    private List<Object> availableResources;
    private List<Object> engagedResources;
    private Integer initialLimit;
    private Integer hardLimit;

    private PoolManager() {
        // Get these properties from already defined config
        // Assume this is defined as per standard or equivalent configurations
        initialLimit = Properties.getInstance().getIntegerValue("POOL_INITIAL_LIMIT");
        hardLimit = Properties.getInstance().getIntegerValue("POOL_HARD_LIMIT");

        availableResources = new ArrayList<>();
        engagedResources = new ArrayList<>();

        // Initialize the initial number of resources in the pool
        for (int i=0; i<initialLimit; i++) {
            availableResources.add(new ResourceObject());
        }
    }

    public static PoolManager getInstance() {
        if (instance == null) {
            synchronized(PoolManager.class) {
                if (instance == null) {
                    instance = new PoolManager();
                }
            }
        }

        return instance;
    }

    public Object getObject() {
        if (!availableResources.isEmpty()) {
            // A sync is required as 2 thread may want to get a free object at the same time
            synchronized(availableResources) {
                Object freeObject = availableResources.remove(availableResources.size()-1);
                engagedResources.add(freeObject);
                return freeObject;
            }
        }
        else if (engagedResources.size() < hardLimit) {
            Object freeObject = new ResourceObject();
            availableResources.add(freeObject);
            return getObject();
        }
        else {
            return null;
        }
    }

    public void releaseObject(Object engagedObject) {
        if (engagedObject != null) {
            try {
                synchronized(engagedResources) {
                    Object freeObject = engagedResources.remove(engagedObject);
                    availableResources.add(freeObject);
                }
            } catch (Exception e){}
        }
    }
}

Downloading a single file from 2 independent apps

Shubham Kumar — Sat, 31 Aug 2024 18:41:00 +0000

Understanding the problem

Let’s say you have a very large log file. And you want to create an app that can analyze this file and generate insights. Also, let’s say you want to create an another app that can simulate the work by reading the logs one-by-one.

Both these apps are dependent on the same log file. Now, there are 2 scenarios.

App1 starts, downloads the file and then App2 starts.
App1 starts, downloading the file and App2 starts while the download is incomlete.

The first scenario is easy to deal with. We can check the md5sum of the local file and the file on the server. If they match, nothing to worrry about. If they don’t then we can have a complex logic to determine the life of the old log file and decide accordingly.

The second scenario is conflicting one and this we can solve in code. The second scenario can also happen when the same app is ran twice simultaneously. Both the instances will start downloading the same file and this will create a havoc.

Solution

The idea is to have an identifier that an app has already started the download and is still downloading the resouce. If the first app has started the download, then wait for the first app to complete the download and then only start the application.

For accomplishing this, we generally use file locking mechanism.

Download with file locking

The process is modified to first create a lock file with extension .lock. This lock file signifies that a download is already in progress. If this lock file exists then wait for the download to complete by the second app. The lock file will have processid_threadid as identifier. This is useful in checking the race condition that can happen while writing the file.

  public static void downloadFileWithLock(String filePath) {
    File lockFile = new File(filePath + ".lock");
    // Check if the file is being downloaded by another app
    // If it is being downloaded by an another app then wait for the download to finish
    // Else proceed with the download
    if (lockFile.exists()) {
      waitForDownloadToFinish(lockFile);
    } else {
      int processID = (int) ProcessHandle.current().pid();
      String identifier = thread + "_" + processID;
      String contents = String.valueOf(identifier);
      writeToFile(lockFile, contents);

      // May be due to race condition, the file is already downloaded by another app
      // Check if this process started the download
      String savedIdentifier = readFromFile(lockFile);

      if (identifier.equals(savedIdentifier)) {
        // Download the file
        System.out.println(thread + " - Downloading file...");
        File downloadFile = new File(filePath);
        try {
          RandomAccessFile randomAccessFile = new RandomAccessFile(downloadFile, "rw");
          randomAccessFile.write("Very important works".getBytes());
          Thread.sleep(5000);
        }
        catch (IOException e) {}
        catch (InterruptedException e) {}
        System.out.println(thread + " - File downloaded successflly.");
      } else {
        waitForDownloadToFinish(lockFile);
      }

      if (lockFile.exists()) {
        lockFile.delete();
      }

    }
  }

The Utilities method - waitFoDownloadToFinish, readFromFile and writeToFile are as follows.

  private static void writeToFile(File file, String contents) {
    try {
      Files.write(file.toPath(), contents.getBytes());
    } catch (IOException e) {
      e.printStackTrace();
    }
  }

  private static String readFromFile(File file) {
    try {
      return new String(Files.readAllBytes(file.toPath()));
    } catch (IOException e) {
      e.printStackTrace();
    }
    return null;
  }

  private static void waitForDownloadToFinish(File lockFile) {
    System.out.println(thread + " - File is already being downloaded by another app. Wait for it to finish.");
    while (lockFile.exists()) {
      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
    }
    System.out.println(thread + " - File download completed.");
  }

Now you can create 2 new apps that will call this method and we will run the apps simultaneously.

public class App1 {
  public static void main(String[] args) {
    String filePath = "downloaded_file.txt";
    FileDownloadUtil.downloadFileWithLock(filePath);
    System.out.println(Thread.currentThread().getName() + " - App1 starting operation...");
  }
}

public class App2 {
  public static void main(String[] args) {
    String filePath = "downloaded_file.txt";
    FileDownloadUtil.downloadFileWithLock(filePath);
    System.out.println(Thread.currentThread().getName() + " - App2 starting operation...");
  }
}

Outputs

# For App1
main - Downloading file...
main - File downloaded successfully.
main - App1 starting operation...

# For App2
main - File is already being downloaded by another app. Wait for it to finish.
main - File download completed.
main - App2 starting operation...

App1 started downloading the file and thus App2 waited for the download to complete. After the download completes, both the apps resumed its operations.

Conclusion and improvements

This is just a basic code that lays the foundation of file locking mechanism for downloading a file simultaneously by multiple apps. This code is not a production ready code. A more complete solution should handle scenarios like downloads in chunks, resume functionality with unexpected shutdowns and other edge cases.

Reflection API in Java

Shubham Kumar — Fri, 21 Jun 2024 18:33:00 +0000

Where is this used?

This is used to analyze/modify the behaviour of a class at runtime. Using this, you can view or change the private/public fields at wish (without exposing any getter/setter). Personally, I have used this in one of our projects at GreyOrange to write unit test cases. Using this in main code is a big no-no as it exposed you critical fields to the world.

Main Class

Let’s create a main class for which we will write some test cases. But we want to test some private fields for which we don’t have a direct getter. The idea is to use reflection api to access such fields and fetch their current value or modify them if required.

Here is a Duck class which has 3 fields of which 1 is static. Each time a duck class is created count which is the static field is increased by one. Each duck has an associated name and age.

public class Duck {
    private String name;
    private int age;
    private static int count = 0;

    public Duck(String name, int age) {
        this.name = name;
        this.age = age;
        count++;
    }

    public static boolean canCreateMoreDucks() {
        return count < 10;
    }

    public String getName() {
        return name;
    }

    public boolean canDrinkAlcohol() {
        return age >= 18;
    }

Test Class – uses reflection API

Change the value of a private field inside a class

Field and getDeclaredField are used to access a variable. Using setAccessible as true will expose any private fields which can be manipulated.

@Test
public void testDuckCanDrinkAlcohol() {
    Duck duck = new Duck("Donald", 5);
    assertEquals("Donald", duck.getName());
    assertFalse(duck.canDrinkAlcohol());

    // change age and check if duck can drink alcohol
    // But I don't want to create a setter for this
    // Use reflection API to change the age
    try {
        Class<Duck> duckClass = Duck.class;
        Field ageField = duckClass.getDeclaredField("age");
        ageField.setAccessible(true);
        ageField.setInt(duck, 20);
    }
    catch (Exception e) {
        e.printStackTrace();
        assert false;
    }

    assertTrue(duck.canDrinkAlcohol());

}

Get the value of a static private variable in a class

A static field can be accessed in the similar way.

@Test
public void testDuckCanCreateMoreDucks() {
    // Instead of creating more ducks
    // I will use reflection API to change the count
    Duck duck = new Duck("Donald", 5);
    assertTrue(Duck.canCreateMoreDucks());

    // Also assert count was 1
    // But I don't want to create a getter for this
    try {
        Class<Duck> duckClass = Duck.class;
        Field countField = duckClass.getDeclaredField("count");
        countField.setAccessible(true);

        // Don't need to pass an instance as count is static
        Object countObject = countField.get(null);
        int count = (int) countObject;
        assertEquals(1, count);
    }
    catch (Exception e) {
        e.printStackTrace();
        assert false;
    }
}

Change the value of a static private variable in a class

You can use setInt to change the value of the Field.

@Test
public void testDuckCannotCreateMoreDucks() {
    // Instead of creating more ducks
    // I will use reflection API to change the count
    Duck duck = new Duck("Donald", 5);

    // change count to 10
    try {
        Class<Duck> duckClass = Duck.class;
        Field countField = duckClass.getDeclaredField("count");
        countField.setAccessible(true);
        countField.setInt(null, 10);
    }
    catch (Exception e) {
        e.printStackTrace();
        assert false;
    }

    assertFalse(Duck.canCreateMoreDucks());

}

Using Jmespath in Emacs

Shubham Kumar — Wed, 07 Feb 2024 03:30:13 +0000

Introduction

When you have a small JSON file, it is quite easy to look for what you want. But querying a large JSON data is very troublesome.

This is where people use tools like Jmespath to filter and transform data into their liking. This post is about a small wrapper over jp CLI utility that you can use while working on JSON files in Emacs.

Installing JP

Linux

On Linux, you can install the utility by first downloding the binary and then installing it.

> sudo wget https://github.com/jmespath/jp/releases/latest/download/jp-linux-amd64 \
> -O /usr/local/bin/jp && sudo chmod +x /usr/local/bin/jp

Mac

On Mac, you can install Jmespath CLI using brew.

> brew install jmespath/jmespath/jp

Adding Jmespath recipe in Emacs

This is my first recipe that I published on MELPA.Here are the steps to install it in Doom using straight. You can use any package manager to install it using MELPA repository.

Doom Emacs

On Doom, you can just mention the below recipe in your package.el.

> (package! jmespath)

Also, add the below line in your config.el.

> (use-package jmespath)

Using Jmespath

There is an interactive function, jmespath-query-and-show that you can use to query the currently opened buffer or a file. To use it with current buffer, you can simply call this function and enter your query. The output will be shown on a new buffer with name JMESPath Result.

To use this with a different file, you can set the Universal Argument using C-u or SPC-u (using evil). Then you can enter the file to execute the query on.

To learn more about JMESPath, visit Jamespath page.

Running PMML models in Erlang using NIF and CPP

Shubham Kumar — Sun, 15 Oct 2023 05:10:32 +0000

Introduction

Erlang is a great language for building concurrent systems that are fault-tolerant and scalable. But it lacks some of the libraries that are available in other languages. One such example is using PMML files for machine learning models. At the time of writing, Erlang doesn't have a library for parsing PMML files. This is a problem for people who want to use Erlang for building machine learning systems. Here I'll show how to use C++ to build a NIF that can be used in Erlang to parse PMML files. More specifically, I'll use the cPMML library to build a NIF that can be used in Erlang.

Erlang NIF

Erlang NIF provides you a way to define your functions in C/C++ and call them in Erlang code natively. The C/C++ program is compiled to generate a library file that can be used in Erlang. This library is dynamically linked to the Erlang VM and is the fastest way of calling the C/C++ code from Erlang. The disadvantage of the approach is that if the C/C++ code crashes, it will crash the Erlang VM. And you will need to maintain the C/C++ code along with the Erlang code.

Prerequisites

cPMML library requires a version of C++ that supports C++11 standard. Also, make sure you have the required header files on the Erlang side.

You can find them on Mac OS or Linux using the find function. If the header files are located in multiple locations, use the Celler one.

find / -name erl_nif.h | grep erl_nif.h

Hello Nif program

C/C++ code

Header file

You need to include the below header file to use the Erlang functionalities. On a lower level, it defines the data structures and environment that Erlang provides to the C/C++ code.

#include <erl_nif.h>

Function Definition

These functions are called from Erlang code. They must follow a specific structure and return a specific type of value. Here, we will define a function that will return a "Hello, World!" string.

ERL_NIF_TERM is the return type of the function. It is an interface for various return types like binary, tuple, list, atom, etc. Meaning, that you can return any of these types from the function.

ErlNifEnv is the pointer to the Erlang environment. It provides you access to various Erlang functionalities like memory management, exception handling and Erlang term creation. For us, it will help in creating Erlang Terms like string.enif_make_string is the function that will create a string from the C/C++ code.

The argc and argv provide the number of arguments and the arguments passed to the function from Erlang code.

Below is the function definition for hello world.

static ERL_NIF_TERM hello_world(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    return enif_make_string(env, "Hello, World!", ERL_NIF_LATIN1);
}

Export functions

You need to specify the functions that you want to export to the Erlang code. The structure is a list of ErlNifFunc objects. Each object has the name of the function that Erlang sees, the number of arguments and the function pointer.

static ErlNifFunc nif_funcs[] = {
    {"hello_world", 0, hello_world}
};

To initialize the NIF library, you need to call the ERL_NIF_INIT function. This takes in the name of your Erlang module and the exported functions.

Let's call our Erlang module hello_nif.

ERL_NIF_INIT(hello_nif, nif_funcs, NULL, NULL, NULL, NULL)

Final code

extern "C" {
  #include <erl_nif.h>
  static ERL_NIF_TERM hello_world(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
      return enif_make_string(env, "Hello, World!", ERL_NIF_LATIN1);
  }
  static ErlNifFunc nif_funcs[] = {
      {"hello_world", 0, hello_world}
  };
}

ERL_NIF_INIT(hello_nif, nif_funcs, NULL, NULL, NULL, NULL);

Erlang code

Erlang module

We will name our Erlang module hello_nif.erl.

-module(hello_nif).

Define your NIF functions

This specifies the functions that are exported from the C/C++ code.

-export([hello_world/0]).
-nifs([hello_world/0]).

Load the library on module load

If you compiled your C/C++ code to a library named hello_nif.so, you can load it using the load_nif function.

-on_load(init/0).

init() ->
    ok = erlang:load_nif("./hello_nif", 0).

Fallback function

The C/C++ code can crash and you need to handle that in the Erlang code. These functions will run when the C/C++ code crashes. You should name each of these functions with the same name in the C/C++ code.

hello_world() ->
    exit(nif_library_not_loaded).

Final code

-module(hello_nif).
-nifs([hello_world/0]).
-on_load(init/0).

init() ->
    ok = erlang:load_nif("./hello_nif", 0).

hello_world() ->
    exit(nif_library_not_loaded).

Compiling and Running

To compile the C/C++ code you can use gcc.

Mac OS

gcc -o hello_nif.so hello.c -I /usr/local/lib/erlang/erts-13.2.2.2/include/ -bundle -bundle_loader /usr/local/lib/erlang/erts-13.2.2.2/bin/beam.smp

Linux

gcc -o hello_nif.so hello.c -I /usr/lib/erlang/erts-13.2.2.2/include -shared -fpic

On Erlng, you can call the hello_world function.

> c(hello_nif).
> hello_nif:hello_world().
"Hello, World!"

cPMML NIF

We saw how to create a simple NIF that returns a string. Now, let's create a NIF that can take input and run prediction using a PMML model file. For this, we will use the same model file we created in the previous post.

We have a PMML model for linear regression, y = 2x + 1 and we want to predict the value of y for a given value of x. Keeping it short, we will expose a function called, predict that will take the input and return the output.

C/C++ code

Assuming you can compile cPMML files as described in the previous post.

Header files

#include <iostream>
#include <vector>
#include <string>
#include <map>
#include "cPMML.h"

using namespace std;

PMML Parser class

We want to load the model once and use it for multiple predictions. For this, let's create a class that will load the model and call predictions on it. We will maintain only one instance of this class and use it for all the predictions.

class PmmlModelParser {
private:
  cpmml::Model model;

public:
    PmmlModelParser(const string& modelname) {
        model = cpmml::Model(modelname);
    }

    string predict(const unordered_map<string, string>& x_input) {
        return model.predict(x_input);
    }
};

// Global variable
PmmlModelParser *pmmlModelParser = nullptr;

NIF implementation

We will create two functions, init and predict. The init function will take the PMML file as input and load it. The predict function will take the input and return the output.

The below code describes the structure of the NIF functions.

extern "C" {
    #include <erl_nif.h>

    static ERL_NIF_TERM init(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {}

    static ERL_NIF_TERM predict(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {}

    static ErlNifFunc nif_funcs[] = {
        {"init", 1, init},
        {"evaluate", 1, predict}
    };
}

ERL_NIF_INIT(lr_model, nif_funcs, NULL, NULL, NULL, NULL)

init function

The init function will take the PMML file as input and load it. We will use the enif_inspect_binary function to get the PMML file as a binary. Then we will convert it to a string and pass it to the PmmlModelParser class.

static ERL_NIF_TERM init(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    ErlNifBinary input_bin;
    if (!enif_inspect_binary(env, argv[0], &input_bin)) {
        return enif_make_badarg(env);
    }
    string input(reinterpret_cast<char*>(input_bin.data), input_bin.size);
    pmmlModelParser = new PmmlModelParser(input);
    return enif_make_atom(env, "ok");
}

predict function

The predict function will take the input and return the output. The input will be a string and the output will be a string on the Erlang side.

But on the C/C++ side, we will convert the input to a map and pass it to the PmmlModelParser class because cPMML library expects a map as input. Don't get intimidated by the code below, it's just converting the Erlang map to a C++ map by iterating over all the keys and value pairs.

For prediction, we will need a map with the key as X and value as the input.

I could have just used a binary input as X and converted it to a string on the C/C++ side. But, to make this more general, I am converting the Erlang map to a C++ map. Now this can be used for any PMML file and not just a specific one.

static ERL_NIF_TERM predict(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
  unordered_map<std::string, std::string> cpp_map;

  if (!enif_is_map(env, argv[0]))
  {
    return enif_make_badarg(env);
  }

  ErlNifMapIterator iter;
  if (enif_map_iterator_create(env, argv[0], &iter, ERL_NIF_MAP_ITERATOR_FIRST))
  {
    do
    {
      ERL_NIF_TERM key, value;
      if (enif_map_iterator_get_pair(env, &iter, &key, &value))
      {
        ErlNifBinary key_bin, value_bin;
        if (enif_map_iterator_get_pair(env, &iter, &key, &value))
        {
          char key_str[64], value_str[64];
          if (enif_get_string(env, key, key_str, sizeof(key_str), ERL_NIF_LATIN1) &&
              enif_get_string(env, value, value_str, sizeof(value_str), ERL_NIF_LATIN1))
          {
            cpp_map[key_str] = value_str;
          }
        }
      }
    } while (enif_map_iterator_next(env, &iter));
    enif_map_iterator_destroy(env, &iter);
  }

  string ret = pmmlModelParser->predict(cpp_map);
  return enif_make_string(env, ret.c_str(), ERL_NIF_LATIN1);
}

Erlang code

We will name our Erlang module lr_model.erl.

-module(lr_model).
-export([init/1, evaluate/1]).
-nifs([init/1, evaluate/1]).
-on_load(init/0).

init() ->
    ok = erlang:load_nif("./lr_model", 0).

init(PmmlFile) ->
    exit(problem_loading_nif).

evaluate(Input) ->
    exit(problem_loading_nif).

Compiling and Running

Compiling the C/C++ code will now take an extra parameter to include cPMML library. As the model is predicting the values for y = 2x + 1, we should get ~3 for x = 1 and ~1 for x = 0.

g++ -std=c++11 library.cpp \
-o lr_model.so \
-lcPMML \
-I /usr/local/lib/erlang/erts-13.2.2.2/include/ \
-bundle \
-bundle_loader /usr/local/lib/erlang/erts-13.2.2.2/bin/beam.smp


> c(lr_model).
> lr_model:init(<<"./lr_model.pmml">>).
> lr_model:evaluate(#{"X"=>"0"}).
"0.967265"
> lr_model:evaluate(#{"X"=>"1"}).
"3.007469"

Conclusion

In this blog, we saw how to use C/C++ libraries in Erlang code. We used a PMML file of a linear regression model to predict the values of y = 2x + 1 for a given value of x in Erlang where the code was implemented in C++. We saw how to use the cPMML library to parse the PMML file and use it for predictions.

Running AI/ML predictions in CPP using cPMML library

Shubham Kumar — Sun, 01 Oct 2023 23:39:47 +0000

Introduction

PMML is a markup language to save your AI/ML model files so that you can use them for predictions later on (maybe during production).cPMML is a library created by the AmadeusITGroup to parse and run predictions in C++. In this blog, we will train a linear regression model inpython and generate a pmml file and then we will run our predictions in C++.

Creating a model file

Dependencies

We will need pandas, numpy, scikit-learn and sklearn2pmml.

pip install pandas numpy scikit-learn sklearn2pmml

Imports

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn2pmml import sklearn2pmml
from sklearn2pmml.pipeline import PMMLPipeline
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

The model

Dataset

For keeping things simple, let's train a linear regression model to match the equation, y = 2x + 1. We can generate a random dataset for this equation.

X = np.random.rand(100, 1)
Y = 2 * X + 1 + 0.1 * np.random.randn(100, 1)

Test/Train data

Next, we'll divide the data into test and train datasets.

df = pd.DataFrame({'X': X.flatten(), 'Y': Y.flatten()})
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
X_train = train_df[['X']]
y_train = train_df['Y']
X_test = test_df[['X']]
y_test = test_df['Y']

Training the model

For training the model, we can get the model from scikit learn library and use the dataset we generated above. We can also check the mse to get an idea of the model's accuracy.

pipeline = PMMLPipeline([
    ("regressor", LinearRegression())
])

pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse}")

Saving the pmml file

If you are satisfied by the performance of your model, you can export the model as a pmml file. We will save the model with the name, lr_model.pmml

sklearn2pmml(pipeline, "lr_model.pmml", with_repr = True)

Using the model file

The main step of focus in this blog is using the model in C++ program. For this, you will need to isntall the cPMML library.

Installing cPMML

To install the libray in your system, you just need to run the below command. This will run cmake, so you should have cmake installed in your system.

git clone https://github.com/AmadeusITGroup/cPMML.git && cd cPMML && ./install.sh

For Mac M1

I ran into some problems while installing this on Mac M1. Here are the steps to install this effortlessly.

Ensure you have the latest version of cmake installed in your system.
You can edit the install.sh script to remove -j 4 flag from the cmake -j 4 .. command. This will turn off the multi processing.
The last line of the install.sh script is sudo ldconfig. Change this to sudo update_dyld_shared_cache. This installs the .dylib or .so library files to proper destination.

Running the predictions

Include the library

The first thing is to import the library.

#include "cPMML.h"
#include <iostream>

Load the model

Then you can load the model.

int main() {
  cpmml::Model model("lr_model.pmml");
  return 0;
}

Start predictions

The cPMML library takes input as an unordered_map of strings. For us, there is only one input which is X.

int main() {
  cpmml::Model model("lr_model.pmml");

  // This shoule yield a value close to 1
  std::unordered_map<std::string, std::string> input1 = {
    {"X", "0"}
  };

  // This should yield a value close to 21
  std::unordered_map<std::string, std::string> input2 = {
    {"X", "10"}
  };

  std::cout<<"X = 0 Y = "<<model.predict(input1)<<'\n';
  std::cout<<"X = 10 Y = "<<model.predict(input2)<<'\n';

  return 0;
}

Compilation

You can compile the code by including the cPMML library.

> g++ -std=c++11 predict.cpp -o predict.o -lcPMML
> ./predict.o
X = 0 Y = 0.967265
X = 10 Y = 21.369305

Conclusion

In this blog, we saw how to store your model as a PMML file and load it in C++ using cPMML library. You can view the code for the above here.

Embedded Jetty server with handlers for legacy Java applications

Shubham Kumar — Thu, 07 Sep 2023 00:00:00 +0000

Introduction

Jetty is a very powerful yet lightweight Java library since ages that helps you create servers and clients for HTTP (literally all the versions), Web Sockets, OSGI, JMX, JAAS and much more.

This post will deal with embedding a Jetty server in a Java application and adding handlers for different requests. I'm assuming you have a basic understanding of Java and Maven and have familiarity with different HTTP methods like GET, POST, PUT, DELETE, etc.

Jetty Server Architecture

An incoming request is handled using 4 components - Threadpool, Connectors, Handlers and Server.Server is the core of the system that handles the management of servers and the entire lifecycle of the server.Connectors help in accepting various requests over different protocols like HTTP, HTTPS, etc.Handlers are the components that process the incoming request.Threadpools are like tiny workers that make the system multi-threaded and help in handling multiple requests at the same time.

An incoming request is first accepted at the Connector. Then it is sent to the Server which connects it to the Handler where the main response is generated. And the response is generated using a particular thread as per Threadpool.

Using Jetty in a Java application

New Java App

First, let's create a new Java app using Maven.

> mvn archetype:generate \
-DgroupId = com.schwiftycold.poc \
-DartifactId = poc_jetty_server \
-DarchetypeArtifactId = maven-archetype-quickstart \
-DinteractiveMode = false

Add the following properties to your POM file. This will set the Java version and language encoding for the app.

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>20</maven.compiler.source>
    <maven.compiler.target>20</maven.compiler.target>
  </properties>

Add Jetty dependency

Add the following dependency in your pom.xml file.

<dependencies>
  <dependency>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-server</artifactId>
    <version>11.0.3</version>
  </dependency>
  ...
</dependencies>

Main server

The main component of the system is the server class. This class will create a server context and register connectors and handlers. Let's call this class MainServer.

Threadpool

There are different kinds of Threadpool offered by Jetty like QueuedThreadPool, ExecutorThreadPool, ScheduledThreadPool, etc.

QueuedThreadPool maintains a fixed number of threads and a queue to manage incoming requests. When a request comes and the thread is busy then it is added to the queue and will be picked when a thread is available. And, due to its nature, this is widely used for handling http requests.

ScheduledThreadPool extends QueuedThreadPool to handle scheduled tasks. It provides us with a scheduler to execute tasks at a certain interval. It is generally used for scheduling background tasks.

ExecutorThreadPool enables you to use custom Executor as the Threadpool for Jetty.

Here, we will be using QueuedThreadPool.

ThreadPool threadPool = new QueuedThreadPool();

Server

The server is the essential component that manages the connectors and handlers. It takes the Threadpool to create a new server instance.

Server server = new Server(threadPool);

Connector

A connector allows us to accept a variety of different protocols like HTTP, HTTPS, Unix domain socket, etc.

HTTP

A simple HTTP connector can be initialized using the ServerConnector class. And you can change the port to listen to using, setPort function. By default, the port is 8080.

ServerConnector connector = new ServerConnector(server);
connector.setPort(9120);
server.setConnectors(new Connector[]{connector});

HTTPS

For using HTTPS you'll need to register the SSL/TLS certificate. We won't be using this here but a quick look into this will give us a glance of the infinite possibilities.

First, you will need the sslContextFactory which defines your SSL/TLS configurations.

SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setKeyStorePath("/path/to/keystore.jks");
sslContextFactory.setKeyStorePassword("keystore-password");
sslContextFactory.setKeyManagerPassword("key-password");


ServerConnector httpsConnector = new ServerConnector(
    server,
    new SslConnectionFactory(sslContextFactory, "http/1.1")
);
httpsConnector.setPort(8443);
server.setConnectors(new Connector[] {httpsConnector});

Handlers

Handlers are the ones that will be creating or response. For this, let's create a new singleton class. Let's set this to null for now.

server.setHandler(null);

Final code

public class MainServer {
  public static void startServer() {
    try {
      ThreadPool threadPool = new QueuedThreadPool();

      Server server = new Server(threadPool);

      ServerConnector connector = new ServerConnector(server);
      connector.setPort(9120);

      server.setConnectors(new Connector[]{connector});
      server.setHandler(null);

      server.start();
      server.join();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Handler class

Handler class is where we will write our logic to process the request. So this requires a special attention.

Let's create a new class that extends AbstractHandler

Extend AbstractHandler

The AbstractHandler required us to implement a handle function. This function is triggered on every new request.

The handle method uses four parameters. The target parameter denotes the endpoint that was triggered. If we trigger the URL, http://localhost:9120/hi then the target will take the value as /hi.

baseRequest and request denote the same thing but in a different context.baseRequest is Jetty specific whereas request donates the servlet-specific APIs. Here, we won't be using the servlet APIs.

The response is the processed result generated by the server which will be sent to the client.

public class MainHandler extends AbstractHandler{

  @Override
  public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)
      throws IOException, ServletException {
        System.out.println("target: " + target);
        System.out.println("baseRequest: " + baseRequest);
        System.out.println("request: " + request);
        System.out.println("response: " + response);     
  }
}

The output of the above code would be something as follows on triggering with curl.

> curl -X POST "http://localhost:9120/hi" -d "p=value"

target: /hi
baseRequest: Request(POST http://localhost:9120/hi)@57ffc3ca
request: Request(POST http://localhost:9120/hi)@57ffc3ca
response: HTTP/1.1 200

Here, you can see the target as /hi, the request denotes the method call and reference to the request object and the response is just 200 which denotes a success. We can manipulate this information to create responses based on different inputs.

Manage different methods

We can get the request method using the response object by calling the function, getMethod.

public void handle(...) {
    if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("POST request received");
    }
    else if ("GET".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("GET request received");
    }
}

Now, calling curl as above will print POST request received on the server side.

Extract the query parameters

The query parameter and its values can be extracted from the base request object by using the getParameterNames method. Here, I'm parsing the values and storing them in a Map for accessing them later.

if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
  System.out.println("POST request received");
  Map<String, String> queryParams = new HashMap<>();

  for (Enumeration<String> e = request.getParameterNames(); e.hasMoreElements();) {
    String name = e.nextElement();
    String[] values = request.getParameterValues(name);
    for (String value : values) {
      queryParams.put(name, value);
    }
  }

  System.out.println("Query params: " + queryParams);
  ...

Now, you can see the below response on the server side.

POST request received
Query params: {p=value}

Generating the result

The last part is generating the result for specific triggers. This is done by using getWriter function to write to the response. You can also set the status of the response using setStatus method. You can create more if statements for each endpoint.

try {
  if (target.startsWith("/hi")) {
    response.setStatus(HttpServletResponse.SC_OK);
    response.getWriter().println("Hello World");
    baseRequest.setHandled(true);
  }
} catch (Exception e) {
  e.printStackTrace();
}

Final handle method

  public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)
      throws IOException, ServletException {
    if ("POST".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("POST request received");
      Map<String, String> queryParams = new HashMap<>();

      for (Enumeration<String> e = request.getParameterNames(); e.hasMoreElements();) {
        String name = e.nextElement();
        String[] values = request.getParameterValues(name);
        for (String value : values) {
          queryParams.put(name, value);
        }
      }

      System.out.println("Query params: " + queryParams);

      System.out.println("Target: " +target.startsWith("/hi"));

      try {
        if (target.startsWith("/hi")) {
          System.out.println("Target: " +target);
          response.setStatus(HttpServletResponse.SC_OK);
          response.getWriter().println("Hello World");
          baseRequest.setHandled(true);
        }
      }
      catch (Exception e) {
        e.printStackTrace();
      }

    } else if ("GET".equalsIgnoreCase(baseRequest.getMethod())) {
      System.out.println("GET request received");
    }
  }

Connect Handler and MainServer

Before testing your new server, you'll also need to add the handler which can be done using the server's setHandler method.

server.setHandler(new MainHandler());

Now calling the hi endpoint will return Hello World as a response.

> curl -X POST "http://localhost:9120/hi" -d "p=value"

Hello World

Conclusion

In this post, we saw how to configure Jetty server and set handlers to generate responses for a particular endpoint trigger. This looks like a long post but it revolves around just creating a 'Hello World' server using Jetty. You can find the code for the above here.

Forem: Shubham Kumar

Working on mulitple braches at once using git worktree

Working on multiple branches at once

Worktree creation

List existing worktrees

Remove a worktree

Advice on working with worktrees

Displaying full content in rss.xml in Hugo

Partials in Hugo

Partials and code reusability

baseof.html

head.html

Defining custom partials

Make CSS styling a partial

Modify the pages to use baseof

Summary

Tailwind support in Hugo

Expectations from this post

Tailwind and other supporting apps

PostCSS config

Tailwind config

Loading Tailwind on the index page

main.css

index.html

Taking a peek

Summary

Upgrading Hugo to latest version on Debian based systems

Object pool design pattern in Java

What is it?

Why is this?

Example

How to implement this?

Downloading a single file from 2 independent apps

Understanding the problem

Solution

Download with file locking

Outputs

Conclusion and improvements

Reflection API in Java

Where is this used?

Main Class

Test Class – uses reflection API

Change the value of a private field inside a class

Get the value of a static private variable in a class

Change the value of a static private variable in a class

Using Jmespath in Emacs

Introduction

Installing JP

Linux

Mac

Adding Jmespath recipe in Emacs

Doom Emacs

Using Jmespath

Read more

Running PMML models in Erlang using NIF and CPP

Introduction

Erlang NIF

Prerequisites

Hello Nif program

C/C++ code

Header file

Function Definition

Export functions

Final code

Erlang code

Erlang module

Define your NIF functions

Load the library on module load

Fallback function

Final code

Compiling and Running

Mac OS

Linux

cPMML NIF

C/C++ code

Header files

PMML Parser class

NIF implementation

init function

predict function