Forem: Bhupesh Varshney 👾

How I reduced the size of my very first published docker image by 40% - A lesson in dockerizing shell scripts

Bhupesh Varshney 👾 — Sat, 03 Feb 2024 00:00:00 +0000

I interact with Dockerfiles every day at work, have written a few myself, built containers, and all that. But never published one on the docker hub registry. I wanted to make ugit - a tool to undo git commands (written as a shell script) available to folks who don’t like installing random shell scripts from the internet.

Yeah, I know, I know. REWRITE IT IN GO/RUST/MAGICLANG. The script is now more than 500+ lines of bash. I am not rewriting it in any other language unless someone holds a gun to my head (or maybe ends up sponsoring??). Moreover, ugit is close to being feature complete (only a few commands left to undo, which are not that commonly used).

Anyway, the rest of the article is about how I went about writing the official Dockerfile for ugit (a shell script) and reduced the image size by almost 40% (going from 31.4 MB to 17.6 MB) by performing step-by-step guided optimization attempts. I hope this motivates other shell enthusiasts to also publish their scripts as docker images!

PS: I am not a DevOps or Docker expert, so if you are and see something wrong or something that could be done better, please let me know in the comments below or reachout somewhere.

The very first Dockerfile ~attempt~
Path to optimization - reducing image size by 40%
2nd attempt - alpine on alpine
- Looks like xargs and awk came free?
3rd attempt - Using scratch at 2nd stage
- Identifying transitive dependencies
- Primer on shared libraries
- Shebangs #! are useless
- A look at terminfo db
Everything needed to run ugit
Could we reduce the size further?
- Reason 1: Pin minimum version of fzf?
- Reason 2: Use the latest bash features?
You didn’t try docker-slim?
You didn’t try docker-squash?
Learnings
Acknowledgements
Resources

The very first `Dockerfile` ~attempt~

# Use an official Alpine runtime as a parent image
FROM alpine:3.18

# Set the working directory in the container to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install dependencies
RUN apk add --no-cache \
    bash \
    gawk \
    findutils \
    coreutils \
    git \
    ncurses \
    fzf

# Set permissions and move the script to path
RUN chmod +x ugit && mv ugit /usr/local/bin/

# Run ugit when the container launches
CMD ["ugit"]

Looks pretty simple, right? It is. You could copy-paste this Dockerfile and build the image yourself assuming you have ugit cloned in the current directory,

docker build -t ugit .
docker run --rm -it -v $(pwd):/app ugit

This should successfully run ugit inside the container.

ugit requires binaries like bash (>=4.0), awk, xargs, git, fzf, tput, cut, tr, nl.

We had to install findutils because it ships with xargs.
We had to install coreutils because it ships with tr, cut and nl.
ncurses is required by tput (which is used to get the terminal info).

That’s all we need to run ugit on a UNIX-based machine or well, a container. The image size sits at 31.4 MB at this point. Not bad for a first attempt. Let’s see how we can reduce it further.

Path to optimization - reducing image size by 40%

During our (desperate) micro-optimization attempts in the upcoming sections, we will be covering the following high-level goals:

Use multi-stage builds to reduce image size.
Get rid of binaries like sleep, watch, du etc. Anything that is not required by ugit to run.
Get rid of unnecessary dependencies that these binaries bring in.
Get a minimum version of all dependencies that are required by ugit to run.
Load up only the required binaries and their dependencies in the final image.

2nd attempt - `alpine` on `alpine`

I now choose to create a multi-stage build. The 2nd stage will be used to copy only the required binaries and their dependencies. I again choose to use alpine as the base image for this stage.

# First stage: Install packages
FROM alpine:3.18 as builder

RUN apk add --no-cache \
    bash \
    gawk \
    findutils \
    coreutils \
    git \
    ncurses \
    fzf

# Copy only the ugit script into the container at /app
WORKDIR /app
COPY ugit .

# Set permissions and move the script to path
RUN chmod +x ugit && mv ugit /usr/local/bin/

# Second stage: Copy only necessary binaries and their dependencies
FROM alpine

COPY --from=builder /usr/local/bin/ugit /usr/bin/
COPY --from=builder /usr/bin/git /usr/bin/
COPY --from=builder /usr/bin/fzf /usr/bin/
COPY --from=builder /usr/bin/tput /usr/bin/
COPY --from=builder /usr/bin/cut /usr/bin/
COPY --from=builder /usr/bin/tr /usr/bin/
COPY --from=builder /usr/bin/nl /usr/bin/
COPY --from=builder /usr/bin/gawk /usr/bin/
COPY --from=builder /usr/bin/xargs /usr/bin
COPY --from=builder /usr/bin/env /bin/
COPY --from=builder /bin/bash /bin/

WORKDIR /app
# Run ugit when the container launches
CMD ["ugit"]

Just by a straightforward multi-stage build, we were able to reduce the image size to an impressive 20.6 MB. Now the image builds successfully, but it won’t run ugit yet.

Error loading shared library libreadline.so.8: No such file or directory (needed by /bin/bash)

Turns out, we are missing transitive dependencies. More about this on our 3rd attempt.

Looks like `xargs` and `awk` came free?

Turns out, that both xargs and awk are present by default on the Alpine image. You can verify this by running the following commands:

docker run -it alpine /bin/sh -c "awk --help"
docker run -it alpine /bin/sh -c "xargs --help"

Rookie mistake. Let’s scratch gawk and findutils from our Dockerfile.

# First stage: Install packages
FROM alpine:3.18 as builder

RUN apk add --no-cache \
    bash \
    coreutils \
    git \
    ncurses \
    fzf

# Copy only the ugit script into the container at /app
WORKDIR /app
COPY ugit .

# Set permissions and move the script to path
RUN chmod +x ugit && mv ugit /usr/local/bin/

# Second stage: Copy only necessary binaries and their dependencies
FROM alpine:3.18

COPY --from=builder /usr/local/bin/ugit /usr/bin/
COPY --from=builder /usr/bin/git /usr/bin/
COPY --from=builder /usr/bin/fzf /usr/bin/
COPY --from=builder /usr/bin/tput /usr/bin/
COPY --from=builder /usr/bin/cut /usr/bin/
COPY --from=builder /usr/bin/tr /usr/bin/
COPY --from=builder /usr/bin/nl /usr/bin/
COPY --from=builder /usr/bin/env /bin/
COPY --from=builder /bin/bash /bin/

WORKDIR /app
# Run ugit when the container launches
CMD ["ugit"]

The image size is now down to 20 MB. We are getting there. ugit still won’t run, though.

3rd attempt - Using `scratch` at 2nd stage

This is the part, that most folks don’t attempt. It’s a bit scary and requires a huge commitment. I was a bit scared to go this route as well because I knew everything could be ~~SCRATCHED~~ 🙃.

A SCRATCH docker image is just an empty file system. It doesn’t have anything at all. To try out a SCRATCH image, you can refer to docker hub’s README on it.

The only thing you need to know is that everything has to be put together by us. Let’s keep our hands on our hearts and replace alpine with scratch.

# First stage: Install packages
FROM alpine:3.18 as builder

RUN apk add --no-cache \
    bash \
    coreutils \
    git \
    ncurses \
    fzf

# Copy only the ugit script into the container at /app
COPY ugit .

# Set permissions and move the script to path
RUN chmod +x ugit && mv ugit /usr/local/bin/

# Second stage: Copy only necessary binaries and their dependencies
FROM scratch

COPY --from=builder /usr/local/bin/ugit /usr/bin/
COPY --from=builder /usr/bin/git /usr/bin/
COPY --from=builder /usr/bin/fzf /usr/bin/
COPY --from=builder /usr/bin/tput /usr/bin/
COPY --from=builder /usr/bin/cut /usr/bin/
COPY --from=builder /usr/bin/tr /usr/bin/
COPY --from=builder /usr/bin/nl /usr/bin/
COPY --from=builder /usr/bin/env /bin/
COPY --from=builder /bin/bash /bin/

WORKDIR /app
# Run ugit when the container launches
CMD ["ugit"]

Doing this reduced the size of our image to 12.4MB, a 60% reduction?? Did we just rickroll ourselves? Let’s try to run ugit.

$ docker run --rm -it -v $(pwd):/app ugit-a3
exec /usr/bin/ugit: no such file or directory

$ docker run --rm -it --entrypoint /bin/bash ugit-a4
exec /bin/bash: no such file or directory

Turns out, we broke the bash binary by not shipping its dependencies. Let’s see what we can do about it.

Identifying transitive dependencies

Okay, time to talk about transitive dependencies. Our script relies on binaries like git, tput, bash; now some of these utils may have their dependencies.

We technically call these dependencies, shared libraries. Shared libraries are .so (or in Windows .dll, or in OS X .dylib) files. ldd is a great tool to identify these dependencies. It lists all the libraries needed by a binary to execute. For example, if we run ldd /bin/bash on a fresh Alpine container, we get the following output:

/ # ldd /bin/ls
    /lib/ld-musl-aarch64.so.1 (0xffff8c9c0000)
    libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff8c9c0000)
/ # ldd /bin/bash
    /lib/ld-musl-aarch64.so.1 (0xffffb905a000)
    libreadline.so.8 => /usr/lib/libreadline.so.8 (0xffffb8f06000)
    libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0xffffb905a000)
    libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0xffffb8e95000)

Primer on shared libraries

Each lib has a soname, which is the name of the library file without the version number. They start with the prefix lib and end with .so. For example, libpcre2-8.so.0 has a soname libpcre2-8.so.
A fully qualified lib includes the directory where it is located. For example, /usr/lib/libpcre2-8.so.0.
Each lib has a real name, which is the name of the library file with the version number. For example, libpcre2-8.so.0.3.6 could be a real name.
The hexadecimal is the base memory address where this library is loaded in memory. Not useful for us. We are only interested in the library names. Let’s do it for all the binaries we need to run ugit.
The path to the right-hand side of => symbol indicates the real path to that shared library.
The lib name starting with libcis the C library for that architecture. Remember our “exec /bin/bash: no such file or directory” error? This is the reason we got it. We didn’t ship the C library for our architecture.

Below is an excerpt from Douglas Creager’s article on Shared library versions which sums up shared libraries, please read the full article if you want to learn more about shared libraries.

With a shared library, you compile the library once and install it into a shared location in the filesystem (typically /usr/lib on Linux systems). Any project that depends on that shared library can use that shared, already compiled representation as-is.

Most Linux distributions further reduce compile times by distributing binary packages of popular libraries, where the distribution’s packaging system has compiled the code for you. By installing the package, you download a (hopefully signed) copy of the compiled library, and place it into the shared location, all without ever having to invoke the compiler (or any other part of the build chain that produced the library).

We use a similar approach to identify all the unique dependencies for all the binaries we need to run ugit.

ℹ️ We did this by running a command like for cmd in /bin/*; do echo $cmd; ldd $cmd; done

# copy lib files
COPY --from=ugit-ops /usr/lib/libncursesw.so.6 /usr/lib/
COPY --from=ugit-ops /usr/lib/libncursesw.so.6.4 /usr/lib/
COPY --from=ugit-ops /usr/lib/libpcre* /usr/lib/
COPY --from=ugit-ops /usr/lib/libreadline* /usr/lib/

COPY --from=ugit-ops /lib/libacl.so.1 /lib/
COPY --from=ugit-ops /lib/libattr.so.1 /lib/
COPY --from=ugit-ops /lib/libc.musl-* /lib/
COPY --from=ugit-ops /lib/ld-musl-* /lib/
COPY --from=ugit-ops /lib/libutmps.so.0.1 /lib/
COPY --from=ugit-ops /lib/libskarnet.so.2.13 /lib/
COPY --from=ugit-ops /lib/libz.so.1 /lib/

Notice, that we are copying libc.musl-* and ld-musl-* from the builder image. This is because the build for these libs depends on the architecture of the host machine.

Shebangs `#!` are useless

If you look at the very first line of ugit, you’ll see a shebang #!/usr/bin/env bash. The use of env is considered a good practice when writing shell scripts, used to tell the OS which shell interpreter to use to run the script, this is ideal in an everyday dev machine. Linux (and older versions of macOS) get shipped with sh, bash, and on top of it, folks install zsh etc.

But since using shebangs is optional, and we already copy the bash binary, we just need to invoke our script using it. This saves us a couple of bytes in the image size as well. Close to 1.9 MB to be precise.

# Run ugit when the container launches
CMD ["/bin/bash", "/bin/ugit"]
# or 
# ENTRYPOINT ["/bin/bash", "/bin/ugit"]

A look at `terminfo` db

ugit has colors, thanks to tput. We load up a bare-bones Alpine image with bash and head over to the /etc/terminfo directory. This directory contains the terminal info database.

37a1a77f70ed:/app# cd /etc/terminfo/
37a1a77f70ed:/etc/terminfo# ls
a d g k l p r s t v x

Each of these letter-based “directories” represents different $TERM types. For example, xterm is a terminal type. If you run tput -T xterm colors on your local machine, you’ll get the number of colors your terminal supports. For xterm it should be 8, and in the case of xterm-256color it should be 256.

Now here’s our chance, to only support 1 terminal type amongst the 40+ that are present in the terminfo db. We can get rid of the rest of the terminal types. This saves us another 97Kb, very little but needed to clear up the clutter.

# copy terminfo database for only xterm-256color
COPY --from=ugit-ops /etc/terminfo/x/xterm-256color /usr/share/terminfo/x/

# Gib me all the colors
ENV TERM=xterm-256color

Everything needed to run `ugit`

The final Docker image sits at 17.6 MB with no security vulnerabilities (as reported by docker scout, at the time of writing this article). We have successfully reduced the image size by 40% compare to our first attempt.

Here’s the final Dockerfile:

FROM alpine:3.18 as ugit-ops

RUN apk add --no-cache \
    bash \
    coreutils \
    git \
    ncurses \
    curl

# Download fzf binary from GitHub, pin to 0.46.0, ugit requires minimum 0.21.0
RUN curl -L -o fzf.tar.gz https://github.com/junegunn/fzf/releases/download/0.46.0/fzf-0.46.0-linux_amd64.tar.gz && \
    tar -xzf fzf.tar.gz && \
    mv fzf /usr/bin/

# Copy only the ugit script into the container at /app
COPY ugit .

# Set permissions and move the script to path
RUN chmod +x ugit && mv ugit /usr/bin/

# Second stage: Copy only necessary binaries and their dependencies
FROM scratch

COPY --from=ugit-ops /usr/bin/ugit /bin/
COPY --from=ugit-ops /usr/bin/git /usr/bin/
COPY --from=ugit-ops /usr/bin/fzf /usr/bin/
COPY --from=ugit-ops /usr/bin/tput /usr/bin/
COPY --from=ugit-ops /usr/bin/nl /usr/bin/
COPY --from=ugit-ops /usr/bin/awk /usr/bin/
COPY --from=ugit-ops /usr/bin/xargs /usr/bin/
COPY --from=ugit-ops /usr/bin/cut /usr/bin/cut
COPY --from=ugit-ops /usr/bin/tr /usr/bin/tr
COPY --from=ugit-ops /bin/bash /bin/
COPY --from=ugit-ops /bin/sh /bin/

# copy lib files
COPY --from=ugit-ops /usr/lib/libncursesw.so.6 /usr/lib/
COPY --from=ugit-ops /usr/lib/libncursesw.so.6.4 /usr/lib/
COPY --from=ugit-ops /usr/lib/libpcre* /usr/lib/
COPY --from=ugit-ops /usr/lib/libreadline* /usr/lib/

COPY --from=ugit-ops /lib/libacl.so.1 /lib/
COPY --from=ugit-ops /lib/libattr.so.1 /lib/
COPY --from=ugit-ops /lib/libc.musl-* /lib/
COPY --from=ugit-ops /lib/ld-musl-* /lib/
COPY --from=ugit-ops /lib/libutmps.so.0.1 /lib/
COPY --from=ugit-ops /lib/libskarnet.so.2.13 /lib/
COPY --from=ugit-ops /lib/libz.so.1 /lib/

# copy terminfo database
COPY --from=ugit-ops /etc/terminfo/x/xterm-256color /usr/share/terminfo/x/

# Gib me all the colors
ENV TERM=xterm-256color

WORKDIR /app
# Run ugit when the container launches
CMD ["/bin/bash", "/bin/ugit"]

I decided to pin the version of fzf to 0.46.0 (the latest at the time of writing this article) because ugit requires a minimum 0.21.0 to run, and I figured what the heck, let’s pin it to the latest version.

ℹ️ docker run --rm -it -v $(pwd):/app ugit will run the ugit container. Make sure your current directory is a git repo.

This is how the final file system tree looks like:

This is everything to make our shell app work. No more, no less. Time for a beer? 🍺

PS: The final docker image came down to 16.2 MB. You can find the updated Dockerfile here. For the sake of this article, I kept the image size at 17.6 MB.

Could we reduce the size further?

Yes, but there are 2 reasons why I didn’t go any further:

Reason 1: Pin minimum version of `fzf`?

At the time of writing ugit, the script relied on fzf minimum version 0.20.0, it’s granted that the latest version is going to be larger than the minimum required version. So we should pin the old version, right? No. because then it introduces security vulnerabilities with fzf’s dependencies, i.e., Golang. As reported by docker scout quickview, the older version of Golang has a total of 66 security issues. Maybe they affect the image, maybe they don’t. But I am not taking that risk, I want to keep the image as clean as possible.

ℹ️ In the Alpine ecosystem, it is generally not advised to pin minimum versions of packages.

Reason 2: Use the latest bash features?

At the time of writing ugit, I relied on bash version 4.0. Both tr and cut could be replaced, if I shifted to a newer version of bash. i.e, 5.0. And that my friends is a breaking change. Getting rid of and would have saved me a couple of bytes, but I didn’t want to break the script for folks who are still on bash 4.0. It doesn’t matter if I am the only one left, my machine still has bash.

You didn’t try `docker-slim`?

I did, it did slim down my image, but it also broke the script with missing dependencies. Slim is great, the reader should check it out. Unfortunately, I couldn’t get it to work for ugit in my limited time.
Moreover, I wanted to learn how to do it with my own hands, rather than rely on a tool to do it for me.

You didn’t try `docker-squash`?

I did, and the size optimization was nearly negligible. Here’s a log of the squash process which I ran on a Linux (AMD) machine (ignore the size of the image, since we are on different architecture, the image size is different):

Learnings

Linux is awesome. Everything, every design decision, every tool inspiration that came out of it inherits the same awesomeness.
Never shy away from going into deep 🐇 rabbit holes of micro-optimizations. You learn a lot. There’s always something new to learn.

Acknowledgements

Big thanks to the authors of ldd, and everyone in the docker & alpine linux community.
Thanks to folks on the developersIndia discord for helping out with advice & suggestions.

Resources

I hope you guys got something interesting to read today. Until next time, happy hacking & before you go please give ugit a star & a docker pull? 🧡

Using multiple git user configs with credentials store

Bhupesh Varshney 👾 — Sat, 18 Feb 2023 15:05:41 +0000

So, you have decided to have 2 separate accounts for work and personal.
Here are steps to follow to use both git configurations simultaneously with the same .git-credentials.

Step 1: Separate Work/Personal directories

Create a work and a personsal directory in your Documents folder. Use these directories to separate your git directories.

  Documents
  ├── work
  │   ├── ...
  │   └── ...
  └── personal
      ├── ...
      └── ...

Step 2: Separate git config for each account

Create .gitconfig-personal in the personal dir.

  [credential]
    helper = store
  [user]
    name = Personal_Username
    email = personal_email@domain.com
  [credential "https://github.com"]
    username = Personal_Username
    helper = store

Create .gitconfig-work in the work dir.

  [credential]
    helper = store
  [user]
    name = Work_Username
    email = work_email@domain.com
  [credential "https://github.com"]
    username = Work_Username
    helper = store

Step 3: Update global `.gitconfig` to switch the profile based on directory

Update your global config using the following command.

  git config --global --edit

Add the following config.

  [includeIf "gitdir:~/Documents/work/"]
      path = ~/Documents/work/.gitconfig-work
  [includeIf "gitdir:~/Documents/personal/"]
      path = ~/Documents/personal/.gitconfig-personal

Step 4: Register Access Tokens

Check your .git-credentials file & add GitHub access tokens for each account.

  https://Personal_Username:PERSONAL_TOKEN@github.com
  https://Work_Username:WORK_TOKEN@github.com

That's it, the next time you pull/push/clone a private repo, git will automatically choose the correct token for each config.

How to stop having disappointing moments in life

Bhupesh Varshney 👾 — Sun, 28 Aug 2022 05:16:00 +0000

How many time have you been let down by your expectations?

Seriously, “expecting” anything from anyone is the root cause of disappointment in your life. You start leveling up your hopes in dream of “getting X” but in reality you don’t. Its sad but that’s how life works, its yin-yang remember?

Consider your average common scenario that can occur in everyone’s life:

Didn’t get the reply from crush?
Didn’t get that job?
Didn’t get that promotion?
Didn’t get that scholarship?

… the list goes on.

Consider a real-life absurd example which recently happened with me, I am a big MCU (Marvel Cenamatic Universe) buff, so I go to every Marvel movie that comes up next.

A lot of folks are excited about the release of Spiderman No Way Home, me on another hand has no expectations from this movie (probably not a big fan). I went to go this movie, and there you go in the first 20 minutes or so Daredevil pops up (I love this character, I would recommend watching the Daredevil TV series on Hotstar to understand why this was a big moment in the movie), nobody knew about this and honestly this moment made the movie 10x for me. No disappointment whatsoever.
Now consider the release of Doctor Strange in Multiverse of Madness, my hopes for this were high because we knew so much could happen in this movie but Marvel decided to go lightweight this time (I mean yes, it was a good storyline) but I was a little “disappointed” thinking a “lot of things” could happen.

I know this is very circumstantial & personal, but I encourage you to have a look at moments from your life on how many times have your felt something similar.

Getting disappointed with something can lead you to demotivation and self-doubt. Now this doesn’t mean to not do anything when expectations are not met. Let’s say you ordered chicken biryani using a food delivery app and if on arrival you don’t see any chicken in biryani then definitely raise a complaint to the app instead of letting it go.

My point is to do things that are in your control. Don’t overload it, Don’t overthink it.

What to do then?

Now I am not a life-coach but I believe simple changes in your thinking capacity can eliminate disappointment from your life.

Just give your best effort on things you can control. Whatever the results may be, positive or negative, accept them and move on. And ofc make a mental note on what things to improve on future.
Another thing that I like to practice nowadays is whenever I start raising my hope levels, I try to ground my thoughts back to reality, Just realising you are thinking to much helps.

Try to adapt a similar practice in your life as well, Say avoid overthinking things like

Oh God, If I had this much money I would buy …
If I get my dream job, I would do this and this …

I hope this helps the reader to atleast ponder over eliminating disappointing thoughts, until next time 👋

Automatically remind contributors to update their pull requests

Bhupesh Varshney 👾 — Sun, 28 Aug 2022 05:09:00 +0000

So from my short professional experience, I have realised that if you are working with a small team of individuals a lot of folks end-up authoring multiple pull-requests across X different repositories in an organisation which sometimes leads to managing multiple feature PRs at a time.

This can sometimes leads to author forgetting to update their branches and test their code with the most stable (or latest) version possible. Since a lifetime of a pull request can be anything from 1 day to 1 week, its important to keep your branches up to date. But we developers forget to do this timely.

This is not a problem of just small teams, opensource projects (be it small or big) usually have this problem as well.

How to fix?

I went ahead and made a minimal github action to solve the above problem by reminding contributors to update their branches whenever base branch gets a new commit. We can do this by simply leaving a comment on pull requests (you can see a demo in the banner of this post)

Here is a simple workflow on how to setup this action

name: PR Update Reminder
on:
  push:
    branches:
      - main
      - dev

env:
  # make sure to set this as env
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 

jobs:
  remind_authors:
    runs-on: ubuntu-latest
    name: Update PR Reminder Test
    steps:
      - uses: actions/checkout@v2
      - uses: Bhupesh-V/update-pr-reminder-action@main

In the above workflow we are telling github to “watch” the main and dev branches of our repository for any new pushes and run the corressponding action. So what’ happening behind the scenes at update-pr-reminder-action?

We use a simple shell script combined with github’s CLI tool, gh and some some shell wizadry ✨

#!/bin/bash

all_open_prs=$(gh pr list --base "${GITHUB_REF#refs/*/}" --json author,number)
printf "%s\n" "PRs with base ${GITHUB_REF#refs/*/}: $all_open_prs"
prs_count=$(echo "$all_open_prs" | jq length)
printf "%s\n" "There are currently $prs_count open PRs"

for (( c=0; c<$prs_count; c++ )); do
    pr_id=$(echo "$all_open_prs" | jq .["$c"].number)
    author=$(echo "$all_open_prs" | jq .["$c"].author.login | tr -d '"')
    printf "%s\n" "Author for PR $pr_id is $author"
    gh pr comment $pr_id --body "Hey @$author 👋🏽 friendly reminder to update your PR/branch because there was a recent commit ($(git rev-parse HEAD)) to the base branch"
done

The first thing we need to do is find the current branch, which is done by ${GITHUB_REF#refs/*/} where GITHUB_REF is an environment variable which is set automatically in every github action

Next we find all the open PRs open using the gh tool

gh pr list --base main --json author,number

Note: gh needs to authenticate before you can use it, we have already done this by setting the env GITHUB_TOKEN. gh uses this token to make any API requests

This will return a json output of all the open PRs like this

[
  {
    "author": {
      "login": "Bhupesh-V"
    },
    "number": 400
  },
  {
    "author": {
      "login": "some-dev"
    },
    "number": 402
  },
  ...
 ]

To parse json, we are using jq here,

# get pull request id
basename "$(echo "$all_open_prs" | jq .["$c"].number)"

Creating a comment is easy,

gh pr comment $pr_id --body "Comment body"

If you see the comment body in our action, we are also referencing the latest commit to base branch using git rev-parse HEAD

Hey @$author 👋🏽 friendly reminder to update your PR/branch because there was a recent merge ($(git rev-parse HEAD)) to the base branch

TODO

To reduce spam in case a pull request takes longer to get merged, we can also check if the branch is up to date by check if the base branches latest HEAD is present inside the authors PR

Closing Thoughts

Ideally this feature should exist alongside scheduled reminders nonetheless I believe its a good enough solution for now.

Resources

A bunch of resources that helped me write this workflow

What's the best risk you have taken?

Bhupesh Varshney 👾 — Sun, 10 Jul 2022 08:59:04 +0000

People talk about taking risks in life. Would love to hear what's a risk that turned to be good for you eventually?

If I could change one thing about tech interviews

Bhupesh Varshney 👾 — Sat, 09 Jul 2022 08:32:10 +0000

So I have been interviewing for a while for my next role in tech. I have given all type of interviews by now. Take home assignments, DSA, solution write-ups you name it. I am fine with all kinds of interviews, honestly at this moment I don’t even care about what style of interviews are good or bad.

What I would like to see a change in our ecosystem is to GIVE FEEDBACK AFTER INTERVIEWING. Yes I am screaming because I can’t even believe this simple and most minimal amount of decency is not followed anywhere, and nobody seems to give a shit about it?

I say forget about giving feedback at application stage, but what about when a candidate sits in your 3-round and 1-hour long tech interviews & you don’t even bother to say why they were rejected? Is this what our industry have come down to?

We talk about upskilling and continuous learning from candidate, but is your company upskilling & learning from your interview taking skills? Or is it always about filling the funnel & hiring the best-of-best?

One personal anecdote

Even among all this non-sense & chaotic style of interviewing, I happen to have one of my best interviewing experience with deepsource.

3 rounds of interview:

Intro call about my work & what I do?
Code Pairing session (sort of pair-programming) where we built something meaningful in 1.2 hrs.
A traditional system design round.

Even though I got rejected, but upon asking for feedback this is what I got in reply.

I whole heartedly agree 💯 with both the points, My system-design skills have always been in a gray area and I try to improve them form time-to-time. Atleast now I have some concrete proof that I need to work in that area.

Some polite feedback for Senior Engineers/Managers/CTOs/Founders

For folks who are in-charge of taking interviews. How do you communicate to the HR that this candidate is not good enough? You obviously have some good reasons right? You know them at the back of your head, what’s stopping you from writing one 2 line paragraph about it?

Aren’t we told that a “good software engineer” has good communication skills apart from coding skills. If that’s the case take a step back and ponder how you folks communicate “during” and “after” the interview process is over. Can you try to improve it?

If you say you don’t have time to right down your thoughts because of large number of interviews, Or maybe back-to-back interviews or maybe tight deadlines etc. Here are some minimal tips you could follow

(Full disclosure: I am not an interviewer, never have I ever interviewed anyone but I believe this is how would I go about doing it).

Reserve some 5-10 minutes before the interview to go through candidate’s resume/portfolio. Get an understanding what do they write about themselves, read every resume as a short story, this would probably take 5 mins max.
During the interview when they are busy solving problems (and there is a awkward silence), right your understanding of the candidate.
After the interview is over, and you have made your mind not to move forward with them, Take 2 mins to write what caused you to make that decision and ask your HR to “copy-paste” and send it to the candidate with the obvious “rejection-template-copy”.

That’s all we ask. Literally how hard is it?

Advice for candidates

Just keep giving your best in all rounds. One thing I have learned recently is that even if everything goes great from your end, there is always a probability of you not getting selected. It’s not a fair game (it never will be). Just don’t give up and keep learning. Hope you get that job 👍

Adios 👋

Questions to ask in an interview as an interviewee

Bhupesh Varshney 👾 — Mon, 20 Jun 2022 05:51:38 +0000

Hey folks, If you are actively interviewing, here are a bunch of questions you can ask as an interviewee.

What’s the biggest priority for the team right now?
Why is this role open?
What’s the biggest challenge for someone stepping into this role?
How does the org structure on the team work?
What a typical day is like?
Is the product live in production? If not, what's the schedule for developing it? (Particularly valid for early stage & stealth startups)
How often are releases done?
Do you have a favorite part of the job? Least favorite?
Budget for conferences?
Management style/structure? Frequent catch-ups aka one-on-ones? Something else?
Learning opportunities?
Ask about last employee who left, what was the reason?
How managers/senior folks help coach employees through a weakness.
How often feedback is given to you and collected from you.

Resources

Interview Questions to Ask Your Interviewer

Feel free to share more questions below

Writing like a pro with vale & neovim

Bhupesh Varshney 👾 — Thu, 26 May 2022 00:00:00 +0000

Vale is a syntax-aware prose linter built for all you writers out there. With more than 100 releases so far vale is 5 year old project and is used by writing folks in companies like Google, Microsoft, IBM, RedHat to name a few. I have recently started to use vale in my everyday writing workflow and it has a significant impact on what words I choose to convey ideas. I mostly use neovim for writing, so we will be covering how to setup vale and use it with neovim.

Just like when writing software we use static analysis tools to find common problems, vale aims to help writers configure what words/prose to choose while writing technical documentation.

Installing `vale`

Download a latest version of vale from their github releases page.

Writing styles & the vale config

Vale requires to have a .vale.ini config file located either in your $HOME directory or a relative project directory. Below is a sample configuration that I use personally

# This goes in a file named either `.vale.ini` or `_vale.ini`.
StylesPath = styles
MinAlertLevel = suggestion

# External packages
Packages = Google, Readability, alex, proselint, IBM
# Only Markdown and .txt files; change to whatever you're using.
[*.{md,txt}]
# List of styles to load.
BasedOnStyles = alex, proselint

Once you create this file, run vale sync to download/update any External packages you have mentioned in the config.

A vale style is a repository of “rules” that define what words need to be reported by vale. The rules are defined in a yml file.

If you prefer to customize the config yourself, I recommend using the config generator

The vale project has come up with 7 style packages. Some of the popular styles are based on writing rules from organisations like Google & Microsoft, some of which are listed below:

You can find more vale styles on github by vale-style topic.

Choosing what styles suits best for your writing needs depends on your writing style, If you are working with a team of writers in a organisation I would suggest using styles from Google & Microsoft which are more strict but good for teams. If you have a personal blog or you write solo, use styles like alex or proselint whose rules are more lenient, I recommend give all the rules a try by switching the BasedOnStyles property in vale config.

Setting up Neovim

The null-ls.nvim neovim plugin let’s you use vale as a prose linter.

Install null-ls.nvim using Plug

Plug 'jose-elias-alvarez/null-ls.nvim'

Require the lua plugin,

require("null-ls").setup({
    sources = {
        require("null-ls").builtins.diagnostics.vale,
    },
})

If you are using a native vim config use lua HERE doc.

lua << EOF
require("null-ls").setup({
    sources = {
        require("null-ls").builtins.diagnostics.vale,
    },
})
EOF

Make sure you have a .vale.ini or _vale.ini somewhere in $HOME or at the current working directory.

Here is how the in-editor suggestions look like

As you can see, some of the suggestions are not being rendered by NeoVim correctly. To fix this, you can use the trouble.nvim plugin to get a nice VSCode like interface for diagnostics.

And that’s it, enjoy writing!

Find pull requests that modify a file path in the terminal

Bhupesh Varshney 👾 — Fri, 22 Apr 2022 06:39:15 +0000

Ever wondered how others are changing the same file you are working on? It can give insights into what a particular piece of code will look like in the future. Certainly, it can also help OSS newcomers to identify how others are building a particular feature

Well, today’s post is to solve just that. Let’s build a simple shell script for ourselves, shall we?

Prerequisites

Before we start working on anything, make sure you have the following CLI tools installed

Once you install gh, make sure to authenticate yourself using gh auth login

Gimme pull requests

Our first job is to ask the user to choose a local file path to look for. This can be done by parsing git ls-files, for better interactivity we are going to use fzf as well

git_file=$(git ls-files | fzf \
  --prompt="Choose File: " \
  --height 40% --reverse \
  --header="Choose a file to find pull requests on github that modify it"
)

Once that’s settled we need to find all the open pull requests, this is done by gh pr list

fetch_prs() {
  pr_fetch_limit=100
  printf "%s\n\n" "Hold tight while we look for PRs ✋👀 that modify $git_file"
  all_open_prs=$(gh pr list --limit $pr_fetch_limit --json number,title,url,files,author,baseRefName,headRefName)
  total_prs=$(jq length <<< "$all_open_prs")
}

By default, gh only loads 30 open pull requests. For now, we have increased that limit to 100, later we will fix this by giving user the control on this value.

If you execute this above code, we will get an output similar to this

[
    {
      "author": {
        "login": "authorUsername"
      },
      "baseRefName": "somebranch",
      "files": [
        {
          "path": ".gitmodules",
          "additions": 2,
          "deletions": 2
        },
        ...
      ],
      "headRefName": "dev",
      "number": 300,
      "title": "my new feature",
      "url": "https://github.com/user/repo/pull/300"
    },
    ...
]

Notice, we get all the files changed in a PR inside the files array. We can simply loop over all the elements of this JSON array to filter our git_file

find_files_prs(){
  for (( pri=0; pri<total_prs; pri++ )); do
      readarray -t changed_files < <(echo "$all_open_prs" | jq .["$pri"].files[].path)

      if [["${changed_files[*]}" =~ ${git_file} ]]; then
          pr_branch=$(jq -r .["$pri"].headRefName <<< "$all_open_prs")
          base_branch=$(jq -r .["$pri"].baseRefName <<< "$all_open_prs")

          printf "%s " "$(jq -r .["$pri"].title <<< "$all_open_prs") ($pr_branch ➜ $base_branch)"
          printf "%s\n" "by $(jq -r .["$pri"].author.login <<< "$all_open_prs")"
          printf "%s\n\n" "PR Link: $(jq -r .["$pri"].url <<< "$all_open_prs")"

          prs_count=$((prs_count+1))
      fi
  done

  if [[$prs_count == 0]]; then
    printf "%s\n" "Oops!, No pull requests found that modify this file path"
  else
    printf "%s\n" "Found $prs_count open pull requests that modify $git_file"
  fi
}

A bunch of things are going on in this piece of code

We convert our JSON array of changed files path to a bash array changed_files.
We loop over each path to see if it matches the user-provided path or not.
Furthermore, we output any necessary info we need to, like PR link, base and target branches, and author username.

Wait something’s missing right 🤔? What about diffs? The gh pr list command doesn’t provide the diff data, although we can use a different command gh pr diff to get diff for a particular PR

show_diff=0

get_diff() {
  if [[$show_diff = 1]];then
    printf "%s\n" "Getting diff for PR #$2"
    # filepath with escaped slashes
    clean_filepath=${1//\//\\/}
    gh pr diff --color always "$2" | sed -n "/diff --git a\/$clean_filepath/d;/diff/q;p"
    printf "\n"
  fi
}

show_diff is going to be a global flag that we are gonna control through script arguments. More on that later note that we need to show diff for only 1 file path, which the user provided. To do that we have used sed to get everything between the pattern 1,/diff --git a/<filepath> and diff which marks the diff for the next file change.

It’s time to put everything together. Let’s add some driver code to execute our script correctly.

while getopts "dl:" o; do
    case "${o}" in
        d)
            d=${OPTARG}
            if [["$d" == ""]]; then
              show_diff=1
            fi
            ;;
        l)
            l=${OPTARG}
            ;;
        *)
            printf "%s\n" "You seem to be lost" && exit
            ;;
    esac
done

if [[$l != ""]];then
  pr_fetch_limit=$l
  printf "%s\n" "Using $pr_fetch_limit as PR fetch limit."
else
  printf "%s\n" "Using $pr_fetch_limit as default PR fetch limit. Pass the -l flag to modify it"
fi

fetch_prs
find_files_prs

Demo & Source

The source is available here if you want to reference or hack anything. I took the liberty to beautify the output from our script. Here is how the finalized version looks like:

To-dos & Takeaways

There are a bunch of things that can take this script to the next level

We need to provide a permalink to the file change inside a PR, although it's limited by GitHub’s API.
Support for fetching closed/merged PRs as well using gh pr list --state all or better asking the user what they want.
Although we were able to parse out diff for a single file path, I believe it’s a very nasty & error-prone way to do it and ideally, this feature should exist within gh. I have created an issue to track this, let’s see where it goes.

What are some things that you don't like about git?

Bhupesh Varshney 👾 — Mon, 18 Apr 2022 17:48:54 +0000

Git has been in the making since 2005, and probably 90% developers use it everyday
What are some pain-points that you face with git everyday or have encountered occasionally?

Leaving the classic, Hitler uses git meme for some gags:)

One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else

Bhupesh Varshney 👾 — Mon, 07 Mar 2022 13:59:41 +0000

I see a lot of folks wandering over the internet asking how to contribute to open-source,
To my disappointment none of the articles, I have read mention what actionable steps should be taken to make your first-time contrib.

Although I see people giving one generic advice like, "choose a package you already work with and contribute there", this still doesn't solve your problem because you might be using packages that are too big, working with these big projects can be sort of overwhelming.
So to make it easier I am gonna share one secret cheat trick with you that is everlasting

Secret 🤫

Ready? here it is ...

"solve static analyzer issues"

Yes! I am serious here, Every language you work with nowadays has support for tools that do static analysis of code and point out issues. Helping fix these issues is a task everyone sort of ignores or forgets because building features and resolving bugs is much more good (and important to be honest).

But to have a "head start" in the game, trust me fixing these minor things still counts as a valuable OSS contribution.
Moreover, working on these "code quality cleanup" issues is better than "fixing typos" or waiting around for a "good-first-issue" to open up, right?

Cool, you have my attention but how?

Any good OSS project also has a bunch of these tools already set up.

Go ecosystem has Go Report Card
Projects based on shell scripting can use shellcheck
Python has a bunch of tools like black, pylint, flake8.
Javascript has ESLint
Tools like deepsource has support for finding security related issue for different language stack (which is again just building and finding patterns via static analysis of code)
Another tool is CodeQL that detects security issues and gives a grade to the project

Here is a demo run of lgtm on pachyderm

Run and use these tools over the project you like to contribute to, the probability is there are going to be a bunch of issues reported.

📝 Pick the ones that are more common in frequency,
⚠️ report these issues to the project,
👀 wait for the maintainers to acknowledge it
🙌🏻 ask to assign the issue to yourself

Filter out issues that can have a positive impact on code quality, say for e.g replacing single quotes with double quotes is not a good issue in comparison to fixing potential unused variables.

If everything works out as expected, congrats! you just took a shortcut to FOSS contribution by opening an issue for yourself, now all you need to do is fix these errors which by now you have already researched about.

Although do note that using these tools can be sort of a hit-trial approach in finding analyzer warnings that matter. I will say don't misuse this trick for contributions by spamming the repo with them, play the game right

So what are you waiting for, surf GitHub and send that PR right now!!

This post was originally shared on my newsletter, subscribe for more such interesting tips

View a python function's history over-time with Git & FZF

Bhupesh Varshney 👾 — Mon, 13 Dec 2021 10:14:17 +0000

I am kinda hooked up on modifying my git terminal workflow and integrate FZF wherever I can, this is such an example script to see changes in a python function overtime in your git history. And yes, its interesting (& stupid)

#!/usr/bin/env bash

# FZF Wrapper over git to interactively search code changes inside functions

readarray -t choices < <(git ls-files | fzf \
  --prompt="Choose File: " \
  --height 40% --reverse \
)

printf "%s\n" "$(grep -o -P '(?<=def ).*?(?=\()' $choices)" | fzf \
--ansi --preview "echo {} | xargs -I{} git log --color -L :{}:$choices" \
--prompt="Choose function/method: " \
--bind 'j:down,k:up,ctrl-j:preview-down,ctrl-k:preview-up,ctrl-space:toggle-preview' --preview-window right:60% \

The actual trick is finding all python functions and methods names in a given file using grep

$ grep -o -P '(?<=def ).*?(?=\()' rich/columns.py
__init__
add_renderable
__rich_console__
iter_renderables

Next we plug the function name with filename in this git command

git log -L :funciton_name:relative_file_path

Forem: Bhupesh Varshney 👾

How I reduced the size of my very first published docker image by 40% - A lesson in dockerizing shell scripts

The very first Dockerfile ~attempt~

Path to optimization - reducing image size by 40%

2nd attempt - alpine on alpine

Looks like xargs and awk came free?

3rd attempt - Using scratch at 2nd stage

Identifying transitive dependencies

Primer on shared libraries

Shebangs #! are useless

A look at terminfo db

Everything needed to run ugit

Could we reduce the size further?

Reason 1: Pin minimum version of fzf?

Reason 2: Use the latest bash features?

You didn’t try docker-slim?

You didn’t try docker-squash?

Learnings

Acknowledgements

Resources

Using multiple git user configs with credentials store

Step 1: Separate Work/Personal directories

Step 2: Separate git config for each account

Step 3: Update global .gitconfig to switch the profile based on directory

Step 4: Register Access Tokens

How to stop having disappointing moments in life

What to do then?

Automatically remind contributors to update their pull requests

How to fix?

TODO

Closing Thoughts

Resources

What's the best risk you have taken?

If I could change one thing about tech interviews

One personal anecdote

Some polite feedback for Senior Engineers/Managers/CTOs/Founders

Advice for candidates

Questions to ask in an interview as an interviewee

Resources

Writing like a pro with vale & neovim

Installing vale

Writing styles & the vale config

Setting up Neovim

Find pull requests that modify a file path in the terminal

Prerequisites

Gimme pull requests

Demo & Source

To-dos & Takeaways

What are some things that you don't like about git?

One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else

Secret 🤫

"solve static analyzer issues"

Cool, you have my attention but how?

View a python function's history over-time with Git & FZF

The very first `Dockerfile` ~attempt~

2nd attempt - `alpine` on `alpine`

Looks like `xargs` and `awk` came free?

3rd attempt - Using `scratch` at 2nd stage

Shebangs `#!` are useless

A look at `terminfo` db

Everything needed to run `ugit`

Reason 1: Pin minimum version of `fzf`?

You didn’t try `docker-slim`?

You didn’t try `docker-squash`?

Step 3: Update global `.gitconfig` to switch the profile based on directory

Installing `vale`