Forem: Raunak Ramakrishnan

Level up your S3 skills by playing this game!

Raunak Ramakrishnan — Thu, 04 Jun 2020 16:47:33 +0000

The best way to learn a skill is through deliberate practice!
S3 Game is an awesome game made by Vasily Pantyukhin which walks you through various features of S3 from beginner to fairly advanced.

What you will need:

A browser (duh!)
aws cli

CLICK HERE TO PLAY!

Each level consists of finding a "treasure" object and getting to the next level using the secret code in the "treasure".

Go ahead and play this game. Each level contains multiple hints with links to S3 API documentation.

Below is a walk through you can refer to if you are stuck on any level. Make sure that you try for a bit before seeing here :)

Walk through (Spoilers)

Level 2

Solution:

Just get the treasure2 object either using browser or aws s3 : https://s3game-level2.s3.us-east-2.amazonaws.com/treasure2
Make sure to save the Access and Secret key somewhere

Level 3

Create an AWS profile "tmp" with the access and secret key obtained from previous level. You will need to use this for almost every level from here on.

aws configure --profile tmp
> AWS Access Key ID
> AWS Secret Access Key
> Default region name - us-east-2

Solution:

aws s3 ls s3://s3game-level3 --profile tmp
aws s3 cp s3://s3game-level3/treasure3_has_no_secret_code . --profile tmp
cat treasure3_has_no_secret_code
# The object data itself does not have the secret code. It is present in the metadata
aws s3api head-object --bucket s3game-level3 --key treasure3_has_no_secret_code --profile tmp

Level 4

Hint

Solution:

# List the bucket ..
# get the file
aws s3 cp s3://s3game-level4-k73045aztqln/treasure4_also_has_no_secret_code . --profile tmp
# looks like the data is empty
aws s3api head-object --bucket s3game-level4-k73045aztqln --profile tmp --key treasure4_also_has_no_secret_code
# ... so is the metadata. Let's try object tags as mentioned in hint
aws s3api get-object-tagging --bucket s3game-level4-k73045aztqln --profile tmp --key treasure4_also_has_no_secret_code

Takeaway:

Object in S3, including its metadata, is immutable. When you edit object's metadata, you are actually overwriting the object with a copy of itself, with its metadata modified.
In contrast, tags are subresources. They are managed separately and can be modified without modifying the object itself.

Level 5

Solution

# We are listing all versions of objects as the hint says that the object is not present in its current version
aws s3api list-object-versions --bucket s3game-level5-8v95e5rv7z4i --profile tmp
# Get the specific version of treasure5_is_deleted using version-id from previous call
aws s3api get-object --bucket 's3game-level5-8v95e5rv7z4i' --key treasure5_is_deleted --version-id '344PQOyFqocF0TI66MbLynNNdQqHfBz3' --profile tmp treasure5_is_deleted

Takeaway

S3 versioning keeps multiple variants of an object in the same bucket.
If you enable versioning for a bucket, S3 automatically generates a unique version ID for the object being stored.
In one bucket you can have two objects with the same key, but different version IDs.
Check if versioning is enabled using: aws s3api get-bucket-versioning --bucket my-bucket

Level 6

Hint

Solution

# We need to get a gzipped object using S3 select
aws s3api list-objects --bucket s3game-level6-vjv45x1gux81 --profile tmp
# Now, do the select on s3select.csv.gz
aws s3api select-object-content --bucket s3game-level6-vjv45x1gux81 --key s3select.csv.gz --expression "SELECT s.Answer FROM s3object s WHERE Category = 'TREASURE'" --expression-type sql --input-serialization '{"CSV": {"FileHeaderInfo": "USE", "FieldDelimiter": ";"}, "CompressionType": "GZIP"}' --output-serialization '{"CSV": {}}' --profile tmp treasure6

Level 7

Hint

Solution

# Get 'somethingstrange' from the bucket
aws s3api get-object --bucket s3game-level7-zhovpo4j8588 --key somethingstrange --profile tmp t7
cat t7
# Visit the pre-signed URL in the file to get URL for next level

Takeaway

Pre-signed URLs are a powerful feature of S3. From the docs:

A presigned URL is generated by an AWS user who has access to the object. The generated URL is then given to the unauthorized user. The presigned URL can be entered in a browser or used by a program or HTML webpage. The credentials used by the presigned URL are those of the AWS user who generated the URL.

A presigned URL remains valid for a limited period of time which is specified when the URL is generated.

Level 8

Solution

# List objects
aws s3api list-objects --bucket s3game-level8-v6g8tp7ra2ld --profile tmp
# Visit the treasure file using the cloudfront URL
curl 'https://d2suiw06vujwz3.cloudfront.net/treasure8_CDN'

Takeaway

You can easily create a Cloudfront distribution backed by an S3 bucket
In such cases, you can restrict public access for the objects in the bucket as they can be accessed using Cloudfront
Ensure that your bucket does not allow unprivileged users to list all objects in the bucket

Level 9

Hint

The bucket has a policy which checks that any request for "arn:aws:s3:::s3game-level9-781xtls2quvy/treasure9_referer" has the referer string "http://s3game.treasure"

Solution

All we need to do is curl with a --refer given in policy
curl 'https://s3game-level9-781xtls2quvy.s3.us-east-2.amazonaws.com/treasure9_referer' --refer 'http://s3game.treasure'

Level 10

Here, we need to fetch an object which is in Infrequent Access storage.

Solution

# We can use the powerful query param of aws cli to just fetch those objects which are in infrequent access storgae
aws s3api list-objects --bucket s3game-level10-gac6tf83erp6 --query 'Contents[?StorageClass == `STANDARD_IA`]' --profile tmp

Takeaway

Amazon S3 offers a range of storage classes designed for different use cases:

Standard for general-purpose storage of frequently accessed data

Intelligent-Tiering for data with unknown or changing access patterns

Standard-Infrequent Access (Standard-IA) and One Zone-Infrequent Access (One Zone-IA) for long-lived, but less frequently accessed data

Glacier and Glacier Deep Archive for long-term archive

Level 11

Here we need to fetch an object which has been encrypted using client side encryption and uploaded to S3. This means that we can not fetch the object unless we have the encryption key.

Solution

# The encryption key is given in the hint
aws s3 cp --sse-c 'AES256' --sse-c-key 'UkXp2s5v8y/B?E(H+MbPeShVmYq3t6w9' 's3://s3game-level11-djq30a807iyq/treasure11_encryption' --profile tmp .
# Alternate way using aws s3api. Here we also need to give md5 of encryption key as additional integrity check
aws s3api get-object --bucket s3game-level11-djq30a807iyq --key treasure11_encryption --sse-customer-key 'UkXp2s5v8y/B?E(H+MbPeShVmYq3t6w9' --sse-customer-algorithm AES256 --profile tmp treasure11_encryption

Takeaway

S3 supports both server side and client side encryption.

VICTORY! - Level 12

Congrats! You have cleared the game!

Practical Python Programming - Online course

Raunak Ramakrishnan — Sat, 30 May 2020 14:47:49 +0000

Course Details:

Folks, the legendary David Beazley has released a free, online course on practical python programming! It was initially content for an in-person training, now online and free for everyone.

Course page: https://dabeaz-course.github.io/practical-python/Notes/Contents.html

About the author:

David Beazley is a very well known member of the Python community. His PyCon talks are extremely interesting and teach you a lot about the workings of Python and building some incredible stuff in 3 hours.

Some of my favorite talks:

Keynote: A Talk Near the Future of Python

Live code a web-assembly interpreter in one hour!

Python Concurrency From the Ground Up: LIVE!

A live-coding session exploring various concurrency models in Python

Keynote: Builtin Superheroes

Solve common data manipulation problems using Python's built-in types.

Lambda Calculus

Learn lambda calculus and derive the powerful Y-combinator using Python

5 NPM and Node tips to make your dev machine safer

Raunak Ramakrishnan — Thu, 28 May 2020 13:39:19 +0000

NPM is used as a convenient cross-platform package manager for a lot of developer tools. For many tools, the defacto way to install is npm install -g $TOOL. But installing anything via npm allows it to run untrusted code on your machine.

Here are a few tips to minimize the risk:

1. NEVER run npm as sudo/root

Node's official documentation recommends not installing global packages as sudo/root. If you have already installed node through nvm ignore this step. If you use a system installed node e.g using Ubuntu's apt-get, read through this guide for Linux/Mac or npm-g-nosudo which is a shell script for Linux.

If you are lazy (like me), here's a summary from the linked guide:

mkdir -p "${HOME}/.npm-packages" && npm config set prefix "${HOME}/.npm-packages"

Add this to your .bashrc or .zshrc:

NPM_PACKAGES="${HOME}/.npm-packages"
export PATH="$PATH:$NPM_PACKAGES/bin"
# Preserve MANPATH if you already defined it somewhere in your config.
# Otherwise, fall back to `manpath` so we can inherit from `/etc/manpath`.
export MANPATH="${MANPATH-$(manpath)}:$NPM_PACKAGES/share/man"

2. Install/Use node using `nvm`

Node's official documentation recommends installing node as an unprivileged user using a node version manager.

Here are the steps:

Install nvm. This does not work for Windows, use nvm-windows which is a totally different project from nvm.
- Unfortunately, nvm suffers from the curl pipe to bash install as its main way to install.
- If you have git installed on your machine, you can also directly clone it using the following steps:
- cd "$HOME" && git clone https://github.com/nvm-sh/nvm.git .nvm
- cd "$HOME/.nvm" && git checkout v0.35.3 && . nvm.sh
- Add to your .bashrc or .zshrc:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

nvm install --lts or whichever version of node you need.
nvm use node
To use a node version by default, echo "lts/*" > "$HOME/.nvmrc"

3. List all your globally installed npm packages and remove any unnecessary ones

Listing global packages is done with npm ls -g --depth 0
Remove any unnecessary packages with npm uninstall -g $TOOL
You can check your shell history/scripts folders to see if you actually use many of the global packages.

4. Audit your packages

Run npm audit in your project regularly to see if any dependencies are vulnerable. This only works for your development projects, not global packages.

A (hacky) way to `npm audit` global packages:

Run npm. The last line will show the folder of the global npm packages e.g $HOME/.npm-packages/lib/node_modules/npm
Go to $HOME/.npm-packages/lib and run npm init -y and then run npm i --package-lock-only. These steps are required as npm audit checks for package.json and package-lock.json
Now run npm audit. Remove any dependencies which have high or too many moderate vulnerabilities. These could either be malicious or unmaintained tools.
Remove the package.json and package-lock.json after the audit

5. Use `npx` to run executables

Many times, it is not necessary to have a tool installed globally e.g create-react-app. Node 6+ comes with npx which allows you to temporarily download and run scripts. The benefits of using npx over npm install -g are beautifully explained in this post.

A gist of the article:

Calling npx when isn’t already in your $PATH will automatically install a package with that name from the npm registry for you, and invoke it. When it’s done, the installed package won’t be anywhere in your globals, so you won’t have to worry about pollution in the long-term.

This feature is ideal for things like generators, too. Tools like yeoman or create-react-app only ever get called once in a blue moon. By the time you run them again, they’ll already be far out of date, so you end up having to run an install every time you want to use them anyway.

Bonus : Use Deno for developer tools

I think a project like deno with good sandbox capabilities and limited permisisons is the best choice for a lot of developer tools currently made using Node. Read this post for more details on why everyone is talking about Deno.

Conclusion

Does this solve all security issues of node/npm on your machine? No! There are a lot of ways in which a malicious attacker can compromise your machine/project. This only prevents a bad npm tool from getting root access and removes globally installed npm tools with known insecure dependencies.

Webhook to auto-deploy on git push to Github

Raunak Ramakrishnan — Mon, 25 May 2020 18:13:21 +0000

What is a webhook?

A webhook is an endpoint on your server which allows you to execute a particular task. Webhooks are usually triggered by some event. A good use-case for a webhook is running tests on a dedicated test server or deploying your latest master branch to staging/production.

Github / Gitlab / Bitbucket allow you to specify a webhook URL in your repository settings. Github triggers the webhook which sends the event data on every push.

Webhook server

Webhook is a very useful golang project which runs any script you specify when a particular endpoint is hit.

Download and extract the binary for your operating system from the releases page. For Linux, it is here.

The program takes as config a hooks.json file:

[
  {
    "id": "hello-world",
    "execute-command": "/home/user/scripts/hello.sh",
    "command-working-directory": "/home/user/webhook"
  }
]

Replace user with the username of your linux user.

The hello.sh script.

#!/bin/bash
echo 'Hello!'

Make the script executable by running chmod +x hello.sh

Start webhook server as webhook -hooks hooks.json -hotreload -logfile webhooks.log. The server will run on port 9000 by default. You can check if everything is working by running curl http://localhost:9000/hooks/hello-world. This will print "Hello!" in the log file.

Deploy script

For the purpose of this post, I'll assume the script is called deploy and is at location /home/user/scripts/deploy. This script will vary depending on your tech stack and the complexity of your CI process.

A simple example deploy script:

#!/bin/bash
# If you have a build server which creates binary/jar/artifact
wget 'ARTIFACT_URL'
# Else, git pull and build on the server itself
# Assuming this script stops old instance of your code and starts a new instance with latest artifact 
restart-service.sh

Configuration to run deploy script

[
  {
    "id": "deploy-from-git",
    "execute-command": "/home/user/scripts/deploy",
    "command-working-directory": "/home/user/scripts",
    "trigger-rule":
    {
      "and":
      [
        {
          "match":
          {
            "type": "payload-hash-sha1",
            "secret": "MyTotallySecretString",
            "parameter":
            {
              "source": "header",
              "name": "X-Hub-Signature"
            }
          }
        },
        {
          "match":
          {
            "type": "value",
            "value": "refs/heads/master",
            "parameter":
            {
              "source": "payload",
              "name": "ref"
            }
          }
        }
      ]
    }
  }
]

The trigger-rule in config above will ensure that the script is only triggered when header from Github request contains "X-Hub-Signature" with a secret string and the push has occured in master branch.

Make sure that the secret string ("secret" : "MyTotallySecretString") is randomly generated. This secret will need to be entered in Github settings as well.

For Gitlab and Bitbucket, example hook config can be found on repo page here

Expose your webhook server safely to the internet

There are 2 ways of exposing the webhook server to github:

Proxy using Nginx
Via a tunnel e.g by downloading ngrok and then running ngrok http 9000

Nginx configuration

Preferably use HTTPS for your domain with Nginx. A good tutorial here.

Example Nginx config (HTTPS):

upstream webhook {
    server localhost:9000;
}

server {
    listen 443 ssl http2;
        server_name YOUR.DOMAIN.COM;
    ssl_certificate YOUR_CERT_CHAIN; # e.g /etc/letsencrypt/live/DOMAIN/fullchain.pem;
    ssl_certificate_key YOUR_CERT_KEY; # e.g /etc/letsencrypt/live/DOMAIN/privkey.pem;
    include /etc/nginx/options-ssl-nginx.conf;
    ssl_dhparam /etc/nginx/ssl-dhparams.pem;
        location ~ ^/hooks/(.+)$ {
        proxy_pass http://webhook;
    }
}

Add your webhook URL to Github

Go to the settings page of your Github repo then click on Webhook. Enter the URL of your webhook server. If using Nginx, it should be something like https://YOUR.DOMAIN.COM/hooks/deploy-from-git. Make sure you select content type as application/json and secret to the secret you generated earlier.

Bonus: Create a systemd user service for webhook (Linux)

Create a systemd unit file with path /home/user/.config/systemd/user/webhook.service . This service does not require sudo/root permissions and can be run by the unprivileged user.

[Unit]
AssertPathExists=/home/user/scripts

[Service]
WorkingDirectory=/home/user/scripts
ExecStart=/home/user/scripts/webhook -hooks hooks.json -hotreload -logfile webhooks.log
Restart=always
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=default.target

Do systemctl --user daemon-reload and systemctl --user start webhook.service. You can systemctl --user enable webhook.service to ensure that the service always runs when your machine is booted.

Bash function to add TILs from the command line

Raunak Ramakrishnan — Sat, 23 May 2020 12:42:20 +0000

Here's a snippet I use in my .bashrc file to quickly write down and review TILs (Today I learned):

TIL_FILE="$HOME/my-notes-repo/til.md"
til () 
{ 
    # checks if the function has been called without any argument
    if [[ -z $1 ]]; then
        # opens the file in my editor, setting the cursor to the last line
        # useful to review latest entries or to write a longer entry
        vim + "$TIL_FILE";
    else
        # adds a line with today's date, a TAB and all arguments supplied ("$@")
        echo -e "- $( date '+%F' ):\t$@" >> "$TIL_FILE";
    fi
}

How to use:

til to open the file. I use this to review what I learned today or if I need to write a longer, multiline entry
til CONTENT to append a line to the file
- e.g til grep --line-buffered to immediately print especially when tailing files will add this: - 2020-05-23: grep --line-buffered to immediately print especially when tailing files

Explanation on the function

In case you missed the comments in the function, here's an explanation of what the various lines do:

if [[ -z $1 ]]; checks if the function has been called without any arguments
vim + "$TIL_FILE" opens the file in vim (my preferred editor), setting the cursor to the last line of the file.
echo -e "- $tdate:\t$@" >> "$TIL_FILE"; adds a line with today's date, a TAB and all arguments supplied ("$@") to til

Setting display brightness on Linux from the command line

Raunak Ramakrishnan — Tue, 19 May 2020 16:53:48 +0000

I auto-adjust the brightness on my display using a cron which runs at 7 pm everyday.

xrandr --output eDP-1 --brightness 0.9

This sets the brightness to 90% of maximum brightness.

Working with Linux Display environments in a cron

The above command works when run from my terminal. But when run on a cron, it does not produce any effect. We need to explicitly mention the "display number" as an environment variable DISPLAY.

8 19 * * * DISPLAY=:0 xrandr --verbose --output eDP-1 --brightness 0.9

How to get the display number of your monitor?

Running xrandr will give you a lot of output like

Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 16384 x 16384
eDP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis)

In my case, the display number is :0

Adjusting backlight brightness vs perceived brightness

If xrandr is unable to find backlight, it only changes "perceived" brightness i.e it does software color correction. If we want to save battery, we will need to reduce the backlight brightness.

In Linux, the backlights can be found in /sys/class/backlight. In my case, it was /sys/class/backlight/intel_backlight. In this folder, there are many files like

actual_brightness
max_brightness
brightness

max_brightness shows highest possible level of brightness for the display. We can adjust the value in brightness file to reduce backlight brightness.
Here's what I do to adjust it: echo 1800 > /sys/class/backlight/intel_backlight/brightness (in root crontab)

Unlike the xrandr command, this does not require setting any DISPLAY variable in the cron file.

Other tools

brightnessctl is an easy to use tool which integrates well with systemd. It is also available in the package managers of various distributions like Debian, Ubuntu, Arch Linux.

Git Tagging Tutorial

Raunak Ramakrishnan — Thu, 22 Nov 2018 11:39:51 +0000

I read a post on dev.to which shows how to create git tags using GUI-based git clients. I think that tags are useful to know even when using the git cli.

What are tags

Tags are specific points in your code history which are useful to re-visit later e.g
you just released a new version of your app. You can tag the commit as v1.0 using git tag v1.0. Anytime you want to reproduce bugs encountered on that version, simply do git checkout v1.0 and investigate.

How to use `git tag` better

Checkout code to the tag

The tag is linked to the specific commit and not to a branch. When you checkout the tag, git tells you that you are in "detached HEAD" state. Do not worry, all it means is that you need to create a new branch if you want to retain any changes you make after checking out the tag.

Create a new branch exactly at the commit of the tag using git checkout -b BRANCH_NAME TAG_NAME

Make your tag more informational!

You can add more information using git tag -a TAG_NAME -m 'MESSAGE'
The tag information can be viewed without having to checkout the tag using git show TAG_NAME.

Create tag at a particular commit

You do not always have to be at the HEAD or in the tip of the branch to create a tag. If you want to create tag say 5 commits before HEAD, you can use git log to get the correct commit hash e.g git log --pretty=oneline -10 which shows the last 10 commits on the current branch.

Then, create tag using git tag -a TAG_NAME -m 'MESSAGE' COMMIT_HASH

List and Delete tags

Listing all tags is simply: git tag
Similarly, delete a tag using git tag -d TAG_NAME

Share your tag with others!

The tag created is not pushed to remote automatically. If you want your tags to be used by other contributors too, you need to push them using git push origin TAG_NAME

Sign your tags!

If you are working on a major project and want to show without any doubt that you have worked on the release, you can sign it using your GPG private key as git tag -s TAG_NAME -m 'MESSAGE'. Anyone who runs git show TAG_NAME on the tag will also see your public key signature along with the tag information.

They can additionally verify the tag using git tag -v TAG_NAME. This checks using your public key whether the signature is indeed yours.

Tags vs Branches

Why use tags when you have branches? Because branches can change and tags are linked to a specific commit. Thus, marking releases with a tag will give you the state of code when the particular software release was done.

Summary

Tags are great way to remember specific commits
It is good to add a message to the tag so that people can easily see why you tagged a particular commit

References

The Git book chapters:

Bash Command Completion - Finding all the cats in your $PATH!

Raunak Ramakrishnan — Thu, 15 Nov 2018 11:50:01 +0000

Today, I was curious to find out how many programs ending with cat I had on my system. I remember using cat, zcat and wanted to know if there are similar programs. Pressing TAB after cat only gives programs which start with cat.

1. `apropos`

In my last post, I had mentioned that apropos is a way to search for what programs are available on your system. The search string for apropos can be any regex. So, apropos 'cat$' should solve the problem. cat$ means all words ending with 'cat'

The output has

STAILQ_CONCAT (3)     - implementations of singly-linked lists, singly-linked tail queues, lists and tail queues
OPENSSL_strlcat (3ssl) - Memory allocation functions
..
bzcat (1)            - decompresses files to stdout
cat (1)              - concatenate files and print on the standard output
fc-cat (1)           - read font information cache files
gencat (1)           - Generate message catalog

Clearly, the top 2 do not look like programs. Why is apropos then returning them?
Let's have a look at the apropos manual

man apropos
# apropos - search the manual page names and descriptions

So apropos searches the man pages. And looks like there are man pages for other things and not just programs...

Digging deeper, let's try manual for the man pages!

man man
# The table below shows the section numbers of the manual followed by the types of pages they contain.
#       1   Executable programs or shell commands
#       2   System calls (functions provided by the kernel)
#       3   Library calls (functions within program libraries)
#       4   Special files (usually found in /dev)
#       5   File formats and conventions eg /etc/passwd
#       6   Games
#       7   Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
#       8   System administration commands (usually only for root)
#       9   Kernel routines [Non standard]

Ok. We are interested in are executable programs i.e section 1 of the man pages. apropos has a way to limit which sections we search using-s flag.
apropos -s 1 'cat$' gives us all programs ending with name cat which have an entry in the man pages but it does not show us any programs which do not have a man page.

2. List all programs on your path

The way Bash knows which programs can be called directly by their name (e.g ls) and not by their full path (e.g /usr/bin/ls) is by looking at the $PATH environment variable.

** Listing all executable files on path **

Here's a small bash snippet which lists the executable files in PATH (let's call it paths.sh)

#!/bin/bash
# The directories in $PATH are separated by ":", so we split by it to get individual directories
for pdir in $(echo "$PATH" | tr ":" "\n")
do
    # We `find` all files in the directory which are executable and print the filename
    find "$pdir" -maxdepth 1 -executable -type f -printf "%f\n"
done

If you prefer Python, here's a small Python program for the same (let's call it paths.py)

from itertools import chain
import os
path_dirs = os.environ['PATH'].split(':') # Split PATH by ':'
print(path_dirs)
all_files = chain(*(os.walk(p) for p in path_dirs)) # Iterable of all files in the directories contained in PATH
is_exec = lambda x : os.access(x, os.X_OK) # Function to check if a filename is executable
execs = chain(*([f for f in fs if is_exec(os.path.join(r,f))] for r,_,fs in all_files))
for x in execs:
    print(x)

Running either our Bash or Python scripts will give us the correct output!

sh paths.sh | grep 'cat$'
python3 paths.py | grep 'cat$'

3. Power of Bash Completion!

When I press TAB TAB after typing a letter, I get a list of suggestions. How does Bash do that? The Bash manual says that it uses complete and compgen built-ins for suggesting completions.

compgen generates completions using a list of words (-W) or list of commands (-c). The latter is of particular interest to us. compgen -c prints every executable on our path and all shell built-ins and shell-functions.

compgen --help prints following message:
compgen: compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat] [-W wordlist] [-F function] [-C command] [-X filterpat] [-P prefix] [-S suffix] [word] Display possible completions depending on the options.

The options stand for:

a : aliases
b : shell builtins
c : executable commands
d : directories in current directory
e : export variables
f : files in current directory
g : groups in system
j : pending jobs (in background / stopped)
k : Bash keywords
s : System services
u : users
v : All shell variables

So compgen -c | grep 'cat$' should give us every single executable ending with cat.

Epilogue

Diving into this rabbit-hole has given me a better understanding of how Bash completion works, how apropos finds relevant programs and why man pages are organized into various sections.

Command-line productivity tips : Getting help in the terminal

Raunak Ramakrishnan — Sun, 11 Nov 2018 18:34:14 +0000

The command-line is often a daunting place for beginners. With nothing but a blinking cursor and an unfriendly dark screen staring back at you, despair sets in.

Here are a few things to do if you are stuck:

1. How to find programs for specific tasks

Many times, you want to do a particular task but you do not remember the name of the program which does that. Many shell utilities are not easy to remember at first with their cryptic 2 letter names.

Worry not, there is a command called apropos to help you out!

For example, you want to show a file with line numbers. You know that there is a program but have forgotten its name. You can just type apropos -a line number to get a list of programs. We use -a flag so that apropos will only return those programs which have the words line AND number in their description.

On my computer, it gives the following output:

apropos -a line number
# Output:
addr2line (1)        - convert addresses into file names and line numbers.
nl (1)               - number lines of files
x86_64-linux-gnu-addr2line (1) - convert addresses into file names and line numbers.

There are 3 programs which have something to do with numbering lines. Let's find out what each of these programs does.

2. Find out what a program does

Let's have a look at our first candidate addr2line. Typing --help after the command is one way of finding out what a program does. It is a convention for command-line programs to print a small help message when you call them with --help. Some programs also allow do the same on calling with -h.

addr2line --help
# Output:
Usage: addr2line [option(s)] [addr(s)]
 Convert addresses into line number/file name pairs.
 If no addresses are specified on the command line, they will be read from stdin
....

Ok. This does program does something with addresses, not what we are looking for. Moving on to the next in our list: nl

nl --help
# Output:
Usage: nl [OPTION]... [FILE]...
Write each FILE to standard output, with line numbers added.
...

Looks like we found our program. Let's try it out. This is our file foo.txt

cat foo.txt
# Output:
one
two
three

four
five

nl foo.txt
# Output:
     1  one
     2  two
     3  three

     4  four
     5  five

It works ok but why is it not numbering blank lines?

3. Getting detailed information about a program

The --help messages are very concise. If we want to know in detail what a program does, we need to consult the manual or man pages. These pages are very detailed documentation of what a program does, all its possible options and arguments. Type man nl and have a look at the information...

Woah! The terminal screen is filled with information!. You can navigate up and down using the arrow keys and press q to quit.

We can see the following in the output:

.
.

-b, --body-numbering=STYLE
              use STYLE for numbering body lines
.
.
STYLE is one of:

       a      number all lines

       t      number only nonempty lines

       n      number no lines

So, nl -b a foo.txt will number all lines in foo.txt which is what we want.

nl -b a foo.txt
# Output:
     1  one
     2  two
     3  three
     4
     5  four
     6  five

That seems to solve our problem.

There is also info command which gives even more detailed usage information. You can try it out using info nl.

4. Getting examples for using a program

But what if we just quickly want to get our command to work without having to read a huge wall of text?

Turns out, there is a nifty utility you can install called tldr. If you use node or python you can install using npm install -g tldr or pip install tldr respectively. The tldr page has a list of other installation options. After installing it, just try out tldr nl in your terminal

tldr nl
# Output:
nl

  A utility for numbering lines, either from a file, or from standard input.

  - Number non-blank lines in a file:
    nl file

....

  - Number all lines including blank lines:
    nl -b a file

There! We can see the example with easy to understand description.

tldr is community-driven! People contribute examples for various commands. If your favorite command does not have an entry, you can submit a pull request.

Recap

To recap, we can:

Find out possible programs which do particular tasks using apropos
Get help information for $program using $program --help or man $program
Get examples using tldr $program

PS: What if we are not able to find programs for our tasks

In Section 1, we assume that we will be able to find programs for our task. But many times, we may not have it installed. In such cases, we can search our distribution's package manager e.g apt-get for Ubuntu or yum or dnf for CentOS / Fedora.

In Ubuntu, we can search all available packages in the repositories using apt-cache search $KEYWORD.

Improving Podcasts on dev.to

Raunak Ramakrishnan — Sat, 10 Nov 2018 07:10:07 +0000

I recently found out that dev.to has a nice podcasts page!

I have some suggestions for better podcast page UX:

Ability to save the podcast for later. Podcasts are generally at least 20 minutes long and it may not always be possible to listen right away.
Upvoting of podcasts using the heart which allows users to discover popular podcasts outside their usual subscriptions.
Ability to tag podcasts based on content. If users/moderators can add tags to podcasts, it will improve quality of search content.
A way to suggest podcasts to include on the page

In general, I think it will be better experience if we can interact with podcast pages in a way similar to articles.

Finally, this podcast of SEDaily with Leslie Lamport is a great watch for all developers.

TLA+ with Leslie Lamport

Software Engineering Daily

Your browser does not support the audio element.

1x initializing... ×

Git: How to keep your fork updated with remote repository

Raunak Ramakrishnan — Fri, 09 Nov 2018 12:52:41 +0000

This is useful when you have forked a repository (repo), cloned it to your local machine and want to keep it in sync with the original repo.

Adding the remote repo

We can list the remote repositories for our repo with git remote -v and add the original repo as follows:

git remote add upstream LINK_TO_ORIGINAL_REPO

Note that it is merely a convention to call it upstream. You can give any name you want.

Check that the repo is added to your remote using git remote -v.

Sync with remote

Check if there are any changes in remote not on your fork using git fetch upstream
Checkout whichever branch you are interested ( git checkout $BRANCH)
Merge with upstream using: git merge upstream/$BRANCH
Push your changes to origin if needed: git push origin $BRANCH

Removing the remote repo

If you no longer want to get changes from the remote repo, it is easy to remove it using git remote remove upstream

References

Github's help page

Ensuring that a shell script runs exactly once

Raunak Ramakrishnan — Tue, 06 Nov 2018 20:10:37 +0000

Many times, we have shell scripts which perform some important stuff like inserting into database, mailing reports, etc which we want to run exactly one instance of.

Enter locks!

A simple solution is to create a "lock file" and check if the file exists when the script starts. If the file is already created, it means another instance of that program is running, so we can fail with message "Try again later!". Once the script completes running, it will clean-up and delete the lock file.

LOCK_FILE=a.lock
if [ -f "$LOCK_FILE" ]; then
    # Lock file already exists, exit the script
    echo "An instance of this script is already running"
    exit 1
fi
# Create the lock file
echo "Locked" > "$LOCK_FILE"

# Do the normal stuff

# clean-up before exit
rm "$LOCK_FILE"

This looks promising but there are issues with this approach. What happens if the script does not end correctly i.e it exits because of some failure before it reaches the clean-up part of the code? Or if it gets forcibly terminated with Ctrl+C or kill command? In both these cases, the created lock file will not be deleted. So next time you run the script, you will always get an error and will have to manually delete the file.

There is another, more subtle error with the above code. A race condition. If two instances of scripts are started around the same time, it is possible that both of them get past the if [ -f "$LOCK_FILE" ] because the second instance may reach that part of the code before the first instance is able to create the lock file. Thus, we have more than one instance running.

A better lock!

Is there a way to create a lock file which is more robust to race conditions and non-standard termination (Ctrl+C, kill command, etc)? Linux offers flock a utility to manage locks from shell scripts. Using flock, we can rewrite the above snippet as follows:

LOCK_FILE=a.lock
exec 99>"$LOCK_FILE"
flock -n 99 || exit 1
# Do stuff and exit!

The exec 99>"$LOCK_FILE" creates a file descriptor numbered 99 and assigns it to LOCK_FILE. File descriptors (fd) 0, 1, 2 are for stdin, stdout, stderr respectively. We are creating new fd with a high number to ensure that it does not clash with numbered fds opened later-on by script.

flock -n 99 || exit 1 does 2 things. Firstly, it acquires an exclusive lock on the file descriptor 99 which refers to our LOCK_FILE. This operation is guaranteed by the linux kernel to be atomic. Secondly, if it fails to acquire the lock, it exits with return code 1. We do not need to worry about any clean up. flock will automatically release the lock when the script exits regardless of how it terminates. This solves our problem!

What if I wanted to add a more informational message instead of exiting directly on failure to acquire lock? We can change the line flock -n 99 || exit 1 as follows:

flock -n 99
RC=$?
if [ "$RC" != 0 ]; then
    # Send message and exit
    echo "Already running script. Try again after sometime"
    exit 1
fi

The flock man page has an example which you can use to add an exclusive lock to start of any shell script:

[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$@" || :

This boilerplate uses the script file itself as a lock. It works by setting an environment variable $FLOCKER to script file name and executing the script with its original parameters after acquiring the lock. On failure however, it does not print anything and silently exits.

$0 here stands for name of the script. $@ stands for all arguments passed to the script when it was called.

Use case for me

My team uses a test machine where we deploy multiple branches of a code-base. We need to make sure that exactly one person is building the project at a particular time. The deploy script pulls the specified branch of code from git and builds the project, deploys the main service and starts ancillary services. The script takes sometime to execute. If someone tries to deploy another branch while a build is ongoing, both can fail.

With the above snippet, calling the script more than once shows the current branch being built and exits with failure.

Forem: Raunak Ramakrishnan

Level up your S3 skills by playing this game!

What you will need:

CLICK HERE TO PLAY!

Walk through (Spoilers)

Solution:

Solution:

Solution:

Solution:

Takeaway:

Solution

Takeaway

Solution

Solution

Takeaway

Solution

Takeaway

Solution

Solution

Takeaway

Solution

Takeaway

VICTORY! - Level 12

Practical Python Programming - Online course

Course Details:

About the author:

Keynote: A Talk Near the Future of Python

Python Concurrency From the Ground Up: LIVE!

Keynote: Builtin Superheroes

Lambda Calculus

5 NPM and Node tips to make your dev machine safer

1. NEVER run npm as sudo/root

If you are lazy (like me), here's a summary from the linked guide:

2. Install/Use node using nvm

3. List all your globally installed npm packages and remove any unnecessary ones

4. Audit your packages

A (hacky) way to npm audit global packages:

5. Use npx to run executables

Bonus : Use Deno for developer tools

Conclusion

Webhook to auto-deploy on git push to Github

What is a webhook?

Webhook server

Deploy script

Configuration to run deploy script

Expose your webhook server safely to the internet

Nginx configuration

Add your webhook URL to Github

Bonus: Create a systemd user service for webhook (Linux)

Bash function to add TILs from the command line

How to use:

Explanation on the function

Setting display brightness on Linux from the command line

Working with Linux Display environments in a cron

How to get the display number of your monitor?

Adjusting backlight brightness vs perceived brightness

Other tools

Git Tagging Tutorial

What are tags

How to use git tag better

Checkout code to the tag

Make your tag more informational!

Create tag at a particular commit

List and Delete tags

Share your tag with others!

Sign your tags!

Tags vs Branches

Summary

References

Bash Command Completion - Finding all the cats in your $PATH!

1. apropos

2. List all programs on your path

3. Power of Bash Completion!

Epilogue

Command-line productivity tips : Getting help in the terminal

1. How to find programs for specific tasks

2. Find out what a program does

3. Getting detailed information about a program

4. Getting examples for using a program

Recap

2. Install/Use node using `nvm`

A (hacky) way to `npm audit` global packages:

5. Use `npx` to run executables

How to use `git tag` better

1. `apropos`