Documenting Makefiles for DevOps teams and Software developmes using mh

Oz Tiram — Thu, 25 Jul 2024 20:31:50 +0000

Modern tech stacks are complex and require one to remember hundreds of combinations of commands and parameters.
As a full stack developer, I use many different tools in the command line.

To name a few: go, python, sass, docker, terraform, kubectl, awscli, eksctl and more.

Remembering the all the tasks requires some brain muscle, and context. Jumping from one project to another, I need to answer many questions before being able to start working on it:

do I push the build docker image to Y?
Does this project use webpack or rollup?
Does it use pipenv or poetry?
Do I type yarn or npm to install js dependencies?

Jumping across teams, or coming back to a certain project after a while, becomes really hard.

Makefiles offer a really nice abstraction to all of these command and tools.

For example, instead of typing:



$ docker build -t $(git describe --always)-f  Dockerfile .

I can now just write



$ make build-img

This not only saves time restructuring the correct command, it requires less typing, and prevents mistakes. It is also great for teams with various skill levels.

make is a very old and boring technology. It is battle tested, and people already wrote mountains of words on why you should adopt Makefiles.

So, after you adopted make, you now have a few Makefiles, and when you come into a project you just need to read the Makefile to see what it is doing, right?

Well... not so fast...

Unfortunately, Makefiles have very cryptic syntax if you are not used to them. Also, they can be mixed with inline shell, and finally they can have a few hundred lines of code.
Hence, understanding all the goals (build targets) is going to be tedious.

Some people already came with a good solution to this problem by integrating an awk script in the Makefile to parse the targets in the file and print a useful overview (here is just one example out of many).

For almost a decade, I used a similar awk script embedded in my Makefile. Adding color, some more bells and whistles. With time, I wanted even more from that script. Most importantly, I wanted to document variables which affect different targets.

So, I rewrote it in Python, which seemed easier than awk. When it grew to about 30 lines of code, I extracted it as a Python package called make-help-helper.
As much as I love Python, building and distributing binaries for this Makefile helper was not so easy, and letting people copying an inline script from one Makefile to another seemed like a very bad way to distribute software ...

enter mh

With the limited scoped of this script, I decided it was a good and fun project to exercise my C programming skills. And so the program mh was born. In the spirit of old UNIX programs, it has a two letter name, which stands for makes help
I'm using it across multiple projects for more than 3 years, and recently I gave it some more polish, and decided to release it to the world, in the hope more people will find it useful.

So without further ado, here is a demo of what mh does.
Running make or make help in a project directory with a specially crafted Makefile you will see a colored output similar to this:

The above output was generated from a Makefile with the following content:



.PHONY: test coverage all watch clean docker-build docker-run save cp-to-server \
   import-on-server run-on-server disable-on-server

SHELL := /bin/bash
.DEFAULT_GOAL := help

.PHONY: help
help:
        @mh -f $(MAKEFILE_LIST) $(target) || echo "Please install mh from https://github.com/oz123/mh/releases/latest"
ifndef target
    @(which mh > /dev/null 2>&1 && echo -e "\nUse \`make help target=foo\` to learn more about foo.")
endif

ifneq (,$(wildcard ./.env))
include .env
    export
endif

run-server: ## run a local server
    python3 sweet.py serve -p 8080 &

test:  run-server ## run the test suite with pytest
    python -m pytest

coverage:  ## run the tests and collect metrics
    python -m pytest -vv --cov=coldsweet tests

all: update

update: CSS_FILES ?= ./static/stylesheets/all.scss:./static/stylesheets/all.css #? the css files to update
update:  ## update css files from sass sources
    sass -f -t compressed --update $(CSS_FILES)

watch:  ## rebuild css on changes to sass sources
    sass --watch $(CSS_FILES)

clean:  ## remove sass cache
    rm -r ./.sass-cache

REGISTRY=registry.acme.org  #? the docker registry
ORG=oz123   #? the organization 
IMG=coldsweet   #? the image name

docker-push:: ## push the built image to the repository
    docker push $(REGISTRY)/$(ORG)/$(IMG):$(shell git describe)

docker-build:  ## build a docker image
    docker build -t $(REGISTRY)/$(ORG)/$(IMG):$(shell git describe) -f docker/Dockerfile-py312 .


docker-run: CMD ?= 
docker-run: TAG ?= $(shell git describe)
docker-run:  ## run the docker image locally
    docker run --rm -e COLDSWEET_DEBUG=1 \
        -e COLDSWEET_INSTALL_DIR=/run/coldsweet \
        -e COLDSWEET_CONFIG_PATH=/etc/coldsweet/config \
        -v $(CURDIR)/coldsweet/templates:/run/coldsweet/coldsweet/templates \
        -p 8080:8080 \
        -it -p 9001:9001 -v $(CURDIR)/data:/var/lib/coldsweet/db -v $(CURDIR)/docker/:/etc/coldsweet/ -w /run/coldsweet $(REGISTRY)/$(ORG)/$(IMG):$(TAG) $(CMD)

docker-save:  ## saves the image to a localfile
    sudo docker save $(ORG)/$(IMG):$(TAG) > $(ORG).$(IMG).$(TAG).img

deploy-k8s:
    make -C k8s apply

cp-to-server:  ## upload the image to a server
    scp $(ORG).$(IMG).$(TAG).img $(SERVER):~/


import-on-server:  ## imports the image on the serve
    ssh -t $(SERVER) "echo $(SUDOPASSWORD) | sudo -S docker load --input  $(ORG).$(IMG).$(TAG).img"

run-on-server:  ## runs the docker image on a remote server
    ssh -t $(SERVER) "echo $(SUDOPASSWORD) | sudo -S docker run --name $(CONTAINERNAME) \
        -d --restart always
        -v $(BASEDIR):/var/lib/coldsweet/db \
        -p $(HOST):9001:9001 $(ORG)/$(IMG):$(TAG)"

disable-on-server:  ## stops the container on a remote server
    ssh -t $(SERVER) "echo $(SUDOPASSWORD) | sudo -S docker rm --force $(CONTAINERNAME)"

compile-sass:
    cd static/stylesheets && sassc all.scss all.css

minify-sass:
    cd static/stylesheets && sassc --sourcemap -t compressed all.scss all.min.css

The biggest addition to the simple AWK script or the Python package is the ability to document global variables and target local variables.

Targets are documented with:



foo: ## foo does bar
   echo bar

local variables are documented with:



NAME ?= world #? great who
greet:
     echo $(HELLO)

global variables are documented with:



REGISTRY ?= docker.io #? where should images be pushed

In addition to having global variables inside the Makefile, I found it useful to have the Makefile read variables from a .env a la 12-Factor applications.
That is done by the following code in the above Makefile:



ifneq (,$(wildcard ./.env))
include .env
    export
endif

Now, if your directory contains a .env file, mh will read the variables and their description and show them on the console:

Usually, my repositories will contain a file called dot.envexample, which will show other developers what can go into this file.
The format which is understood by mh is:



FOO=BAR #? control the foo
MY_SECRET_API_KEY=123456 #? the secret api key

You can learn more about a specific target with make help target=<name>, for example:

If you found it so far useful, you can get a version of mh as a statically compiled binary for Linux or Dynamically compiled Mac OSX binary in the GitHub release page.

Of course, you can study the code, or provide feedback and contributions too.

Forem: Oz Tiram

Documenting Makefiles for DevOps teams and Software developmes using mh

enter mh