Forem: Shivan Moodley

Using the IFS variable to do things with CSV files

Shivan Moodley — Wed, 05 Feb 2020 02:44:16 +0000

The Problem

Earlier this week, I had to do a mass-insert into a database table from a CSV file.

The CSV file had ~20 columns and ~500k rows. I only really cared about 4 of the columns.

Visually, the problem can represented accurately as the following:

Previously I had seen this problem solved with massive amounts of over-engineering, and sometimes with clever usage of Microsoft Excel. However, in my never-ending quest to use the tools available to me more effectively. I challenged my self to see if I could generate the inserts I needed without leaving my terminal window.

First Steps

Some Example Data

For this post I am going to use the heart.csv data set sourced from Kaggle

It has the following (14) columns:


age age in years
sex(1 = male; 0 = female)
cp chest pain type
trestbps resting blood pressure (in mm Hg on admission to the hospital)
chol serum cholestoral in mg/dl
fbs (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)
restecg resting electrocardiographic results
thalach maximum heart rate achieved
exang exercise induced angina (1 = yes; 0 = no)
oldpeak ST depression induced by exercise relative to rest
slope the slope of the peak exercise ST segment
ca number of major vessels (0-3) colored by flourosopy
thal 3 = normal; 6 = fixed defect; 7 = reversable defect
target 1 or 0

Enter bash and IFS

Let's read this using a small script that leverages IFS:

#!/bin/bash

# Read a CSV file and presumably do stuff with what we read

# Configure the Internal Field Seperator (see: Delimeter)
IFS=','

# while stuff in FILE, read COLUMNS using $IFS
while read -r age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal target
do
        # do things
        echo "$age $sex $cp $trestbps $chol $fbs $restecg $thalach $exang $oldpeak $slope $ca $thal $target"
done < "$1"

We can use this script by running the following:

# make it executable
chmod +X <SCRIPT_NAME>
# run it 
./<SCRIPT_NAME> <PATH_TO_CSV_FILE>

Which should yield:

age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal target
63 1 3 145 233 1 0 150 0 2.3 0 0 1 1
37 1 2 130 250 0 1 187 0 3.5 0 0 2 1
41 0 1 130 204 0 0 172 0 1.4 2 0 2 1
56 1 1 120 236 0 1 178 0 0.8 2 0 2 1
57 0 0 120 354 0 1 163 1 0.6 2 0 2 1
57 1 0 140 192 0 1 148 0 0.4 1 0 1 1
56 0 1 140 294 0 0 153 0 1.3 1 0 2 1
44 1 1 120 263 0 1 173 0 0 2 0 3 1
...
57 0 0 140 241 0 1 123 1 0.2 1 0 3 0
45 1 3 110 264 0 1 132 0 1.2 1 0 3 0
68 1 0 144 193 1 1 141 0 3.4 1 2 3 0
57 1 0 130 131 0 1 115 1 1.2 1 1 3 0
57 0 1 130 236 0 0 174 0 0 1 1 2 0

Let's further assume that we have a database schema DATA that contains a table called HEART in it.

The table was created with the following SQL statement:

CREATE TABLE DATA.HEART(
   ID   INT              NOT NULL,
   AGE  INT              NOT NULL,
   THAL INT,
   CHOL   DECIMAL,       
   PRIMARY KEY (ID)
);

Generating SQL Inserts

Let's modify our pre-existing script to create a bunch of insert statements from our heart.csv file for this table:

#!/bin/bash

# Read a CSV file and presumably doing stuff with what we read

# Configure the Internal Field Seperator (see: Delimeter)
IFS=','

# Keep track of the sequence
declare -i id=0
# while stuff in FILE, read COLUMNS using $IFS
while read -r age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal target
do
        # ignore the first row since it contains column headers
        # craft an insert statement and append to the end of a file
        if [ $id -eq 0 ]
        then
                echo "ignoring first row"
        else
                echo "INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES ($id, $age, $thal, $chol);" >> $2
        fi
        id=$((id+1))
done < "$1"

We can now run this with a second parameter (the output file name):

./<SCRIPT_NAME> <IN_FILE_NAME> <OUT_FILE_NAME>

Which gives us something like this:

INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (1, 63, 1, 233);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (2, 37, 2, 250);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (3, 41, 2, 204);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (4, 56, 2, 236);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (5, 57, 2, 354);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (6, 57, 1, 192);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (7, 56, 2, 294);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (8, 44, 3, 263);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (9, 52, 3, 199);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (10, 57, 2, 168);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (11, 54, 2, 239);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (12, 48, 2, 275);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (13, 49, 2, 266);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (14, 64, 2, 211);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (15, 58, 2, 283);
INSERT INTO DATA.HEART (ID, AGE, THAL, CHOL) VALUES (16, 50, 2, 219);
...

And you're done! (kind of, you'd still have to actually run the INSERT script)

Caveats

This post doesn't cover how to deal with internal commas and special character escaping weirdness. But it is a good starting point for a bare-bones CSV parser.

I might follow this up with another small post about using this script to easily pull out and aggregate different columns.

One of the things I've learned from using this approach to deal with CSV files is that the simplest tool is often the best and can yield great results without much overhead.

If you enjoyed this post or have any thoughts, feel free to comment or find me on Twitter.

Thanks for reading!

References

IFS Usage - https://bash.cyberciti.biz/guide/$IFS

Finally published my personal portfolio/blog

Shivan Moodley — Mon, 13 Jan 2020 19:05:57 +0000

After a few iterations and trying to strive for the 'perfect' blog and personal portfolio, I finally caved and have published my site for the world to see.

You can find it here

And as a bonus, I quickly added in a dark mode toggle using the use-dark-mode hook.

Things I want to improve on:

a11y : I haven't given any thought to accessibility yet, and it's the first on my list of things to tackle
responsiveness and device consistency : I have only tested on my mobile device and laptop. So i'm not exactly sure how good this looks on other resolutions and devices
posts : currently I load posts into a Firebase real-time database via a manual JSON payload upload. I want to automate this process.
colors: color is subjective, so even though I like then color scheme, everyone else may not feel the same!

Any feedback is appreciated, I plan to post canonically on my personal site then cross-post to DEV!

What is your prototyping stack?

Shivan Moodley — Mon, 06 Jan 2020 05:15:59 +0000

If I ever need to put together a PoC or test out an idea I generally tend toward using React (for anything visual), Go + Gorilla (api/web server stuff) and Heroku for hosting. (Although now I've started leaning toward DigitalOcean even though it takes slightly longer on the config side)

What's your preferred 'playground' tech stack?

A 2019 retrospective and looking forward to 2020

Shivan Moodley — Wed, 01 Jan 2020 14:11:23 +0000

2019 was a year of 'doing stuff' for me. With the momentum I gathered this year, I want to make 2020 the year of 'doing more stuff'.

So in the spirit of 'doing more stuff', let's start 2020 off on the right foot.

I've hacked together a quick visualisation of 2019 with React!

Check it out here (not optimised for mobile 😉)

The things I want to do in 2020 are the following:

Continue learning React and mastering all it's intricacies
Launch a SaaS product and get a few users
Set up a personal blog and post there first and link to DEV
Be an active part of the DEV community and build a following
Improve my understanding of Go
Continue my focus on building scalable systems in Java and Spring Boot
Keep going with gym!

And of course, I will be writing about most of it here. This post is mostly to build some accountability for myself.

How often do you do informal retrospectives?

Shivan Moodley — Sun, 29 Dec 2019 23:02:57 +0000

Recently I've been doing informal retrospectives at the end of each 'development' session. (Just keeping the notes in Google docs for now)

Usually it's just writing down what I've done, what I'd like to do next and the things I hacked and need to fix. (or reminders to do something that I've been meaning to do for a while)

Does any one do anything similar? Or maybe with a different format?

Deploying to Heroku: Docker, Go and React

Shivan Moodley — Sun, 29 Dec 2019 12:54:14 +0000

We're going to deploy a React Application to Heroku that will be served from a Go backend which is then neatly wrapped in a Docker image.

You'll need the following:

Golang (I am using v1.13.5)
npm
A text editor (VSCode is what I am using)
A unix based terminal
Docker
A Heroku account and the Heroku CLI

What is Heroku?

From their page

"Heroku is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud."

Use CRA to set up a React app

We're going to use create-react-app to bootstrap our React project.

npx create-react-app deployment-demo

Write the Go backend to serve it

Let's change directory into our newly created react project, create a sub-directory called server and create a file called server.go in it:

cd deployment-demo/ && mkdir "server" && cd "$_" && touch server.go

Open up your favourite text editor and paste the following into server.go:

package main

import (
    "log"
    "net/http"
    "os"
)

func main() {
    /*
        Grab the environment variable that has been hopefully set, but also set up a default
    */
    port := os.Getenv("PORT")
    defaultPort := "8080"
    /*
        Serve the contents of the build directory that was produced as a part of `npm run build` on the root `/`
    */
    http.Handle("/", http.FileServer(http.Dir("./build")))

    /*
        Check if the port environment variable has been set and if so, use that, otherwise let's use a reasonable default
    */
    if !(port == "") {
        log.Fatal(http.ListenAndServe(":"+port, nil))
    } else {
        log.Fatal(http.ListenAndServe(":"+defaultPort, nil))
    }
}

Caveat: Heroku gives us a port to bind our web application to, so we grab that from an environment variable, you can read more about it here

Build a `production` version of our application

From our current directory ../deployment-demo/server go up a level back to the root of our React project, and run:

npm run build

This will produce a directory called build.

Let's copy this into our server directory, which will serve as our production payload.

cp -r build/ server/

We can now navigate back into our server directory and start deploying.

Docker

Create the following Dockerfile in the server directory:

# Stage 1
FROM golang:alpine as builder
RUN apk update && apk add --no-cache git
RUN mkdir /build 
ADD . /build/
WORKDIR /build
RUN go get -d -v
RUN go build -o deployment-demo .
# Stage 2
FROM alpine
RUN adduser -S -D -H -h /app appuser
USER appuser
COPY --from=builder /build/ /app/
WORKDIR /app
CMD ["./deployment-demo"]

Heroku

Create an app

If you don't have a Heroku account, go and create one! (it should take less than 5 minutes)

Once that's done, you're going to need the Heroku CLI, which can be easily installed on Ubuntu by running:

sudo snap install heroku --classic

After installing the CLI, run the following command to login:

heroku login

This will open a browser window for you to login (it's an extremely cool system)

Now run:

heroku create [YOUR_APP_NAME]

in the deployment-demo/server/ directory.

An app will be created for you on your Heroku account and you should be able to see it on your dashboard

The Container Registry

The Heroku Container Registry allows you to deploy Docker images to Heroku.

You can read more about it here.

Run the following to login in to the registry:

heroku container:login

Run this command to build your Docker image and push it to Heroku:

heroku container:push web --app [YOUR_APP_NAME]

Notice that we specify web, this indicates the process type we want to associate with this application. Further reading on process types can be found here.

And finally, run this to release the image to your application:

heroku container:release web --app [YOUR_APP_NAME]

We should now be able to navigate to our application hosted on Heroku by running:

heroku open --app [YOUR_APP_NAME]

And we should see the following:

That's it! That's all you need to do to deploy a Docker image running a Go server that serves a React application to Heroku!

Building a simple CI/CD pipeline for local testing using Go, Docker, Minikube and a Bash script

Shivan Moodley — Tue, 24 Dec 2019 18:14:04 +0000

Wat

In this article I will detail the thought process and implementation of an automated build and deployment pipeline that I wrote to reduce the time taken to get feedback for code changes I make to my code bases.

The bulk of this code is written in Go and Bash. It’s a very hacky implementation but was extremely fun to write and use.

Disclaimer: This post is quite code heavy, but I try to highlight the thought-process behind the code.

You can find the code for this entire post here cicdexample

This is a quick GIF of what we're going to build:

That sounds cool, but why?

Feedback is important

The most important part for me when writing any sort of code is how fast I can get feedback.

Earlier this year I started learning Go in my spare time and whilst learning, I wanted to Dockerize my application code (I was writing my first web app in Go) and eventually host it on Heroku.

So this is what I was doing for a few days:

In total, it was taking me ~ 10 minutes to verify new functionality in a
'production' environment.
No exactly bad turnaround time when you’re working on something for your day-job, but for a pet project it wasn’t really that fun.

An added downside to the above pipeline is that it is really hard to test functionality when I don't have a consistent internet connection, which happens rather often. So I started to think about alternatives.

I wanted to have minimal latency between ctrl-s and testing the change - kind of like this diagram:

Environment consistency trade-off

I realised that testing on Heroku wasn’t a hard requirement for my development process. But I did want some resemblance to my 'production' environment in my test environment.

I started to sketch out a simple CI/CD pipeline I could run on my laptop that would allow me to test my application running in Docker on similar cloud infrastructure.

As an aside, I kind of knew that I was probably going to write something that already exists, but I also knew that it would be a fun exercise and learning experience.

A simple start

What do you need to have?

Go (any version will do for this article)
Some Bash knowledge and a Unix terminal
Minikube
Kubectl
Docker
An IDE or Text Editor (I interchangeably use Vim and VSCode throughout this article)
Linux (Ubuntu 18.04 LTS is the distro I am using)

Where do we start?

I have some experience working with build tools like Jenkins and I also have some experience with running an automated build pipeline tied to an automatic deployment tool.

So using the ideas and context I had from those tools. I started simply, I ignored Heroku for the moment and wrote a simple Shell script that would build my go code base, and then based on a command line argument, build and tag the Docker image(s).

I extended the Shell script to include a kubectl command to deploy to Minikube after building a docker image.

I realised that now that I had this Shell script, I had in essence created a deployment and build job for my application. So I started putting together a Go app to automate the running of this script based on a trigger.

How exactly did you do that?

To illustrate how I did this without complicating this post with excessive technical detail, I’ve created a little checklist to follow, which this article expands upon:

a. Create a small Go application that serves a HTML page
b. Dockerize it!
c. Write a simple Bash script that builds the docker image
d. Push the docker image into Minikube and see it run
e. Write a small command line tool that runs the script automatically

Creating the test application

Let’s create an example Go application that starts a simple file server and serves a simple HTTP page (that uses Bulma CSS to make it look a bit better) on ‘/’

The web server looks like this:

package main

import (
    "log"
    "net/http"
    "os"
)

func main() {
    // for heroku since we have to use the assigned port for the app
    port := os.Getenv("PORT")
    if port == "" {
        // if we are running on minikube or just running the bin
        defaultPort := "3000"
        log.Println("no env var set for port, defaulting to " + defaultPort)
        // serve the contents of the static folder on /
        http.Handle("/", http.FileServer(http.Dir("./static")))
        http.ListenAndServe(":" + defaultPort, nil)
    } else {
        http.Handle("/", http.FileServer(http.Dir("./static")))
        log.Println("starting server on port " + port)
        http.ListenAndServe(":" + port, nil)
    }
}

The html file in /static is quite simply:

<html>
    <head>
        <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>CI/CD Example</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.8.0/css/bulma.min.css">
    </head>
    <body>
        <div style="text-align: center" class="container">
            <h1 class="title is-1">Example CI/CD Stuff</h1>
            <p>Changes should make this automatically redeploy to our local test environment</p>
        </div>
    </body>
</html>

Great, now if we build (go build) this Go App and run it, we should be able to see the following at localhost:3000

Containerisation

We now have an app, which is really cool and does a whole bunch of stuff (nothing) but that is beside the point. We’re focused on how to get this little guy building himself. First things first, we need to Dockerize it.

If you don’t have Docker installed, check out their docs here (https://docs.docker.com/install/)

Let's create the following simple 2-stage Dockerfile:

# Stage 1
FROM golang:alpine as builder
RUN apk update && apk add --no-cache git
RUN mkdir /build 
ADD . /build/
WORKDIR /build
RUN go get -d -v
RUN go build -o cicdexample .
# Stage 2
FROM alpine
RUN adduser -S -D -H -h /app appuser
USER appuser
COPY --from=builder /build/ /app/
WORKDIR /app
CMD ["./cicdexample"]

This Dockerfile first creates a builder image with the entire contents of the local directory copied into a directory called /build on the image. It then fetches the dependencies for our little app, and builds it - producing a binary called cicdexample

The second stage actually creates the image we will ultimately run. We use the base Alpine image and create a user called appuser that uses a directory called /app as it’s home directory.
We then copy the contents of the /build directory from the builder image into the /app directory on the Alpine image, set the working directory to /app and run the Go binary that we just copied. It’s important to note that we copy the entire /build directory contents since we need the static resources directory that is not part of the Go binary.

Once we’ve created the Dockerfile and can successfully run
docker build -t example:test . which produces (output is slightly trimmed):

Sending build context to Docker daemon  7.638MB
...
Successfully built 561ee4597a93
Successfully tagged example:test

we're ready to move on.

Bashing things into shape

Next up we need a script to automate the process of building this Docker image for us. But we don’t just want to build an image with a script, we want to create a feedback loop using go build so that we don’t waste time setting up a Docker image when our code doesn’t even compile.

We don’t technically have to flag success from the result of go build, but rather have the error output redirected to a file, which we can then check for ANY content, and signal failure if there is anything in stderr as a result of go build

The logical steps for this script are best represented graphically:

The script is as follows (with a nice timestamps and command line output included):

#!/bin/bash

# Timestamp Function
timestamp() {
    date +"%T"
}

# Temporary file for stderr redirects
tmpfile=$(mktemp)

# Go build
build () {
    echo "⏲️    $(timestamp): started build script..."
    echo "🏗️   $(timestamp): building cicdexample"
    go build 2>tmpfile
    if [ -s tmpfile ]; then
        cat tmpfile
        echo "❌   $(timestamp): compilation error, exiting"
        rm tmpfile
        exit 0
    fi
}

# Deploy to Minikube using kubectl
deploy() {
    echo "🌧️    $(timestamp): deploying to Minikube"
    kubectl apply -f deploy.yml
}

# Orchestrate
echo "🤖  Welcome to the Builder v0.2, written by github.com/cishiv"
if [[ $1 = "build" ]]; then
    if [[ $2 = "docker" ]]; then
        build
        buildDocker
        echo "✔️    $(timestamp): complete."
        echo "👋  $(timestamp): exiting..."
    elif [[ $2 = "bin" ]]; then
        build
        echo "✔️    $(timestamp): complete."
        echo "👋  $(timestamp): exiting..."
    else
        echo "🤔   $(timestamp): missing build argument"
    fi
else
    if [[ $1 = "--help" ]]; then
        echo "build - start a build to produce artifacts"
        echo "  docker - produces docker images"
        echo "  bin - produces executable binaries"
    else
        echo "🤔  $(timestamp): no arguments passed, type --help for a list of arguments"
    fi
fi

Running it, with the varying arguments, gives us the following:

./build build bin (compilation failure)

🤖    Welcome to the Builder builder v0.2, written by github.com/cishiv
⏲️  16:40:47: started build script...
🏗️ 16:40:47: building cicdexample
# _/home/shiv/Work/dev/go/cicdexample
./main.go:25:1: syntax error: non-declaration statement outside function body
❌ 16:40:47: compilation error, exiting

./build build docker (slightly trimmed output)

🤖    Welcome to the Builder builder v0.2, written by github.com/cishiv
⏲️  16:40:05: started build script...
🏗️ 16:40:05: building cicdexample
🐋    16:40:06: building image example:test
Sending build context to Docker daemon  7.639MB
...
Successfully tagged example:test
✔️  16:40:06: complete.
👋    16:40:06: exiting...

This seems reasonable, we can compile our code and build a docker image with one command.

Before we head to the next step, let's have a look at our checklist so far:

a. Create a small Go App that serves a HTML page ✔️
b. Dockerize it! ✔️
c. Write a simple Bash script that builds the docker image ✔️
d. Push the docker image into Minikube and see it run
e. Write a small command line tool that runs the script automatically

Minikube-ing

We finally have some sort of automation for our build pipeline, but it doesn't really give us a way to test our application yet.

Enter Minikube to save the day (sort of).

We want our testing pipeline to be as follows:

If you don’t have Minikube installed you can check out the docs here (https://kubernetes.io/docs/tasks/tools/install-minikube/) on how to get it up and running.

You will also need to grab kubectl.

Once Minikube is installed, it is as simple as running minikube start to start up a single node cluster.

The next step is setting up a Kubernetes deployment for our app, that we can push into the Minikube cluster.

Since this is not an article about Kubernetes, I will keep this step short. We want to have a deploy.yml file where we can tell Kubernetes to create a deployment and a service for our application. It would be nicer to have separate files for creating the deployment and service, but for this example, we’ll just recreate both of them every time we want to redeploy.

So we need the following file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
spec:
  selector:
    matchLabels:
      app: example
      tier: example
      track: stable
  template:
    metadata:
      labels:
        app: example
        tier: example
        track: stable
    spec:
      containers:
        - name: example
          image: "example:test"
          ports:
            - name: http
              containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: example
spec:
  type: NodePort
  selector:
    app: example
    tier: example
  ports:
  - protocol: TCP
    port: 3000
    targetPort: http
    nodePort: 30000

We tell kubernetes to create a service called example that is exposed on NodePort 30000 (so that we can access it via a URL that doesn't change every time we recreate the service) , with an application running on port 3000 in the container.

At this point, because we have specified a NodePort in our deployment descriptor we should be able to simply refresh our web page, and see our changes.

To get the application deployed into the cluster, run the following command in the terminal:

./build build docker && kubectl apply -f deploy.yml

The application should now be live on the Minikube cluster and be exposed on a NodePort.

To get the URL for the application, run this command:

minikube service example --url

You should get a URL similar to this:

http://10.0.0.101:30000

example is the service name we specified in our deploy.yml file.

If we navigate to our URL, we should see our app.

We can now test our pipeline for the first time.

Make a code change to index.html and run:

./build build docker && kubectl apply -f deploy.yml

You might find that this doesn't actually work yet. The kubectl command outputs:

deployment.apps/example unchanged
service/example unchanged

And rightly so, since we don't tag our Docker image differently each time we create it - Kubernetes doesn't actually recognise that our code has changed. A quick hack to remedy this, since we're testing locally, is to amend our command to the following:

./build build docker && kubectl delete deployment example && kubectl delete service example && kubectl apply -f deploy.yml

This allows us to cleanly recreate the service and deployment every time we re-run the script. It's important to note that this is very much a hack, and a better way to do this would be to tag the Docker images differently each time they are produced, and update the deploy.yml file with the correct Docker image tag.

So running:

./build build docker && kubectl delete deployment example && kubectl delete service example && kubectl apply -f deploy.yml

Will allow us to see the change we made to our HTML.

This seems reasonable, however to clean it up a bit, let's add the additional kubectl commands to our Bash script.

It is simple enough to do by adding the following function, and making a change to the conditional logic slightly, allowing for a deploy parameter to be passed to the ./build build ... command:

Function:

# Deploy to Minikube using kubectl
deploy() {
    echo "🌧️    $(timestamp): deploying to Minikube"
    kubectl delete deployment example
    kubectl delete service example
    kubectl apply -f deploy.yml
}

Conditional logic:

if [[ $1 = "build" ]]; then
    if [[ $2 = "docker" ]]; then
        if [[ $3 = "deploy" ]]; then
            build
            buildDocker
            deploy
        else
            build
            buildDocker
        fi
        echo "✔️    $(timestamp): complete."
        echo "👋  $(timestamp): exiting..."

We can now run:
./build build docker deploy
to quite easily drive code changes all the way from compilation to deployment! (with a turn-around time of ~1s)

The Final Act: True Automation

Finally, we want to wrap the Bash script we have created into a purpose built Go application that automates this process.

We have to decide on a trigger for our build pipeline. There are 2 options at this point, which are:

Builds are triggered on a combination of time elapsed and file changes.
Builds are triggered on commits into a git repository

To illustrate the concept without much technical overhead, we will go with the first option

This is essentially what we want to write:

I won't paste the entire source code here, however I will discuss a few key points regarding it (the full code can be found here)

We need to address 5 problems:

What sort of hashes are we going to use for files?
How often should the hash recalculation be done?
How do we set up polling on an interval?
How do we run our script in the context of a Go application?
What race conditions are there?

The first problem is solved quite succinctly in Go, we can calculate a sha256 hash for a file with just the following code snippet:

func CalculateHash(absoluteFilePath string) string {
    f, err := os.Open(absoluteFilePath)
    HandleError(err)
    defer f.Close()
    h := sha256.New()
    if _, err := io.Copy(h, f); err != nil {
        log.Fatal(err)
    }
    return hex.EncodeToString(h.Sum(nil))
}

The second problem has a non-trivial answer and it specific to your use case, however it should be reasonable enough to recalculate hashes every 15 seconds or so - this means that we should have automated deployments running every 15 seconds if there was a code change in that window.

I ran a few benchmarks on the actual time taken to run a build in the Go application so that we could make an educated guess on how often to poll for file changes.

The snippet I used to run the benchmark is as follows:

func stopwatch(start time.Time, name string) {
    elapsed := time.Since(start)
    log.Printf("%s took %s", name, elapsed)
}

Simply add defer stopwatch(time.Now(), "benchmark") at the top of the function you want to benchmark!

After the benchmark I estimated that we need to allow for 7 seconds for a past build to complete, and 8 seconds of overhead for the unexpected. Which gives us a total of 15 seconds.

Problem number 3 is solved by defining the following function and using it like illustrated:

package main

import (
    "fmt"
    "time"
)

func main() {
    go DoEvery(10*time.Second, f, "test")
    for {}
}

func DoEvery(d time.Duration, f func(time.Time, string), action string) {
    for x := range time.Tick(d) {
        f(x, action)
    }
}

func f(t time.Time, action string) {
    fmt.Println(action)
}

We simply create a function that will be invoked every X seconds.

The 4th problem also has a rather simple and effective solution in Go, we can define an action string and run it using the os/exec package in Go.

package main

import (
    "bytes"
    "log"
    "os/exec"
)

func main() {
    runAction("./build build docker deploy")
}

func runAction(action string) {
    log.Println("Taking action, running: " + action)
    cmd := exec.Command("/bin/sh", "-c", action)
    var outb, errb bytes.Buffer
    cmd.Stdout = &outb
    cmd.Stderr = &errb
    err := cmd.Run()
    if err != nil {
        log.Printf("error")
    }
    log.Println(outb.String())
    log.Println(errb.String())

The last problem is an important one, one glaring race condition is that if we monitor the hashes for ALL the files in a directory, then we will most likely get caught in some sort of recursive build loop, since we are actively changing the files we monitor (by producing a binary). We can borrow a concept from git here, and implement a whitelist, that is, a list of files to ignore in our hash calculation. Something to this affect,

var whiteList []string

func CreateWhiteList() {
    file, err := os.Open("./.ignore")
    if err != nil {
        log.Println("no .ignore file found, race condition will ensue if jobs edit files -- will not create whitelist")

    } else {
        defer file.Close()
        scanner := bufio.NewScanner(file)
        for scanner.Scan() {
            log.Println(scanner.Text())
            whiteList = append(whiteList, scanner.Text())
        }
        if err := scanner.Err(); err != nil {
            log.Fatal(err)
        }
    }
}

Combining the solutions to these problems, we're able to produce this)

Building the builder with
go build -o pipeline will result in a binary called pipeline.

We can then move this binary into our working directory. We also need to create an ignore file in our working directory to ignore the binary produced by the builder for our application.

The ignore file is simply:

pipeline
cicdexample

We ignore .git programatically, since it's never something we want to include.

We can finally run our builder

./pipeline

Making a change to the index.html of our app, should kick off an automated build and redeployment, exactly as we set out to do.

As you can see from this output:

shiv@shiv-Lenovo-ideapad-310-15IKB:~/Work/dev/go/cicdexample$ sudo ./pipeline
2019/12/24 18:23:42 map[]
2019/12/24 18:23:42 creating whitelist
2019/12/24 18:23:42 pipeline
2019/12/24 18:23:42 cicdexample
2019/12/24 18:23:42 builder
2019/12/24 18:23:42 building registry
2019/12/24 18:23:42 starting directory scan
2019/12/24 18:23:42 pipeline is whitelisted, not adding to registry
2019/12/24 18:23:42 computing hashes & creating map entries
2019/12/24 18:23:57 verifying hashes
2019/12/24 18:24:12 verifying hashes
2019/12/24 18:24:12 ./static/index.html old hashbebc0fe5b73e2217e1e61def2978c4d65b0ffc15ce2d4f36cf6ab6ca1b519c17new hash16af318df74a774939db922bcb4458a695b9a38ecf28f9ea573b91680771eb3achanged detected - updating hash, action required
2019/12/24 18:24:12 Taking action, running: ./build build docker deploy
2019/12/24 18:24:20 🤖    Welcome to the Builder builder v0.2, written by github.com/cishiv
⏲️  18:24:12: started build script...
🏗️ 18:24:12: building cicdexample
🐋    18:24:13: building image example:test
Sending build context to Docker daemon  10.11MB
...
Successfully tagged example:test
🌧️  18:24:19: deploying to Minikube
deployment.apps "example" deleted
service "example" deleted
deployment.apps/example created
service/example created
✔️  18:24:20: complete.
👋    18:24:20: exiting...

2019/12/24 18:24:20 
2019/12/24 18:24:20 --------------------------------------------------------------------------------
2019/12/24 18:24:27 verifying hashes

That's it! We're done. We've successfully written, a useful - albeit simple CI/CD pipeline to improve our development process.

What are the takeaways from this experience?

Tips

It might be tempting to use a pre-existing solution for this kind of pipeline, but my recommendation is that if you have the time and energy to write a small solution to a problem you’ve encountered, you definitely should. It forces you to think about how your application(s) work in a production setting as well as how your development process can be improved. It is also a lot of fun.

Extensions

There are many possible extensions to this project, some of which I want to tackle soon, I've listed some of the more interesting ones here:

Allow for the creation of build scripts via the builder app
Build from VCS (i.e git clone a repo and build it based on a job description)
A UI for this build pipeline
Run the builder app in Docker on Minikube itself
Host the builder app on a cloud platform and have configurable deployments and builds
Let users create .json job files for builds
Multi-language support

Caveats

This kind of quick and dirty pipeline isn’t better than a pre-existing solution, but it is a lot more fun to use, since I can quickly make changes to it based on a need I may have.

I am relatively new to Go, Kubernetes and Docker. So the style I used may not be best practise, but it does work for me.

Usage Examples

I am currently using a similar pipeline to automate the deployment of a project that I am working on called Crtx (http://crtx.xyz/) - into a test environment as I write code. It is primarily a Go code base with multiple apps that are deployed continuously to a Minikube cluster. This pipeline makes testing data flows between applications much easier.

Closing Remarks

This is the first time I’ve written a technical article and would love to hear feedback on the approach I took, as well as on the actual content!

I plan to write more about the tools I use and build, so if you enjoyed reading this please let me know!

You can give me feedback on Twitter or right here in the comments ✍️

Forem: Shivan Moodley

Using the IFS variable to do things with CSV files

The Problem

First Steps

Some Example Data

Enter bash and IFS

Generating SQL Inserts

Caveats

References

Finally published my personal portfolio/blog

What is your prototyping stack?

A 2019 retrospective and looking forward to 2020

How often do you do informal retrospectives?

Deploying to Heroku: Docker, Go and React

What is Heroku?

Use CRA to set up a React app

Write the Go backend to serve it

Build a production version of our application

Docker

Heroku

Create an app

The Container Registry

Building a simple CI/CD pipeline for local testing using Go, Docker, Minikube and a Bash script

Wat

That sounds cool, but why?

Feedback is important

Environment consistency trade-off

A simple start

What do you need to have?

Where do we start?

How exactly did you do that?

Creating the test application

Containerisation

Bashing things into shape

Minikube-ing

The Final Act: True Automation

Tips

Extensions

Caveats

Usage Examples

Closing Remarks

Build a `production` version of our application