Forem: James D.

Multi Stage Builds in Docker

James D. — Tue, 03 Aug 2021 02:12:26 +0000

Multi Stage Builds

Back Story

So way back in the beginning of this we briefly discussed layers. Each line in a dockerfile adds a layer. Which means it adds size to the dockerfile.

Lets look at when we added git to our first image.

RUN apk update && apk add --no-cache git

Its just git. No big deal right? It can't take that much space. But lets look at all the extra things that happened to add git. We ran apk update. This meas we reached out to the Alpine repos and fetched the basic details for every package in the apk repositories and saved them into our image. Then we did apk add --no-cache git. So we fetched the git package and any of its dependent packages and stashed them in our image so that we can install them. Then the installer unpacks those packages and installs the files where they are needed. Thats a lot, and thats only adding git.

So you have all these dependencies that you need to build your base image. Git, the dependencies to build your application, the compiler itself, and a bunch of other junk. We don't need all that in production. We really don't want all of that in production. Your dev dependencies are extra attack vectors. You aren't going to be pushing to git from your production container. So how do we get rid of all of that?

About Multi Stage Builds

This is where Multi Stage builds come in. Before Multi Stage Builds, you would have to do a bunch of funny shell tricks to clean up every layer as you used it just to keep the artifacts that you needed and get rid of everything else. Now with Multi Stage Builds you can declare the artifacts you want in each image. So lets start with an example

FROM golang:alpine as builder
# Install git.
# Git is required for fetching the dependencies.
RUN apk update && apk add --no-cache git
WORKDIR $GOPATH/src/integratnio/go-rest-api/
COPY go-rest-api .
# Fetch dependencies using go get.
RUN go get -d -v
# Build the binary.
RUN go build -o /go/bin/hello

FROM golang:alpine
# Copy our static executable from the builder.
COPY --from=builder /go/bin/hello /go/bin/hello
# Run the binary.
ENTRYPOINT ["/go/bin/hello"]

Alright. Looks good right? So far this is pretty simple. We used the same base image but we only copied our binary from the first image into the second. We could also use a different base image if we wanted. This is a real simple app with no external dependencies. So the fat trimmed is minimal. But pattern still helped us save some space.

REPOSITORY          TAG    IMAGE ID       CREATED          SIZE
helloFromIntegratnIO    dev    285c3af572b0   5 seconds ago    321MB
helloFromIntegratnIO    prod   0c2fb67c8edc   14 seconds ago   306MB

Just look at those savings. But we can go deeper, and we will in lesson 7.

Developing Inside A Docker Container

James D. — Thu, 29 Jul 2021 23:28:54 +0000

Now we are going to start getting into the weeds of leveraging docker. I think an easy way to start this one would be with a user story. So here we go.

As a developer I would like to have an environment to develop that alleviates the "Works on my box" problem. It should work just the same on my box as it does in production. I shouldn't be able to install something in my environment that doesn't exist when I deploy my app to production. Any other developer should be able to pick up my app and develop the exact same way without a huge time sink spent getting their environment to look just like mine.

Pipe dreams, am i right?

Lets see how we can pull this off with docker.

Most languages these days support some sort of live reload to allow developers to instantly make a change and have it running live for them to play with. So lets start out by adding that functionality to our little API.

Running The Container

Running this container so that we can actually work inside it with our editor is going to take a bit more than we've done before. We are actually going to leverage an image we haven't built ourselves this time. Lets take look at the image we are going to use. cosmtrek/air. Please take a minute to explore the image link if you haven't explored Dockerhub before. The Overview tab will give you a rundown of how to use the image. The Tags tab will show you different tags you can consume. So if we scroll down on the overview you will see how to run it The Docker Way. So lets do that.

First we need to pull the image down.

docker pull cosmtrek/air

If we wanted to consume a specific tag then we could do that as well. Just make sure you also use that tag in the following docker run command.

Lets add a little config for air so that it will cleanup for us when its done. In $projectRoot/go-rest-api/, lets add .air.conf and fill it with the following then save.

# .air.conf
# Config file for [Air](https://github.com/cosmtrek/air) in TOML format

# Working directory
# . or absolute path, please note that the directories following must be under root.
root = "." 
tmp_dir = "tmp"
[build]
# It's not necessary to trigger build each time file changes if it's too frequent.
delay = 1000 # ms
# Stop to run old binary when build errors occur.
stop_on_error = true
[color]
# Customize each part's color. If no color found, use the raw app log.
main = "magenta"
watcher = "cyan"
build = "yellow"
runner = "green"

[misc]
# Delete tmp directory on exit
clean_on_exit = true

Now lets run this image and load our code into on the fly

docker run -it --rm\
    -w /app/ \
    -v `pwd`/go-rest-api:/app/ \
    -p 8080:10000 \
    cosmtrek/air \
    -c .air.conf

Theres a lot going on there. So lets break it down.

docker run: We've seen this before. Just the base command to run a container

-it: This is the shorthand way to declare --interactive --tty that we discussed earlier

--rm: This will cleanup the filesystem that our container creates. Normally this persists. But we really don't want that happening. Whenever we start this we want make sure its fresh with just the code we are working with.

-w: Just like in our Dockerfile. This lets us declare WORKDIR

-v: This is short for --volume This will mount our $projectRoot/go-rest-api at /app inside the container. the pwd\ is just a little shorthand to get our present working directory. Otherwise we would have to declare the full path of what we want to mount inside the container.

-p 8080:10000 Port mapping just like before.

cosmtrek/air Declaring the image we want to use.

-c .air.conf Because we declared -it we are able to pass parameters to ENTRYPOINT of the container. We are leveraging this to pass in the configuration file we created.

You should have gotten an output like this.

$ docker run -it --rm\
    -w /app/ \
    -v `pwd`/go-rest-api:/app/ \
    -p 8080:10000 \
    cosmtrek/air \
    -c .air.conf

  __    _   ___  
 / /\  | | | |_) 
/_/--\ |_| |_| \_ // live reload for Go apps [v1.11.1]

mkdir /app/tmp
watching .
!exclude tmp
building...
running...
Starting Web Server
Preparing to handle requests
Ready for requests

So looking at this right off the bat. We see that it spun up v1.11.1. Thats old. Lets do this again using a version tag to run a newer version. (If you are doing this later. You may have gotten a different version. But at the time of writing. It looks like new images are being pushed but latest isn't being updated)

docker run -it --rm\
    -e GO111MODULE=off
    -w /app/ \
    -v `pwd`/go-rest-api:/app/ \
    -p 8080:10000 \
    cosmtrek/air:v1.15.1 \
    -c .air.conf

You'll notice this time. We skipped the pull step. Yay Shortcuts. There was one other little thing we added.
-e GO111MODULE=off: GO111MODULE in its most simplest terms tells go whether to look for and require go modules. Getting into all of that really isn't the point of this. But if you are curious, this blog was a great read.

But we are running again. And look. Our go version is more recent

$ docker run -it --rm\
    -e GO111MODULE=off \
    -w /app/ \
    -v `pwd`/go-rest-api:/app/ \
    -p 8080:10000 \
    cosmtrek/air:v1.15.1 \
    -c .air.conf

  __    _   ___  
 / /\  | | | |_) 
/_/--\ |_| |_| \_ v1.15.1 // live reload for Go apps, with Go 1.15.5

mkdir /app/tmp
watching .
!exclude tmp
building...
running...
Starting Web Server
Preparing to handle requests
Ready for requests

So lets see if our curl still works.

curl localhost:8080

Output:

$ curl localhost:8080
Hello From integratnio

Great the app is still working. Now lets make a code change and watch the live reload do its thing.

Making a change

Open up $projectRoot/go-rest-api/main.go. On line 11 you will see message declared. Lets change the value of that line and save the file. Feel free to change it to anything you want. But I'm going to change it to Yay we did it

Watch the terminal running your dev container when you hit save.

main.go has changed
building...
running...
Starting Web Server
Preparing to handle requests
Ready for requests

Magic.

Lets test that curl request again.
Output:

$ curl localhost:8080
Yay We Did It

Look at that. This is just one example of how to do this. For a node app you could use npm run dev:watch as your entrypoint and get a similar effect. Pick your code flavor and I bet you could figure out a way to do this.

Docker Training 4 - Managing The Build Context

James D. — Wed, 28 Jul 2021 13:22:57 +0000

So far we have created a small api that runs in a container. We've tested that its working. Worked with environment variables inside the container on the docker run command. Accessed a shell inside the container with the docker exec command. Now I want to talk about the docker build context.

Docker Build Context

The most common way to work with the build context is to build an image based on our current path. How we do docker build . -t repository:version We are doing this with a relatively small app. So we haven't gone over a few things that you will see in a larger project. One of the first things you will see in a larger project is a .dockerignore file. This is a lot like your .gitignore. We don't need every file in our repo to build our image. So why are we going to bring them into the build context. If you are doing a huge data science project with large sample model files. Or a financial model project with 100s of .csv files to test against. Now, I won't get into the argument about if that data really belongs in the repo. Thats not the purpose of this series. But if it is we don't need it to build our base image to run our app.

So take this structure for example:

myApp/
    .git/
    .github/
    docs/
        getting_started.md
        install.md
    src/
       myApp/
    scripts/
       install.sh
    dockerfile
    .dockerignore

Pretty simple app structure. What part of this do we really need to build this app? the src/myApp for sure. But the rest would all just be bloat to our build context.

Remember the output we had from our docker build command?

$ docker build . -t helloFromIntegratnIO
Sending build context to Docker daemon  144.4kB
Step 1/7 : FROM golang:alpine
 ---> 6af5835b113c
Step 2/7 : RUN apk update && apk add --no-cache git
 ---> Using cache
 ---> 80212a0a3d3b
Step 3/7 : WORKDIR $GOPATH/src/integratnio/go-rest-api/
 ---> Using cache
 ---> 75eb0766eb86
Step 4/7 : COPY go-rest-api .
 ---> 103a86626a1d
Step 5/7 : RUN go get -d -v
 ---> Running in 88aa3cef85b2
Removing intermediate container 88aa3cef85b2
 ---> 22e8e7d430e6
Step 6/7 : RUN go build -o /go/bin/hello
 ---> Running in e6c5ada5af3e
Removing intermediate container e6c5ada5af3e
 ---> ff7cb854f70f
Step 7/7 : ENTRYPOINT ["/go/bin/hello"]
 ---> Running in 1a4d5925b331
Removing intermediate container 1a4d5925b331
 ---> e8d948afae99
Successfully built e8d948afae99
Successfully tagged helloFromIntegratnIO:latest

Look at line 2 of that output.

Sending build context to Docker daemon  144.4kB

That happened so quick we didn't even talk about it before. When building your container. Everything in your local context will go into the build context unless you declaratively exclude it. Not a big deal when your repo is only a couple megabytes. But when you get into the Gigabytes and you are trying to speed up the build you will come back here and find a way to trim a couple minutes right up front.

So lets create a .dockerignore for the file structure example I posted earlier.

.git/
.github/
docs/
scripts/

There are other ways to pass a build context. You could pass a URL, a tar file and more. Docker has built great documentation around all of it here

Docker Training 3 - My First Docker Deployment

James D. — Mon, 26 Jul 2021 22:18:21 +0000

Lets recap where we are so far. We've gone over some of the basics of a Dockerfile. Talked a bit about some basic commands that will come in handy as we continue. Wrote a Dockerfile to build our little Golang API. Finally we have built an image and discussed tags.

Docker Run

So lets deploy this image and play with it a little bit. Open up a terminal and type this command.

$ docker run --publish 127.0.0.1:8080:10000/tcp helloFromIntegratnIO

Lets step through this command 1 piece at a time.

docker run: Here we are telling the Docker cli that we want to run an image as a container.

-publish: Publish will bind a local port to a port in the container.

127.0.0.1: Keeps us only exposing localhost, so our container won't be available outside of our machine.

:8080: is the local port on our host machine that we are binding to
:10000: is the port inside the container that we want 8080 to impersonate.

/tcp: is the protocol we want to use. You could also use udp or sctp if needed. You can learn more about that in the User Guide

If you haven't hit enter yet. Go ahead and hit it now and kick off that Docker run command.

You should see the following output.

$ docker run -p 127.0.0.1:8080:10000/tcp helloFromIntegratnIO
Starting Web Server
Preparing to handle requests
Ready for requests

Lets see if it works.

Open up a second terminal and enter:

$ curl localhost:8080

Output:

$ curl localhost:8080
Hello from Integratn.IO!!!

Now we are cooking. But what else can we do with this little container.

Lets change the response from the curl request. We can do this by passing an environment variable into the container. We can do this one of two ways. We can edit the docker file. Add the ENV instruction. Tell it it we want MESSAGE to equal something else. Rebuild our image. Then finally run it again. This is a great method for establishing default values for environment variables. Or we can add the parameter --env with a value to our docker run command and change it on the fly. Environment Variables passed during a docker run command will always trump what is in your Dockerfile. If you haven't exited the container already hit ctrl +c to exit the container from the terminal.

Lets run the following command:

docker run --env MESSAGE="Hello from outerspace" -p 127.0.0.1:8080:10000/tcp helloFromIntegratnIO

Now in another terminal lets use that curl statement again.

$ curl localhost:8080
Hello from outerspace

Look at that. We changed the response with the environment variable.

Docker exec

Lets look at the docker exec command while our container is running. Docker exec allows us to run commands inside our container while its already running. This can be very handy for debugging a container. But first we have to figure out what our container is named.

Lets look at our running container with a command we covered in lesson 1:

$ docker ps

You will get an output similar to this but your NAMES field will not be the same.

$ docker ps                     
CONTAINER ID   IMAGE              COMMAND           CREATED         STATUS         PORTS                       NAMES
56290825f889   helloFromIntegratnIO   "/go/bin/hello"   2 minutes ago   Up 2 minutes   127.0.0.1:8080->10000/tcp   hungry_bohr

Now that we have found our running container. Lets use the container name and execute a shell to get into our container.

docker exec --interactive --tty hungry_bohr /bin/sh

Output:

$ docker exec --interactive --tty hungry_bohr /bin/sh
/go/src/integratnio/go-rest-api #

Does that path look familiar? Its the same path we set for WORKDIR in the Dockerfile. We are in the shell of the container. Type exit and hit enter to leave the container.

Lets look at the command real quick.

docker exec is the base command. This is declaring that we want to execute something against a running container.
--interactive will keep STDIN open even if we aren't attached.
--tty gives us a pseudo tty.
hungry_bohr is the name of my running container. Yours will be different.
/bin/sh This is the command we want to run in the container. One caveat to this. Whatever you run against the container has to actually be part of the container. In an effort to trim containers down a lot of container images don't have a shell command available.

So now, we've ran our container. We've exec'd into our container so that we can explore it. Thats it for this time. Get ready for next time when we have a brief exploration into the build context.

My Git Worktree Workflow

James D. — Sun, 25 Jul 2021 21:01:34 +0000

I recently made a change to how I work on repositories when I have to work with others. It has been a life saver.

There is a TLDR example at the bottom to get you interested in the rest of the article.

Pain Points

Lets start with the normal git workflow and the pain points it causes. First you clone the repo. You create your branch and you start coding away. Then your buddy hits you up on Slack with a huge PR and scrolling the diff in Github isn't enough for you to approve it. Great. Now you have to stash the code you are working on, update your base so that you have their branch, and pull it all down. Now you can finally review the PR. This is nuts. Its annoying and you completely lose track of whatever you were just working on. I didn't even cover getting back to your branch and getting going again.

Painting a Picture With a Dev Story

As a developer I want to be able to review my teammates code without having to stash my work or maintain a second copy of the entire repository. I also want to be able to pause my work and work on a different branch again without having to stash my work or maintain a second copy.

Enter Git Worktree

First off, here is a link to the git worktree documentation. I won't cover every caveat. So check it out for a lot more info.

Git is capable of maintaining multiple working trees of a single repo. You heard that right. With git worktree you can check out more than one branch at a time and sensibly maintain each of those branches.

I had to rethink my folder folder structure when I started using worktree. It was weird at first. But its been amazing after I settled in.

When I clone a repository with worktree in mind I will do it a little differently.

$ git clone git@github.com:googleapis/python-tasks.git python-tasks/main

This is where the main branch lives. I really don't touch it except to fetch/pull and create new branches with worktree. Its my clean source of truth and I like to keep it that way.

From here I will create a new branch to for whatever I'm working on But I won't use git branch ${branchName} or git checkout -b ${branchName}. I'll do it with worktree while i'm in the main branch folder.

$ git worktree add -b my-awesome-branch ../my-awesome-branch main

Pay attention to that ../ If you forget it you will end up putting the branch in the folder where main lives and that can get real ugly, real fast.

So now we have:

~/Projects
  /python-tasks
    /main
    /my-awesome-branch

You can now list all of your working trees with the list argument.

$ git worktree list
~/Projects/python-tasks/main                63df2ef [master]
~/Projects/python-tasks/my-awesome-branch   63df2ef [my-awesome-branch]

I change to that directory. I do my work. Next thing I know I get a message from a teammate and he needs me to do that PR review we've been talking about. With git worktree its too easy. I just go back to main and add his branch to my worktree.

$ git worktree add --track -b add-appengine-flexible-tasks-samples ../ppr-review origin/add-appengine-flexible-tasks-samples

Preparing worktree (new branch 'add-appengine-flexible-tasks-samples')
Branch 'add-appengine-flexible-tasks-samples' set up to track remote branch 'add-appengine-flexible-tasks-samples'  from 'origin'.
HEAD is now at e2c8eee chore: generate noxfile.py for samples

There is a lot going on in that command. So check out the doc for the full explanation, but I'll try to break it down.

git worktree add: We've seen this. Its the base command to create a branch and a working tree

--track: This sets up tracking mode. We need it to track a remote branch

-b: This will create a new branch and we give it the same name as the remote branch because its the right thing to do. But really we could call it anything.

../pr-review The path we want it checked out to. Don't forget that ../

origin/add-appengine-flexible-tasks-samples Finally the <remote>/<branch> that we want check out.

Now if we do git worktree list We'll see our friends branch in our working tree ready for review.

$ git worktree list
~/Projects/python-tasks/main                63df2ef [master]
~/Projects/python-tasks/my-awesome-branch   63df2ef [my-awesome-branch]
~/Projects/python-tasks/pr-review           e2c8eee [add-appengine-flexible-tasks-samples]

Pretty sweet.

Now I've finished my PR review and I don't need the code anymore. so all we have to do from main is

$ git worktree remove pr-review

Poof, The folder is gone and we don't have to clog our working tree up with it anymore. Sometimes we get lazy right? maybe I did rm -rf ./pr-review. This is bad. I just broke my working tree. The folders gone but git still knows about it when I do git worktree list How can we fix this? Is it time to just throw the whole repo away and clone again? What do we do?

From main as usual:

$ git worktree prune

And now our mistake has been removed from the git and we can continue on about our day.

There are a few other worktree commands out there. Checkout lock and unlock if you need to put a working tree on a thumb drive or a network share thats not always mounted. That will protect it from prune. This was a long one. If you stuck around and found it useful, feel free to hit me up on my socials to tell me about it or teach me a new trick you found thats even better.

TLDR Example

$ git worktree add -b emergency-fix ../temp main
$ pushd ../temp
# ... hack hack hack ...
$ git commit -a -m 'emergency fix for boss'
$ popd
$ git worktree remove ../temp

My Hugo+Netlify Setup

James D. — Sat, 24 Jul 2021 17:40:16 +0000

So here is how I got my blog all set up. It was a lot of back and forth on how I set this up. So this post is for those that read it. But also for me. It was a lot of work and I want to capture what I did so I can reproduce it later.

This is by no means a complete set of instructions that you can iterate through. But more of a flow. I try to call out the things that were pain points for me directly though.

Setting up Hugo

First thing I did was install hugo. You can find instructions for it here. After I had it installed I used hugo new site ${siteName} to create my site.

Version Control is Key

After that I wanted to have git setup so that I had solid version control incase I broke anything. git init solved that really quickly. Then I created my repo on Github. After you do that it will give you instructions on how to create your first commit and change your origin for the remote and push your code.

Gotta Make it Pretty

Next up is adding the theme. You can find a lot of them on the Hugo site. Use git submodule add ${url to theme repo} themes/{theme-name} to put it in place.

After you get your theme in place you will need to update your config.yaml to actually use it. You would do this by adding a line like:

theme: "anubis" # This is the name of the folder in the themes directory

Deploy It

Now I have everything installed. Lets see it run. To run it locally to make sure everything works you can use hugo serve. But if you really want to get going then you will need to deploy it somewhere online. I put mine on netlify as I knew the name already and the process was simple enough.

Create your account on netlify. After your account is created you can create a new site with this magic button

After this you will go through some steps to connect it to your github repository and a deploy should be triggered. When its all said and done you should have your new site up and ready to go.

This includes setting up your netlify.toml

[build]
publish = "public"
command = "hugo --gc --minify"

[context.production.environment]
HUGO_VERSION = "0.85.0"
HUGO_ENV = "production"
HUGO_ENABLEGITINFO = "true"

[context.split1]
command = "hugo --gc --minify --enableGitInfo"

[context.split1.environment]
HUGO_VERSION = "0.85.0"
HUGO_ENV = "production"

[context.deploy-preview]
command = "hugo --gc --minify --buildDrafts --buildFuture -b $DEPLOY_PRIME_URL"

[context.deploy-preview.environment]
HUGO_VERSION = "0.85.0"

[context.branch-deploy]
command = "hugo --gc --minify --buildDrafts --buildFuture -b $DEPLOY_PRIME_URL"

[context.branch-deploy.environment]
HUGO_VERSION = "0.85.0"

[context.next.environment]
HUGO_ENABLEGITINFO = "true"

Adding Netlify CMS

You can find the getting started documentation here. You'll notice we didn't use the Hugo Template that they offer. That thing is outdated and I was unable to get it to work well with a theme for local development. It was much easier to add it to an existing site. Read that doc. It will answer most of your questions. But here is my config.yml (make sure you use .yml and not .yaml. It only looks for the one).

backend:
  name: git-gateway
  branch: main
  repo: jamesattensure/integratn-io

publish_mode: editorial_workflow
media_folder: "static/images"
public_folder: "/images"

collections:
  - name: "posts"
    label: "Posts"
    folder: "content/posts"
    create: true
    slug: "{{year}}-{{month}}-{{day}}-{{slug}}"
    fields:
      - {label: "Draft", name: draft, widget: "boolean", default: true}
      - {label: "Layout", name: "layout", widget: "hidden", default: "post"}
      - {label: "Title", name: "title", widget: "string"}
      - {label: "Publish Date", name: "date", widget: "datetime"}
      - {label: "Categories", name: "categories", widget: "list"}
      - {label: "Tags", name: "tags", widget: "list"}
      - {label: "Series", name: series, widget: "list"}
      - {label: "No Comment", name: "nocomment", widget: "boolean", default: false}
      - {label: "Body", name: "body", widget: "markdown"}

Your collections are built based on your hugo archetypes. There are a couple extra widgets that I don't use. Those can be found in the widget doc

Make sure you get through the whole doc and complete the authentication steps or anyone will be able to access the cms features.

Additional Things to Setup

Analytics

Pay for netlify analytics or add Google analytics. Most themes have Google analytics baked in and you just have to add the params to your config.yaml to connect it. Here are the two posts I followed to get it setup. cloudywithachanceofdevops.com and gideonwolfe.com.

Make sure you filter out your connections to your site so that you don't skew your own analytics. This blog post from daniloaz.com really helped get me through it. I went with the second method of filtering out my public IP as internal traffic.

Search

After you have Google analytics setup. You can setup Google Search to the same property that you setup in google analytics. It helps to have a sitemap.xml to add. The easiest way I found to do that was to use the sitemap netlify plugin I recommend just installing it from the plugins tab on your Netlify site. Much easier to manage.

Additional Plugins

Here are a couple other plugins I installed.

Hugo cache resources:
This plugin caches the resources folder after build. If you are processing many images, this would improve build duration significantly.

Submit sitemap:
This plugin will notify the search engines after every production build about your latest sitemap. The plugin can be used without any configuration if using the defaults.

Final Thoughts

There is a lot to learn when getting started with blogging. At the time of posting this I've only been at it a couple weeks. I spent most of that time bouncing through various starters and trying different things out to figure out what worked and what didn't. Hell, a week of that was probably just trying to pick a theme that I could live with until I understand enough to build my own theme. I hope this helped get you up and running. Enjoy and happy blogging.

Docker Training - My First Docker Build

James D. — Sat, 24 Jul 2021 17:37:25 +0000

The goal of this project is not to learn how to build APIs in Golang or anything extra. If you want to learn more about the snippet below feel free to dig through the Go Docs. Now lets create files and get started.

To get started we need to create a project to store our app we are going to work with. Create the following structure. We'll keep filling more in as we go later.

DockerTraining/
    go-rest-api/
        main.go
    dockerfile

Initialize the folder as a go project

$ GO111MODULE=on go mod init helloFromIntegratnIO

Add the following code to main.go

package main

import (
    "fmt"
    "log"
    "net/http"
    "os"
)

func helloFromIntegratnIO(w http.ResponseWriter, r *http.Request) {
    message := "Hello From Integratn.IO"
    if m := os.Getenv("MESSAGE"); m != "" {
        message = m
    }
    fmt.Fprintf(w, "%s\n", message)
    fmt.Println("Endpoint Hit: helloFromIntegratnIO")
}

func handleRequests() {
    http.HandleFunc("/", helloFromIntegratnIO)
    log.Fatal(http.ListenAndServe(":10000", nil))
}

func main() {
    fmt.Println("Starting Web Server")
    fmt.Println("Preparing to handle requests")
    fmt.Println("Ready for requests")
    handleRequests()
}

Not a whole lot going on up there. We create a ResponseWriter that prints some text whenever you hit the app on port 10000.

Lets get our dockerfile filled in.

FROM golang:alpine
# Keep older versions of Go behaving the same as the latest
ENV GO111MODULE=on \
    CGO_ENABLED=1
# Install git.
# Git is required for fetching the dependencies.
RUN apk update && apk add --no-cache git
WORKDIR $GOPATH/src/integratnio/go-rest-api/
COPY go-rest-api .
# Fetch dependencies.
# Using go get.
RUN go get -d -v
# Build the binary.
RUN go build -o /go/bin/hello
ENTRYPOINT ["/go/bin/hello"]

Looks familiar, right?

Lets step through this one more time. We declared our base image with the FROM instruction. Then we use the RUN instruction to install git. Used WORKDIR to change the path we are working from. Copied in our code with the COPY instruction. Used the RUN Command again to get any possible dependencies that were needed for our program. One more final use of RUN to build our app into an executable. Then finally we declared our ENTRYPOINT with the final instruction pointing to where our app is located.

Now we are ready to finally step through a build. Lets issue our first docker command.

Run $ docker build . -t helloFromIntegratnIO

Lets look at the output.

✭ ᐅ docker build . -t helloFromIntegratnIO
Sending build context to Docker daemon  84.48kB
Step 1/7 : FROM golang:alpine
alpine: Pulling from library/golang
596ba82af5aa: Pull complete 
344f2904b0c6: Pull complete 
d3bda26d9fa1: Pull complete 
24e1a14bb4a2: Pull complete 
f0b175b107d5: Pull complete 
Digest: sha256:07ec52ea1063aa6ca02034af5805aaae77d3d4144cced4e95f09d62a6d8ddf0a
Status: Downloaded newer image for golang:alpine
 ---> 6af5835b113c
Step 2/7 : RUN apk update && apk add --no-cache git
 ---> Running in db684e4aab79
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
v3.13.0-114-g822507f819 [https://dl-cdn.alpinelinux.org/alpine/v3.13/main]
v3.13.0-115-g41ee0c8f55 [https://dl-cdn.alpinelinux.org/alpine/v3.13/community]
OK: 13880 distinct packages available
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/6) Installing brotli-libs (1.0.9-r3)
(2/6) Installing nghttp2-libs (1.42.0-r1)
(3/6) Installing libcurl (7.74.0-r0)
(4/6) Installing expat (2.2.10-r1)
(5/6) Installing pcre2 (10.36-r0)
(6/6) Installing git (2.30.0-r0)
Executing busybox-1.32.1-r0.trigger
OK: 19 MiB in 21 packages
Removing intermediate container db684e4aab79
 ---> 80212a0a3d3b
Step 3/7 : WORKDIR $GOPATH/src/mypackage/myapp/
 ---> Running in 60c0c79d0d5c
Removing intermediate container 60c0c79d0d5c
 ---> 29ab2f51aa11
Step 4/7 : COPY go-rest-api .
 ---> 7ded2d7874e0
Step 5/7 : RUN go get -d -v
 ---> Running in da6e8b285779
Removing intermediate container da6e8b285779
 ---> f7eb2d97bd6f
Step 6/7 : RUN go build -o /go/bin/hello
 ---> Running in 0e503ced7243
Removing intermediate container 0e503ced7243
 ---> 21bbd2acaf3c
Step 7/7 : ENTRYPOINT ["/go/bin/hello"]
 ---> Running in 42165a2c670f
Removing intermediate container 42165a2c670f
 ---> 86fd2d2eb271
Successfully built 86fd2d2eb271
Successfully tagged helloFromIntegratnIO:latest

So whats happening here?

Step 1/7: Our FROM instruction is consumed and we see it fetch all the layers needed to build our base image.

Step 2/7: RUN executes and there is some normal apt output while it installs git and the required dependencies.

Step 3/7: We see WORKDIR set our new path that we are "working in".

Step 4/7: COPY pulls our source code into the image.

Step 5/7: RUN uses the go executable to fetches dependencies for our app. Something to notice here. We never installed Go locally. All of this runs in the build container. As long as the base image you work from has the binary you need or you add it to the container some how. You can run whatever you need in the container without worrying about something locally on your machine getting in the way.

Step 6/7 RUN uses the go executable again to build the app.

Step 7/7 ENTRYPOINT declares the what to start with the container.

The image is built. But where did it go? Lets use docker image ls to find it

ᐅ docker image ls
REPOSITORY          TAG       IMAGE ID            CREATED             SIZE
helloFromIntegratnIO    latest    dee090a053a3        About an hour ago   321MB

Heres how this output breaks down:

REPOSITORY: The name we gave our image.

TAG: latest is always the most recently built version of the image.

IMAGE ID: This is a unique hash that identifies our image

CREATED: How long ago the image was created

SIZE: The size of our image.

Lets take a quick second and talk about tags. Tags are important. They are a great way to reference what version of our image is running. They help us be declarative in what we run in any environment. So, lets create an app with an actual tag instead of latest.

ᐅ docker build . -t helloFromIntegratnIO:dev
Sending build context to Docker daemon  100.9kB
Step 1/7 : FROM golang:alpine
 ---> 6af5835b113c
Step 2/7 : RUN apk update && apk add --no-cache git
 ---> Using cache
 ---> 80212a0a3d3b
Step 3/7 : WORKDIR $GOPATH/src/integratnio/go-rest-api/
 ---> Using cache
 ---> 75eb0766eb86
Step 4/7 : COPY go-rest-api .
 ---> Using cache
 ---> 9ea13e96b401
Step 5/7 : RUN go get -d -v
 ---> Using cache
 ---> cf8a5cf85856
Step 6/7 : RUN go build -o /go/bin/hello
 ---> Using cache
 ---> e4e107828679
Step 7/7 : ENTRYPOINT ["/go/bin/hello"]
 ---> Using cache
 ---> dee090a053a3
Successfully built dee090a053a3
Successfully tagged helloFromIntegratnIO:dev

Wait a second. This output looks different. What happened?

Here is one of the advantages of Docker. Your Docker cache maintains a history of your images and the layers used to build those images. If your dockerfile and you code haven't changed it will use a cached image to build that layer. Since all we did was retag the image it just consumed the cache all the way down.

Lets look at our images again.


/Volumes/CaseSensitive/docker-training (main ✘)✹ ᐅ docker image ls                              
REPOSITORY            TAG        IMAGE ID            CREATED             SIZE
helloFromIntegratnIO  dev        dee090a053a3        About an hour ago   321MB
helloFromIntegratnIO  latest     dee090a053a3        About an hour ago   321MB
<none>                <none>     86fd2d2eb271        About an hour ago   321MB

So, we have the dev tag we just created. But lets look at the IMAGE ID. The dev tag and the latest tag have the same IMAGE ID. This is what we want because it is the most recently created image.

So what is this <none> image? If you scroll up. You will see that the <none> IMAGE ID matches the IMAGE ID from our first build. This shows you what happens when you don't properly tag your images. If you built another image using the dev tag. The same thing would happen to that image. But this is local testing and not a production release. So no big deal.

Docker Training - Getting Started

James D. — Thu, 22 Jul 2021 00:27:55 +0000

Before we can really dive into this project we need take care of a few things. First we need to get our environment setup to run docker. Then we need to go over a couple things about Dockerfiles and the Docker CLI

Installing Docker

Mac

Mac install is pretty straight forward. Follow the instructions below

Instructions

Windows

There are 2 ways to install docker on windows. 1 is supported with a Hyper-V Backend and the second is supported with a WSL2 backend. Windows home users have to use the WSL2 backend method. Pro, Education, and Enterpise can use either.

WSL2 Instructions

Hyper V Instructions

Linux

Like all things linux. Docker setup is a bit more manual. You will have to install both Docker and Docker Compose to get through these lessons. Please find the directions for your preferred distro here. You will also have to run the Post-Installation steps or run all your docker commands as the root user.

Docker Install
Docker Compose Install
Post Installation

What is a dockerfile

Lets get started on understanding a dockerfile. A dockerfile is the instruction set that is used to build a docker image. Each instruction within the dockerfile will create a new layer or intermediate image. These layers are like commits in a git repository. Any of the layers is a fully built image that could be ran as a Docker container. Here is a basic example of a dockerfile for Go.

FROM golang:alpine
# Install git.
# Git is required for fetching the dependencies.
RUN apk update && apk add --no-cache git
WORKDIR $GOPATH/src/mypackage/myapp/
COPY . .
# Fetch dependencies.
# Using go get.
RUN go get -d -v
# Build the binary.
RUN go build -o /go/bin/hello
ENTRYPOINT ["/go/bin/hello"]

Lets step through the various instructions and see what they do. I have linked all of the instruction identifiers below. There are many others and can be found in the Dockerfile Builder Documentation

The FROM is an instruction that initializes a new build stage sets a base image. This is the container that will run while we build our image. Right now we are consuming the Golang image from Alpine. We chose this image because it is lighter and faster than the official Golang image. You can find more about Apline Linux and their images from here. There are many other images to choose from and they can be found at Docker Hub.

The RUN is how we execute a shell command inside the container that we are building from. This instruction will execute any commands that are passed to it.

The WORKDIR instruction is used to change the working directory within the project. Much like how the command cd is used in a shell. If the directory doesn't exist it will be created even if it is never used in any other Dockerfile instructions.

The COPY instruction will copy everything from the declared path relative to your project to the destination of where it will live inside your image. In this case we are copying everything from the base of the project directory to $GOPATH/src/mypackage/myapp/. The first . represents your project directory. The second . represents the WORKDIR we set earlier.

The ENTRYPOINT instruction is executable that will be ran when the container is started.

Getting started with the Docker CLI

The Docker CLI is the main way we will interact with Docker. We will use it to build Docker images, run Docker containers and see whats going on in our Docker environment.

See the help menu:

docker help

For any docker command, use docker help $command to learn more about the command and the options that are available for it. I highly encourage you run this on ever command below. There are many options that you can pass to commands and it will list the aliases that can be used to make the commands shorter and save save some typing

Building an image:

docker build . -t $MYIMAGENAME:$MYIMAGETAG

Usage: docker build [OPTIONS] PATH | URL | -

Generally when building an image you are in the root directory of your project. That directory also contains the Dockerfile that will be used to build the project. It is also important to tag your images so that you can find them in the future. This is the purpose of the -t in the command above. If you don't when you search through your images you will only see None as the image name.

Listing images in your local registry:

docker image ls

List running containers

docker container ls

List all containers (Including exited)

docker container ls --all

Get logs from container

docker logs $CONTAINERNAME | $CONTAINERID

This is very helpful if you have a container that keeps crashing on startup.