Forem: Eric Smalling

The Docker project turns 10! Looking back at a decade of containers

Eric Smalling — Fri, 17 Mar 2023 16:30:00 +0000

The Docker project turns 10! Looking back at a decade of containers

March 15, 2023 marked the 10-year anniversary of Solomon Hyke’s famous PyCon lightning talk, when he introduced the world to Docker.

Let’s look back at how much has changed and hear from some folks who have stories about blazing the trail toward the containerized world we live in today. I opened up my little black book of container and cloud-native contacts and asked them to share stories about their introduction to Docker, and any interesting tales from the trenches over the decade since we first met Moby and Molly.

Hello wowrld!

Solomon Hykes speaking at PyCon 2013

In 2013, the world was introduced to Docker. For many developers, it was our first experience with the concepts of c-groups, namespaces, and other Linux technologies used to “contain” processes. As the Docker folks are fond of saying, they “democratized” container technology, making it simple to use without a degree in Linux systems administration.

Mind-blowing and Magical

Some adjectives seemed to come up repeatedly when I asked people about their first experiences with Docker; words like magical, mind-blowing, and “ah-ha moment.”

Nirmal Mehta speaking at DockerCon 2015

Picture it! Portland Oregon, 2013, at the Oreilly OpenSource Conference. This was a few months after Docker open source came to life, and it was gaining popularity already within various IT communities: open source, cloud, devops etc. It was the last day of the conference and I attended a Friday early morning session about this new thing called Docker! Solomon Hykes was the speaker and he dove right into the container layer concepts, explaining the capabilities and underlying technologies. He then finished his presentation with a demo, where, if my memory serves me, he proceeded to spin up 5, 10, 15, 20! apache/httpd containers within seconds! Even to the small, bleary eyed crowd that morning and especially to me, it was mind blowing … I instantly had the feeling that I was witnessing the start of a paradigm shift and I wanted to know everything about it!

I know Docker wasn’t the first to pull the underlying technologies together to create isolated processes/containers, but the user experience demonstrated that day — and carried through to today — was magical.

- Nirmal Mehta, Principal Specialist SA at AWS

Brandon Mitchell bundled up with Docker swag

I remember using Docker the first time, after going through the process to setup a VM to run nginx with tools like Ansible and Chef. I’d spend an hour coming up with a playbook, configure it for the VM that was spun up, and then wait for it to run the install. Then I ran the Docker command to start nginx, and less than 30 seconds later, it had started the process and run it in this magical isolated environment. That was my hook that started the adventure to discover what just happened and how Docker works.

- Brandon Mitchell, Solutions Architect at BoxBoat, an IBM Company

Discovering Docker was like discovering magical powers. This had happened when virtualization took the place of servers, giving me the power to pack VMs into physical infrastructure and get the most use out of it. Docker was like descending into another dream level — now I could pack applications into VMs and get even better use out of the hardware.

- Adrian Goins, Retired Developer Advocate at Rancher/SUSE

Andy Clemenko, early Docker adopter and field engineer

In early 2015 I was asked by my employer to become a “Docker” expert. Knowing very little, other than a few Orange site posts, I started playing with it. As with everyone, I started with the classic Docker hello world example of Nginx. For me the “ah ha” moment was instant. Having been a long time system admin, I saw the process isolation as a massive leap forward. From that day forward I shifted my entire career focus towards containers. I was lucky to help the US Government start adopting containers across quite a number of agencies. After a year and a half I was fortunate to be able to continue the Docker journey by joining the company.

- Andy Clemenko, Field Engineer at Rancher Government Solutions

Doomsday scenario

David Flanagan, a year-one Docker adopter, quickly saw how it could revolutionize deployments at his company.

David Flanagan, Founder Rawkode Academy and fun dad

I first heard of Docker from Solomon’s PyCon demo in 2013. At the time I was working for a UK radio and magazine company as the Director of Development and trying to help them bring their business into the 21st century… digital transformation, right? Their biggest problem was scale, and we actually had a “doomsday” scenario of “How do we handle Lemmy from Motorhead dying?”. Our load was extremely predictable, until it wasn’t. You can’t predict the news and you have to scale in real-time as quickly as possible.

We had used Vagrant and VMs for a while, but scaling those quickly is painful. We had to over provision early and dial back fast when the “event” was over in-order to reduce costs. So when we saw Docker it was a light bulb moment. Except… Docker didn’t have “docker build” back then… However, it followed shortly after and they adopted the tag line still used today, “Build. Ship. Run”. They didn’t just solve the runtime problem, they made the DX of building container images incredibly simple AND provided distribution (ship). Kubernetes and Cloud Native owes the early dotCloud team a huge debt of gratitude, regardless of which container runtime we may be running today.

- David Flanagan, Founder, Rawkode Academy

Stabilizing the grid

I, too, have been using Docker since the very early days and — if I may be so bold — here’s how it went for me.

James Spurin’s twitter screenshot: himself, Eric Smalling, Bret Fisher, Chad Crowell, Kunal Kushwaha and Ramesh Kumar at Kubecon 2022

I started playing with Docker in late 2013, using it in our CI pipelines where we were tasked with running hundreds of functional end-to-end tests against every commit of a large e-commerce web application. We often were running upwards of 2000 browser instances via VM-based *Selenium2 Grids** during the peak hours of the day, and the bane of our existence were flaky test results due to grid stability issues. We were constantly spinning up and down instances in the cloud and tried multiple different strategies over the years to improve start-up times, maintain stability, and minimize costs. We even dabbled with spot instances and got into bidding wars in various regions!*

We had already been experimenting with using Docker for running the web app and test suite controllers, but the lightbulb moment for me was when I realized that I could build an image with the browser and xvfb in it and start it instantly. This allowed us to run stable Selenium grids with as many browsers as we could fit onto a single node! (Our grid stability issues usually stemmed from the node’s inability to scale up many browsers per node, thus requiring many small nodes with small numbers of browsers each.) This was so successful that we largely were able to abandon the cloud-based instances and use just a few local VMs for these grids, saving thousands of dollars per week.

- Eric Smalling, Sr. Developer Advocate at Snyk

Convincing the masses

To anyone new to enterprise software development, the advantages of containers and the ecosystem that evolved around them might seem a forgone conclusion, but for the first 5 or 6 years, Docker’s future was far from certain.

Results speak for themselves

Change is often hard for humans, but Docker’s demonstrable benefits sold it easily:

I had a history of bringing in new shiny tech, so there was always an air of frustration when “Dave’s got another toy,” and the majority of my team were Mac users, so they definitely hated me!

But Docker was mostly server-side for us, there wasn’t boot2docker, yet so the team continued with Vagrant for dev, and we brought it into that environment eventually when the story kind of told itself. The team saw how it simplified our deployment pipeline, and it won them over.

It was hard to argue against when our deploys went from 40 minutes to about approximately 3!

- David Flanagan

Sevi Karakula making shift happen!

When I first encountered Docker, I had been working as a software developer long enough to know how hard it can be to get a new member onboarded to the team with all of the tools they needed. Also, whenever we had tried to introduce new frameworks and languages into the development environment — to make everything eventually a bit lighter — it was often a pain because the usual reaction from most of the developers would be an eye-roll.

Docker made our lives so much easier because it was just this one magical tool to teach the team, and then they’d the benefits immediately without having to deal with all the complexity to install and configure them. It was such a big moment of developer liberation, not having to install every tool on your tightly monitored machine, not having to chase requests to get approved and not having to think about licensing. If there is an image, you were free to go.

- Sevi Karakulak, Engineering Lead at Container Solutions

Not “enterprise ready”

The term “enterprise ready” is subjective, and can mean different things for every company you deal with. Matt Bentley recounts an interesting interpretation of when, as a Solutions Engineer for Docker, he had to address how “attractive” Docker was — or, in this case, wasn’t — in the early days.

Matt Bentley at DockerCon 2019

… the customer loved Docker as a technology, [and] they loved the Docker Trusted Registry product, but until we had something that wasn’t just API and command line based, their leadership would never see it as being enterprise ready.

They had a process where they needed to see the products demonstrated to them internally, and if they showed what I did with a CI/CD pipeline built in Jenkins to take code, build & test it in a container, deploy it, promote the image, they wouldn’t understand it because enterprise-ready solutions had pretty user interfaces.

- Matt Bentley, Manager, Solutions Engineering at VMware

A more common hesitancy heard often was about the relative immaturity of the project. Even though the underlying technologies had been around for many years, hitching your wagon to an open-source project spun out of a startup, with cartoon mascots and a pet turtle doing deployments, was a bridge too far for many.

Rachel Leeken and Eric Smalling at Kubecon EU 2022

I started learning about Docker in 2015 and based on the potential I saw, I proposed it might be worth looking into for modernization instead of an older, expensive vendor’s solution at the time. The customer didn’t go with it because they said there were not sure about container technology and if it would even be around for the 5 years of the contract.

- Rachel LeeKin, Containers Specialist SA at AWS

Self-inflicted wounds

Sometimes getting a team to adopt containers was not the hardest task; it was coaching them to do it right.

Adrian Goins and the Iron Condor, his paramotor quad

In the beginning, it was hard to convince my clients to do it… Those who did get the idea didn’t always get the benefits or how to leverage a container. I remember one client who had containers that they would shell into, where they’d recompile their application or its dependencies and then commit the container like it was a code repository. Their operational containers were huge and probably more fragile than their original non-container infrastructure.

- Adrian Goins

Interest grows

As the years went on, more and more people started to see the potential containers provided, especially as orchestrators matured — although they brought challenges of their own. Adrian Mouat (@adrianmouat | @adrianmouat@hachyderm.io), author and another OG Docker Captain, recalls the experience of the early DockerCon conferences and the flurry of activity as more and more people started to dip their toe in the water and companies started offering solutions to address their needs.

Adrian Mouat with Betty Junod, Eric Smalling, and Matt Jarvis at KubeCon NA 2022

One thing I really remember is how speakers (myself included) would ask, “Who’s using Docker?” in 2014. A forest of hands would go up. “Who’s using Docker in production?“ Pretty much all the hands would go down.

Having said that, the number of people using Docker in prod before it even went 1.0 (Oct 2014) and declared production ready was astounding.

We used to use shipping metaphors and talk about how it solved the “but it works on my machine problem” — unfortunately, k8s came along and recreated that one.

When I was at Container Solutions, we helped organize the first *Docker Con EU** at the NEMO Science Center. The buzz was pretty amazing; everyone knew they were at the start of something big. CoreOS were there (launching Rocket!), Alexis Richardson was pushing Weave networking, Luke Marsden was running ClusterHQ and the flocker data manager, Timo Derstappen was doing amazing stuff with Giant Swarm. Enterprise companies were everywhere just trying to figure out what the hell was going on.*
- Adrian Mouat, Product Manager at Chainguar

Bret Fisher wins “Tip of the Captains Hat” at DockerCon 18 Europe

I saw Solomon’s PyCon 2013 talk and tried docker to improve our Node.js testing in 2014, and I just didn’t get it. I kept trying to shove a whole server in a container image (and failing), and the networking was complete voodoo to me. It was such a huge stack of new automation magic and concepts that I gave up and returned six months later. This time, it clicked and hit me like a train. The 1–2–3 combo of building images, storing them in registries, and running containers from them blew my mind, and I never looked back.

- Bret Fisher, DevOps dude & creator of Docker Mastery

The orchestrators

If Docker’s announcement at PyCon 2013 is the spark that lit the fire, the advent of container orchestration platforms are the winds that ignited the forest. Mesosphere, Rancher, Swarm, Nomad, and Kubernetes were the “killer apps” that really pushed containers over the edge. Volumes have been written about the pros and cons of each platform, but it can’t be overstated how important they have been to the mass adoption of containers.

Kubernetes is the dominant platform today but there is still a very loyal community of Swarm users, and Nomad has its pockets of popularity too. Mesos predates Docker, and I’m sure it still has a fair number of users as well. For some, Kubernetes seemed overly complex but as its market share bears out, most people came around to adopting it.

When Kubernetes came along, I hated it. It was another layer of abstraction that didn’t provide a lot of additional benefit…until it did. Then I loved it because now I could take those servers, with their VMs, and their containers, and make a little mini datacenter out of it. The fact that a process existed to babysit the thing 24x7 meant that I was free to go do something else.

- Adrian Goins

While Kubernetes is now the de facto standard way to deploy containers, it is interesting to see the evolution of the space and how there are so many abstractions being created around it.

When it became clear that orchestrators were going to be key to container adoption at scale, I thought that there would be room for multiple major orchestrators to be successful. Each was good as it’s own thing and I saw each as having ideal use cases. The thing I loved about Swarm was that I could take someone who understood simple docker run commands or could put together an easy to understand Docker Compose file and get a Swarm service deployed in minutes. The simplicity of the docker CLI and tools like Docker Compose were brilliant because if someone knew how to run containers on a single host with either, it wasn’t a big lift to get them orchestrating containers. Now the industry is working to abstract Kubernetes away from developers because it took too much focus away from developers developing and put it on orchestration. Time spent by developers on orchestration means less focus on delivering business value through the apps they’re working on.

- Matt Bentley

Career-changing and life-changing

Moby, the Whale at Docker’s former SFO office

The common thread among everyone I reached out to for this article was how the Docker project and the community that has grown up around it have changed the careers and, indeed, livelihoods of so many people.

I knew it was the next evolution in infrastructure. I was obsessed and shifted my whole career to nothing but containers.

- Bret Fisher

From that day forward I shifted my entire career focus towards containers… To this day I am still on the mission of educating and leading the US Government on their container journey. It is amazing to see such a transformative technology last for 10 years.

- Andy Clemenko

The more I learned, the more I got hooked by the possibilities containerization offered. Eventually, I started teaching Docker and decided to make a pivot in my career towards containers and then Kubernetes. Looking back now, I can safely say that Docker changed my life.

- Sevi Karakulak

Snyk loves open source

We applaud the Docker project for a decade of transformational work, as evidenced by the powerful testimonies of those quoted here as well as the millions of developers who build, ship, and run in containers around the world every day.

Snyk was founded in 2015 with the goal of empowering developers to code and use open-source software securely — including the containers you put them in. Because we believe so strongly in the power and importance of the open-source development model, we have offered our scanning tools to individual developers free of charge from day one.

Keep your open source dependencies secure

Snyk provides one-click fix PRs for vulnerable open source dependencies and their transitive dependencies.

[Start free with Github](https://app.snyk.io/auth/auth0/github) | [Start free with Google](https://app.snyk.io/auth/auth0/google-oauth2)

In addition, for open-source project maintainers we offer expanded capabilities. See https://snyk.io/open-source-projects/ for program details and to submit your projects for consideration.

Attribution

Thanks to all who contributed to this article!

Nirmal Mehta

Principal Specialist Solutions Architect at AWS
Docker Captain 2016 — Present.
LinkedIn
@normalfaults
hachyderm.io/@nirmal

Brandon Mitchell

Solutions Architect at BoxBoat, an IBM Company
Docker Captain 2018 — Present
OCI image-spec maintainer.
LinkedIn
@sudo_bmitch
@bmitch@fosstodon.org

Adrian Goins

Retired Developer Advocate at Rancher/SUSE
**content creator and aviator
long-term Rancher/SUSE Advocate
LinkedIn
@creator_aviator

David Flanagan

Andy Clemenko

Field Engineer at Rancher Government Solutions
former Docker SA and SE
LinkedIn
@clemenko
@clemenko@hachyderm.io

Sevi Karakulak

Engineering Lead at Container Solutions
LinkedIn
@sevikarakulak

Matt Bentley

Manager, Solutions Engineering at VMware
LinkedIn
@matthewbentley
@mbentley@hachyderm.io

Rachel Leekin

Containers Specialist Solutions Architect at AWS
LinkedIn
@Rachel_LeeKin

Adrian Mouat

Product Manager at Chainguard
Docker Captain 2016 — Present
author: Using Docker (O’Reilly Media, 2016)
LinkedIn
@adrianmouat
@adrianmouat@hachyderm.io

Container images simplified with Ko

Eric Smalling — Mon, 10 Oct 2022 14:24:49 +0000

In a previous article, I wrote about how — and why — you might want to use the Google Open Source group’s Jib tool to build your Java application container images. Jib builds slim, JVM-based, OCI-compliant images that follow best practice guidelines without the need for a container runtime like Docker, and it removes the need to write and manage Dockerfiles. What if you are building Go applications, though? Well, there is another open source tool for Go that works similarly called Ko.

Note: During the drafting of this article, the Ko project was migrated from a Google owned GitHub repository to it’s own top-level organization, “ko-build“. We have updated verbiage in this post to reflect this but references to “Google Ko” may still be found online and in module names for a while, be assured that this is the same project. Congratulations to the Ko project for the success that merited this migration!

In this article, we’ll look at using Ko to build container images without Dockerfiles, SBOMs, and integrating with Kubernetes.

What is Ko?

Ko is a single binary, command line tool that is designed to be used in your development process in place of where you run the go compiler today. In addition to compiling your application, it will also generate an ultra slimmed-down container image that has your application installed in it. Like Jib, Ko will push the image to a registry or drop it into your local Docker image cache, depending on how you configure and/or run it.

Ko also has a few additional tricks up its sleeve related to software bill of materials (SBOM) construction as well as Kubernetes integration to help make iterative development and deployment processes super simple.

What problems is Ko trying to solve?

Many programmers, regardless of the language they work with, are new to container construction and often have a lot of questions about image building such as:

Which base image should I use, and is it compliant with my organization’s policies?
How do I best combine commands to minimize layer bloat?
How much of my app should I copy into the image to run my application?
What tooling do I need to learn to build the image? (Docker? Buildah? BuildKit?)
Are there standard annotations I need to include per my organization’s requirements?

Application architects and leads also want to make governance and standards implementation easy for their teams to follow, but maintaining uniform practices can be challenging when every team has unique and organically crafted Dockerfile patterns.

Security teams also care greatly about what goes into an image and the tooling used to construct it. For example, the practice of exposing the Docker engine — or the host machine’s docker.sock — to a CI build node can give build environments elevated access levels on those nodes.

“[The Docker] daemon… has a lot more capabilities beyond building and interacting with registries. Without additional security tooling, any user who can trigger a docker build on this machine can also perform a docker run to execute any command they like on the machine… Not only can they run any command they like, but also, if they use this privilege to perform a malicious action, it will be hard to track down who was responsible.”

— Container Security by Liz Rice, Chapter 6

On the other hand, introducing new build tools can be complex and increase the learning curve for those in charge of the build systems. Ko aims to address each of these areas while providing a superior experience to the developers using it.

Image building with and without Ko

To describe how Ko addresses the above-listed challenges, let’s first look at an example of how it compares to existing Go + Docker build steps.

Building the image without Ko

For this example, we will build the classic Go tutorial application, the “Hello World” web app.

1. In an empty directory, create the following as hello.go:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", HelloWebServer)
    http.ListenAndServe(":8080", nil)
}

func HelloWebServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, %s!", r.URL.Path[1:])
}

2. We will go ahead and build the binary as part of our Dockerfile as that is a common pattern:

FROM golang AS build
WORKDIR /go/src
COPY . .
RUN GOOS=linux go build -ldflags "-linkmode external -extldflags -static" -a hello.go

FROM gcr.io/distroless/static:nonroot
USER nonroot:nonroot
COPY --from=build /go/src/hello .
EXPOSE 8080
CMD ["./hello"]

This is a multi-stage build, with the first stage named build, based on the golang official image. In this stage, we copy the content of our app (which is simply the hello.go file at the moment) to the image under /go/src and simply run the go build tool with the parameters needed to produce a static binary.

In the second stage, we start from the Google Distroless project’s static:nonroot base image, set the default user to nonroot, copy the hello binary from the build stage into the top-level directory, set some metadata about the port we want to expose, and define that ./hello gets run by default at container startup.

Why two stages?

It is a common practice to do the build of your application in the Dockerfile as this guarantees that the right version of your compiler gets used both on a developer’s workstation and in any automated builds. A common error, though, is to deploy the same image that the build happened in. This is a security risk because you end up including not only extraneous tooling like the compiler, but also the source code of your application in that deployed image. If an attacker is able to exploit a vulnerability or get a copy of your image, they will have a lot of information and tools at their disposal to expand their attack. By starting from the distroless/static:nonroot base image, we have an extremely minimal filesystem with a non-root default user.

3. With our Dockerfile sorted out, we now can use the docker build command to construct a locally cached image and give it a tag:

$ docker build -t localhost:5000/hello-go:1.2.3 .
[+] Building 5.6s (12/12) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 298B
 => [internal] load .dockerignore
 => => transferring context: 73B
 => [internal] load metadata for gcr.io/distroless/static:nonroot
 => [internal] load metadata for docker.io/library/golang:latest
 => [stage-1 1/2] FROM gcr.io/distroless/static:nonroot
 => CACHED [build 1/4] FROM docker.io/library/golang
 => [internal] load build context
 => => transferring context: 5.15kB
 => [build 2/4] WORKDIR /go/src
 => [build 3/4] COPY . .
 => [build 4/4] RUN GOOS=linux go build -ldflags "-linkmode external -extldflags -static" -a hello.go
 => [stage-1 2/2] COPY --from=build /go/src/hello .
 => exporting to image
 => => exporting layers
 => => writing image sha256:eb9735e61e1dec63bd557dd5c61d8789733f2f4456a0d01687816bd0d135a7bc
 => => naming to localhost:5000/hello-go:1.2.3

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost:5000/hello-go 1.2.3 eb9735e61e1d 11 minutes ago 9.23MB

Note: Your output may look a little different depending on which version of Docker you are running and if you have BuildKit enhancements enabled.

4. For others to use this image, we need to push it to a registry repository. In this example, I’m running a local repository on my workstation at port 5000 via the registry:2 image from DockerHub. (This is why I tagged the image with the localhost:5000/ prefix.)

Pushing to a registry with Docker is pretty straightforward using the docker push command:

$ docker push localhost:5000/hello-go:1.2.3
The push refers to repository [localhost:5000/hello-go]
0ba424468cb9: Pushed
ca623f32e759: Pushed
1.2.3: digest: sha256:ddcabff499d90fdf6850ef0b7addb33db7def…

Note: If this were a managed registry, I would have needed to authenticate to it with docker login first.

5. Finally, to run our image, we could use docker run or an appropriate kubectl deployment. For this simplicity’s sake, we’ll do the former:

$ docker run --rm -d -p 8080:8080 localhost:5000/hello-go:1.2.3
Unable to find image 'localhost:5000/hello-go:1.2.3' locally
1.2.3: Pulling from hello-go
45e68f4d0d8c: Already exists
ce1a03145a01: Already exists
Digest: sha256:ddcabff499d90fdf6850ef0b7addb33db7defe4669f9af1079894e84ad407199
Status: Downloaded newer image for localhost:5000/hello-go:1.2.3
2ef3c3dc0363ad10e9bf21baa7e78c63ca4df904bcba1fdab3af7732fce3857d

and we can test the app with a simple curl command:

$ curl http://localhost:8080/Patch
Hello, Patch!

If we were to visualize the steps we just performed, it might look something like this:

Now let’s rewind back to the beginning and see how this would work with Ko.

Building the image with Ko

1. We’ll start with the same hello.go file in an empty directory:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", HelloWebServer)
    http.ListenAndServe(":8080", nil)
}

func HelloWebServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, %s!", r.URL.Path[1:])
}

2. Next, we set our image registry into an environmental variable and run ko build:

$ export KO_DOCKER_REPO=localhost:5000
$ ko build hello.go
2022/09/19 14:12:37 No matching credentials were found, falling back on anonymous
2022/09/19 14:12:40 Using base gcr.io/distroless/static:nonroot@sha256:2a9e2b4fa771d31fe3346a873be845bfc2159695b9f90ca08e950497006ccc2e for hello.go
2022/09/19 14:12:40 Building hello.go for linux/amd64
2022/09/19 14:12:44 Publishing localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:latest
2022/09/19 14:12:44 existing blob: sha256:2952e4f69ebf4bea5cc557f73626f95649cb546424fd998481ba690a08d9db7f
2022/09/19 14:12:44 existing blob: sha256:a706af0bb599ee120bd57c0e6abca55f66fd714f9e74706d9c97a583fc79d37e
2022/09/19 14:12:44 localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:sha256-3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c.sbom: digest: sha256:7d444debc3cd2d5545e88606dc529fa7ed90b1bb581ff8ac30b0f475b68d4ec0 size: 367
2022/09/19 14:12:44 Published SBOM localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:sha256-3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c.sbom
2022/09/19 14:12:44 existing blob: sha256:5ec5232d47ab0ad088792a191b672fe6ec27db63b19daebd7322ad64a2cd8676
2022/09/19 14:12:44 existing blob: sha256:250c06f7c38e52dc77e5c7586c3e40280dc7ff9bb9007c396e06d96736cf8542
2022/09/19 14:12:45 pushed blob: sha256:2d23903e55394a021ba4936cfad8ccbec6998413164416fcae4f6bf888665fce
2022/09/19 14:12:45 pushed blob: sha256:1cd0595314a53d179ddaf68761c9f40c4d9d1bcd3f692d1c005938dac2993db6
2022/09/19 14:12:45 localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:latest: digest: sha256:3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c size: 750
2022/09/19 14:12:45 Published localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7@sha256:3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c
localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7@sha256:3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c

That command:

compiled our application into a static binary
put it into a well-formed image, and
pushed it to my registry.

No Dockerfile or container runtime engine was required to do any of this. Additionally, you’ll notice references in the output about an SBOM being created and published; we’ll come back to that later.

Note: The image tag here contains an md5 hash that, by default, is the import path of your go application. Since we don’t have a full Go module in this example, that hash is simply taken from hello.go. See the Ko documentation for more information on tags.

3. Now we will run the image via docker run:

$ docker run --rm -d -p 8080:8080 localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:latest
Unable to find image 'localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:latest' locally
latest: Pulling from hello.go-7a204cfb24536a350234d9132276cae7
1cd0595314a5: Pull complete
250c06f7c38e: Pull complete
5ec5232d47ab: Pull complete
Digest: sha256:3925979ac92afb8cb89fc80438097873daf067195e4d0b9d2fd6f55d6201355c
Status: Downloaded newer image for localhost:5000/hello.go-7a204cfb24536a350234d9132276cae7:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
70877dcd53d23c66873e8e1e5a2092b46740e699b214a545e386113e5e098d7c

… and test with curl:

curl http://localhost:8080/ko
Hello, ko!

Visualizing the Ko pipeline might look something like this:

By letting Ko do the heavy lifting of image authoring, creation, and pushing, we reduced the number of steps to perform, the dependence on the container runtime engine at build time, and the expertise and maintenance requirements that the Dockerfile required.

Smart defaults, deterministic specializations

Ko users benefit from the collective definition of best practices by the open source community, but what if your organization has custom standards that don’t comply? This is where Ko’s command line options and/or ko.yaml configuration file comes in handy.

Custom base image

Let’s say, for example, that your company has a specific image from which all Go applications need to use as their base. Simply include defaultBaseImage: line in a ko.yaml file in the top level directory with that image as it’s value.

defaultBaseImage: repo.mycorp.com/myteam/corp-approved-scratch:220923

Go compiler flags

To explicitly specify the go compiler -ldflags as we had in our original example, simply add a builds: entry to the same ko.yaml file with appropriate settings as specified in the Ko documentation.

builds:
 - id: hello
   dir: .
   main: hello.go
   env:
     - GOOS=linux
     - CGO_ENABLED=0
   ldflags:
     - -extldflags "-static"
     - -linkmode external

Docker labels

Docker labels — a.k.a. OCI annotations — are a way to apply metadata to an image that is useful for referencing information such as the source repository it came from, CI build that created it, or any other data your team wants to embed. For more about image labels and when you might want to use them, check my blog on How and when to use Docker Labels.

Ko supports adding labels via its command line --image-label flag:

$ ko build hello.go --image-label foo=bar -L –image-label org.opencontainers.image.source=https://repo.mycorp.com/superteam/hello
…
ko.local/hello.go-7a204cfb24536a350234d9132276cae7:acf1dce794131205d488ed8fe4818866ebf509a61f4cf60aa07463e2b054d97d

$ docker image inspect ko.local/hello.go-7a204cfb24536a350234d9132276cae7:latest --format '{{json .Config.Labels}}'
{"foo":"bar","org.opencontainers.image.source":"https://repo.mycorp.com/superteam/hello"}

Note: As of the publishing of this article, there is no way to specify image labels from the .ko.yaml configuration file but there is an open issue for adding that functionality.

Other interesting Ko features

Image SBOMs

As we saw above, Ko will automatically generate and push an SBOM for your new container image. By default, it will be in SPDX format, but CycloneDX is also available by passing the flag --sbom=cyclonedx on the command line. You can also turn this feature off by passing --sbom=none; these and other configuration details are in the Ko documentation.

A full discussion of SBOMs and their role in creating a secure supply chain is out of the scope of this article but if you’d like to learn more, check out Building SBOMs for open source supply chain security.

Kubernetes integration

If your image needs to be tested in the context of a Kubernetes cluster, you undoubtedly have run into the hassle of needing to manage a recurring cycle of steps like:

Build my image
Deploy it to my registry (or otherwise load it into the Kubernetes cluster)
Update my deployment YAML with the new image tag
Redeploy via kubectl

Ko can make this much easier by automating all of these steps into a single command: ko apply. With a simple change to your deployment YAML, replacing the image tag with a specially formatted ko:// one, ko apply will automatically perform all of the above steps for you, deploying to whatever cluster you have actively set in your kubectl config context.

For example, let’s imagine you have a sandbox or locally running Kubernetes cluster and need to iteratively deploy and test your work as you go. Here is a deployment manifest for our “hello” application to deploy there:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-server
spec:
 selector:
   matchLabels:
     run: hello-server
 replicas: 2
 template:
   metadata:
     labels:
       run: hello-server
   spec:
     containers:
       - name: hello-server
         imagePullPolicy: IfNotPresent
         image: ko://github.com/myteam/hello
         ports:
           - containerPort: 8080
             name: http

You can see that the image: line which contains ko://, followed by the path to the Go module of our application in GitHub.

Now, we simply run ko apply and give it the YAML file in an -f flag like we would with kubectl:

$ ko apply -f myfile.yaml
2022/09/27 14:35:51 Using base gcr.io/distroless/static:nonroot@sha256:2a9e2b4fa771d31fe3346a873be845bfc2159695b9f90ca08e950497006ccc2e for github.com/myteam/hello
2022/09/27 14:35:51 Building github.myteam/hello for linux/amd64
…
2022/09/27 14:35:56 Published myteam/golang-ea0a77f5cbe6ba6aea599ad83048ae7b@sha256:bd7eb1052cf5b8664890f23683930f07423aba0089150bb818a53e76ef2ebf68
deployment.apps/hello-server created

$ k get pods
NAME READY STATUS RESTARTS AGE
pod/hello-server-5b5cc95db4-gc9rn 1/1 Running 0 10s
pod/hello-server-5b5cc95db4-rhwnp 1/1 Running 0 10s

We now can test our application, make a change and re-run ko apply -f myfile.yaml as many times as we need and it will do all the heavy lifting for you!

As you would expect, there are other Kubernetes commands that you likely would need, such as:

delete for removing your deployments, and
resolve for generating YAML that you can feed into kubectl or other tools.

See https://github.com/ko-build/ko#kubernetes-integration for full documentation details.

Challenges to using Ko

We’ve talked about all of the reasons why you might want to take the plunge and start using Ko in your workflows, but before you start, let’s talk about some of the challenges that you may face using Ko.

Cross-platform concerns

If you develop on a non-Linux based machine, the lack of a container runtime can, in some ways, make things a little harder. Having a container based build image allows you to encapsulate both the build tools as well as the OS and its libraries into a common base build image and effectively ignore the fact that your workstation platform may not match that of your production environment.

Case in point: The Go+Docker part of the example above will work on just about any platform because, regardless of where you build it, the golang base image provides the necessary Debian based, glibc libraries available for the Go builder to create the static binary. If, however, you try to run the Ko example with the same ldflags options on a MacOS machine, for example, you’ll get errors about missing libraries needed to do the static bindings.

This is because MacOS/Darwin does not include the necessary libraries to do the cross-complication for a Linux static binary (at least not at the time I am writing this). Not to pick on Apple too much, similar issues can come up for disparate CPU architectures with Linux and Windows WLS workstations too.

Skillset atrophy

The skills and knowledge leveraged to create and maintain well crafted container images is a valuable commodity and using tools which abstract or remove the need to have them can reduce the ability to deal with issues if something fails. In fact, the fewer the number of people on a team that understand container technologies, the greater the dependence is on those people with the needed experience when it comes to troubleshooting and maintenance at runtime. In extreme cases, this can lead to burnout, attrition, and/or outages.

Security complacency

If the process of container image construction becomes completely automated away from the developers, compliance with processes like image vulnerability scanning may falter. Vulnerabilities in un-updated images, packages, libraries, etc. can creep in exposing your application to attack so make sure such scanning is still being done via other automated means such as:

Build scriptings or Makefiles
Git hooks
CI build steps

If you are not already doing image scanning, Snyk offers free container image scanning that finds vulnerabilities in images and provides simple, actionable remediation advice.

Secure your containers for free

Create a free Snyk account for quick and easy image scanning.

Simplifying container images

Ko — and other alternative image building tools — can greatly simplify your Go application container image construction tasks and help streamline and standardize the way your teams build images. One of the greatest benefits is to your CI tooling as you no longer need to expose a container runtime to your build environment, greatly reducing privileged access attack vectors in your SDLC. Regardless of whether you choose to use a build tool like Ko, be sure your teams are well versed in container technologies and the security ramifications of any processes and tools they use to build their images.