Forem: STeve Shary

Tips on Building a Docker image (quickly) in a CI/CD Pipeline

STeve Shary — Thu, 05 Aug 2021 17:50:03 +0000

Building docker images has been evolving over the years and there are new strategies to do this. One of the pain points that comes up frequently is the (lack of) speed when building an image on a CI/CD pipeline. Many pipelines are now docker containers themselves and are a total blank slate. We lose out on many optimizations if we don't realize it.

Below are a series of tips and tricks to make your docker image speed up in your build pipeline. I have seen build times drop from 10's of minutes to less than 60 seconds, though your miles may vary!

Use .dockerfileignore: The first step of every docker build is that docker will tarball the entire directory and send the data to docker. This is output as the line Sending build context to Docker daemon XX.XXMB This should only be the source code and configuration files that are used in your build. It really should not be more than 10's of MB. Make sure to exclude everything that is not needed.
Things that don't often change should be pushed up in the file. The initial package installs, static environment variables, should be moved up. Docker has a concept of caching, but the first line in the dockerfile that has a change, all lines after it will no longer be cached. The last line of the dockerfile should be adding in your code, or compiling it!

A typical file should have the following flow:

  FROM base-cotnainer

  # Add all static environment variables, folders, users...
  ENV IS_APPLICATION=1
  RUN mkdir app
  WORKDIR /app  
  RUN useradd -ms /bin/sh newuser
  COMMAND ["python3", "main.py"]

  # Install all base image packages
  RUN apt install ...

  # Copy over your library depency files (Python example below)
  COPY Pipfile /app/Pipfile
  COPY Pipfile.lock /app/Pipfile.lock

  # Install your libraries (Python example below)
  RUN pipenv install --deploy

  # Lastly, copy the code over
  COPY . /app/

Don't use a wildcard with the COPY command. Docker will never cache that line. Be explicit!
Build your containers once. There should be zero reason to build your application multiple times.
Build your containers at the same time. If you have an application, test, integration, performance, et.al container,
build them all as parallel tasks/steps/jobs in your pipeline. Much of the docker builds is around I/O tasks and can and should be parallelized.

Copy your library dependency file definitions manually and then run the library install before you copy your source code! Doing
this step first will allow you cache the pulling of libraries (which feels like a full download of the internet). Below are
language specific examples of doing this:

Javascript

  FROM [...]
  mkdir /app
  WORKDIR /app
  # ...
  COPY package.json /app
  COPY package-lock.json /app
  RUN npm install

  COPY [all your code and configuration]

Python (requirements.txt)

FROM [...]
mkdir /app
WORKDIR /app
# ...
COPY requirements.txt /app
RUN pip3 install

COPY [all your code and configuration]

Python (pipenv)

FROM [...]
mkdir /app
WORKDIR /app
# ...
COPY Pipfile /app
COPY Pipfile.lock /app
RUN pipenv install --deploy

COPY [all your code and configuration]

Java (Maven)

FROM [...]
mkdir /app
WORKDIR /app
# ...
COPY pom.xml /app/pom.xml
RUN mvn install

COPY [all your code and configuration]    
RUN mvn package

Java (Gradle)

Add this to your build.gradle file:

task downloadDependencies(type: Exec) {
configurations.testRuntime.files
commandLine 'echo', 'Downloaded all dependencies'
}

Example Dockerfile to pull dependencies as a cached layer.

FROM [...]
mkdir /app
WORKDIR /app
# ...
COPY build.gradle /app/build.gradle
RUN ./gradlew downloadDependencies

COPY [all your code and configuration]    
RUN ./gradlew build

DotNet (Nuget)

FROM [...]
mkdir /app
WORKDIR /app
# ...
# Copy all solution files .sln (NO WILDCARDS!)
COPY WebApp.sln /app/WebApp.sln 
# COPY all .csproj files (NO WILDCARDS!)
COPY WebApp.Application/WebApp.Application.csproj /app/WebApp.Application/WebApp.Application.csproj
# Yep.. gotta manually add them all (... Pokemon!)

# This command will pull all library 
RUN dotnet restore

COPY [all your code and configuration]

Don't pre-pull images. It's slow and it doesn't work. Using the new BUILDKIT defined below, Docker will be smarter and only pull the image layers that are cached. This is faster,more efficient and it works.

Use docker's new library BUILDKIT and inline cache information into your builds. (This is a simple and big one)
There are three things we need to set to get proper image caching in a pipeline:
1. Set the DOCKER_BUILDKIT environment variable to 1
2. Add a build argument to tell the new BUILDKIT library to line caching information in each image layer: --build-arg BUILDKIT_INLINE_CACHE=1
3. Tell the docker build command where you can get your image from to cache --cache-from [remote-repo-url]/[CONTAINER_NAME]:latest The full command would look something like this: DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from [remote-repo-url]/[CONTAINER_NAME]:latest --tag [CONTAINER_NAME] .`

Lastly, don't forget to push an updated image to your remote docker repo!

What if I shared everything I know...

STeve Shary — Fri, 28 May 2021 14:37:35 +0000

At some point in your career, you will probably sit in a room and realize that you know more about the topic of discussion than anyone else. It might be some piece of functionality, a current bug, or maybe something larger like an entire system, business model, or technology. I challenge you to share everything you know.

Some practitioners view their niche specialty as their ability to have: job security, higher wages, better title or some other intangible prestige. It's easy to understand. It's nice to hear these things:

"I hear you are the expert at..."

"Our company would be really screwed if you ever tried to quit here."

"Wow, without you in these discussions, we could have made a big mistake with our assumption."

And we also have the fear of being forced to share our knowledge. I get it. I mean if everyone could do my job, I wouldn't be able to have a job.

When we start our technical careers, we have a small amount of knowledge. Whether through a degree, certification, or DIY knowledge, we have a glimmer to show that we can work in the technology field. There is an assumption that new employees, especially new to the field, will require training and ramp-up time. We have to teach you all the specific stack, development lifecycle, and codebase to be able to do your job. Your value to the company grows tremendously in the first year.

Let's talk about what that value can look like to a company:

In the x-axis, we have the number of weeks that a new software engineer has been at a company. The y-axis is value to the company. We have the grey dotted line, the best employee currently at the company as a baseline. Finally, we have three different curves representing different scenarios. This diagram is not exact, but is here to illustrate some principles.

The first is the somewhat common expectation that is baked into many project plans: new software engineers will provide the same level of output/value in a negligible amount of time. It's obviously wrong, but used much of the time because: it pushes the team to output more, it is difficult to actually calculate individual contribution levels as they onboard, and it looks good to upper management that their newly hired employee was worth the money.

The second situation is "most companies." If you think you are not in this group, you probably are. Here are some hallmarks of the vast majority of companies:

Waiting for your equipment to arrive.
A generic onboarding document/checklist.
Multi-day get my system setup process.
Multiple: "How do I get an account to this system?"
Being given simple tasks to complete on your own that grow in complexity over time.
Being put on support with and having the anxiety: "Oh my God: who do I call if I get an alert!?!"

The final line is one where an employee quickly provides value, but then can also grow above the current best employee. I have worked at a number of companies, but only one where I felt like I was in the red curve. This is where I was with a team that wanted to share everything that they knew with me.

I am quickly going to say that sharing everything I know is bounded to the context of your profession. Not necessarily personal life. This can happen in a sharing culture but isn't required.

There are many ways that we can share our knowledge. I'm going to boil it down to three different methods:

Writing it down.
Sharing it verbally.
Practicing it together.

Writing It Down

This I feel is the most difficult and provides least amount of value (most of the time). Writing is difficult. It requires practice to write well formed sentences. It is difficult to form them into points, arguments, and ideas. It is hard to know your audience. Most of the time, if you publish on the internet, anyone can be your reader.

Secondly, many finer details take up so much space to explain them that you have gigantic volumes written. These include:

Technical manuals.
Textbooks.
RFCs
Enterprise software documentation.

The vast, vast majority of people don't read these through. They just don't. I have written many technical guides, designs and documents that are very thorough and never read. Bleh, what a waste.

I think the one shining exception of this is https://stackoverflow.com/. This is where very precise questions can be asked and then precisely answered. All of us software engineers live and die by this site.

Lastly, I think that their is an incorrect assumption that writing it down will make it last forever. Yes, there are a few timeless classics out there, but most of the time not. Most written documentation is out-dated, and incorrect.. We move too fast and change too many things. We write documentation after we do something. Many people would rather no documentation than incorrect documentation.

Sharing It Verbally

Verbal sharing I think is an improvement in many ways to written. It's easier to talk than to write for many people. We usually talk more than we write. We often have a single audience, or a group that we know. Occasionally, we get to speak in front of a larger group that we don't know, but that is more the exception than the norm.

I like sharing my knowledge verbally because I can also get feedback. Non-verbal head nods are some, but also questions, comments and affirmations. I can repeat the same idea in different ways until the audience tells me that they understand.

This can also hone my understanding of what I am sharing. People ask lots of questions that I haven't thought about. It makes me figure out the answers and grow deeper in my knowledge.

Some may not like verbal knowledge sharing because it is temporal. "You would have to repeat it over and over again." Yes, but that assumption is that knowledge is shared by just one person. If everyone is doing it, then it spreads more like gossip.

Practicing It Together

This is my favorite way to share knowledge with other people. I find it to be the most rewarding and the most impactful. Rather than I tell you how to do something, let's do it together.

I do this in a couple of different ways:

pair sessions
mob sessions
lead the way

In a pair session, we just sit down and do something together. I can "drive" (use the keyboard) or you can. In longer sessions, we switch out. I encourage anyone that I am pairing with to ask any question they have. I get many of these:

"Why are you putting that in a different function?"
"How did you do that?" (Using an IDE function)
"Why did you start there?"
"What will that do?"

These don't have to be long, or 100% of the time, but I can take an employee on the first day of their job and pair with them and we can get work done. If I pair with them, their onboarding grows exponentially. Also, the majority of the time, they know something I don't. I get to learn everything you do when we pair.

In a mob session, we just expand the idea of pairing to just have a larger group work together. This can seem even crazier to a project manager. "You mean that the entire team is going to work together on a single ticket!?" In a mob session, I will define the goal: "We want to build a dashboard for our new feature." I will typically have two TVs hooked up to computers: one for writing code, and the other for googling any questions (Stack Overflow). All other laptops are banned. We pass the keyboards to the right every 5 minutes. All people in a session must participate. It forces everyone to pay attention because their turn is going to come up.

Lastly, I like to lead by doing. As a lead engineer, it is common for me to start a new team/project. I will setup the codebase, linting, automated testing, pipeline, deployment, alerting. All of it. And I do it with the newly formed team.

Need to do a zero-downtime deployment?

I'll show you how to do it.

Want to know how to develop a SLO/SLI/SLA for your service?

Let's do it together!

Want to do something that you have no idea how to do?

I don't know either, but let's figure it out!

I personally won't stay on a team for longer than 6 months. Most of the time, we have gotten the major pieces working together. The first several releases are done. Support has been figured out. The team dynamics are known. I just take a step back and let the team do their thing.

Wrapping it up.

At the end of it, I have plateaued in many of my technical skills. There are new things to learn here and there, but my value doesn't grow now based on how much technical knowledge I have. My value grows now by sharing it with others. The real value growth is that if I can teach others to do what I am doing. Teaching others to share their knowledge as well makes my value exponential. If I share 10 things with 10 people this year, and they each share those 10 things with 10 others.

My value to a company grows most when I share everything I know.

Why did my Java Web App stop working with MySQL!?!

STeve Shary — Mon, 10 May 2021 17:40:48 +0000

There are different kinds of bugs and they show up in different places. This bug was hard because it showed up where we didn't expect it.

We noticed last week that our Java web app deployments to kubernetes (k8s) stopped working last week. It started with a smaller team that did a deployment (to non-prod). The error started with a failed deployment. We use liveliness and readiness probes to help k8s let us know if the application is running correctly. Our liveliness probe was failing. That's strange. It just tells us if we can say "hello" back on a http GET.

Our first idea is to look at our logs. We saw the error:

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Hmmm... That's weird. The stack trace came from our MySQL client connection. There were no changes to:

The code that connects to MySQL.
The client library version.
The MySQL instance.

We reverted the commit, and wanted to see if going back to the previous commit would allow us to deploy. Nope. So now we have the "same" code running in a docker container running on the same deployment running against the SAME database.

This is the part in a show where the director speeds up time to show that the cast is doing A LOT of work but doesn't want to spend the time going through each minute detail. The same is here for us. We started trying every conceivable library and change that we could. (Fast forward several hours...)

We then went back and revisited the error:

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

It didn't make sense to us and so I did what I normally do not do. I scrolled through the 5,000+ lines of stack trace "caused by" lines to the bottom of the pit. At the bottom, I found this message: Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate). This was our best breadcrumb yet! This indicated that there was a problem establishing a secure connection. Hmmm...

One point that you have to understand is that Java handles TLS (a.k.a SSL) connections in the JVM. When you open a connection to another service, the JVM does all of the secure connection setup.

So I asked the question: "Has the Java JVM/JRE changed recently?" The answer was: "yes." We found that a change to the JVM happened to make the allowed TLS versions more strict: https://bugs.openjdk.java.net/browse/JDK-8254713.

You see our services run using a docker container. We have based our docker container off the current LTS of Java: gcr.io/distroless/java:11. What we didn't think about is that this is not a specifically pinned version. This means that the docker repo maintainers will periodically updated it. Turns out that they did after the latest dot release of Java 11. We were able to fix this immediately by changing the base image to: openjdk:11.0.8-jre. This pins to specific release of the JVM. We also put out a request to upgrade our TLS to our database.

Well, I hope if you see this, that it will be more obvious and take less time to drive to the solution!