Forem: Kyle Wagner

How I Fell Into DevOps

Kyle Wagner — Wed, 27 Nov 2019 22:43:59 +0000

This post is basically how I learned to DevOps. It's not "How to Become a DevOps Engineer". There is no one answer. I had a lot of luck and a lot of passion. Instead, this is about my career path, and how I ended up the Lead SRE of a multi-national company.

First of all, I love CLI tools. My everyday toolkit is zsh, vim, grep, ssh, and git. I learned all these at my first job working on custom hardware. At the time, I was a new CS grad with a go-get-em attitude towards proper software design and building apps. The iPhone had just come out, and I was sure I'd work on apps for the rest of my career. Side note: I have never published a single app.

The job wasn't like that. Instead, I wrote kernel modules and learned all about /var/logs. Grep became indispensable. I wrote GTK+ widgets in object oriented C (not Objective-C), and that is the worst thing I've done in my career. On the plus side, I learned how to read some Chinese due to all the helpful Google results on GTK+ being Chinese forums.

Linux became second nature and its low level machinery was no longer a mystery. My very own patch was accepted into the kernel to fix a ioctl call that was causing our systems to crash. Even more insane, I wrote a complex build/deployment system in bash that required translating a home brew cmake like build system from Perl to Python.

One day, Google contacted me for a job interview. I passed the initial foot-in-the-door part of the process, and the recruiter asked me where I'd like to apply to within the company. I had no clue. I thought maybe Android? She asked me to describe my abilities and current job. Without hesitation, she suggested the Site Reliability Engineering group, which I had no idea what that was.

Sadly (or gladly), I didn't get the job at Google. Instead, I took a year off to teach myself about modern application design. I built microservices, messed around with these things called Docker, started learning the then brand new React framework. After a bunch of interviews, I ended up a mid-level QA Automation Engineer for a local mid-stage startup.

There was good year of work there in the automation role before things started to sour. I gained knowledge in build systems, integration testing and continuous integration with TravisCI. Then, they made some high level decisions that saw engineers leave. As the guy with the most knowledge of the cloud infrastructure, I was suddenly promoted to lead DevOps.

I was in way over my head, so I bought The DevOps Handbook. That single book changed, I'm not exaggerating, everything. It all clicked. All the skills I had gathered over the years since graduation made sense. I wanted to learn all the best practices, the Toyota Manufacturing System, what an SRE is, and how to put it all in place. I was determined to try.

The following year was filled with putting forth the lessons in the book. I did my best to achieve the DevOps transformation, but it didn't happen. The failure still hurts and is lessons learned. The problem was management buy-in. They didn't see the need for the changes and just wanted me to continuously fight fires. So, I quit.

Finding a company with the management buy-in is how I found my current job. I interviewed them as much as they interviewed me. So far, they haven't lied to me. Even when they merged with another company, they allowed me to move the DevOps way up the chain so all our teams can practice those best practices. I feel lucky. I'm happy to be a SRE doing the best DevOps I can.

I know this isn't a helpful post about "How to Become a DevOps Engineer" but it's how I did it. I just kept learning. That's the real answer.

Cover: Source

Continuous Integration Builds using Docker

Kyle Wagner — Wed, 23 Oct 2019 17:11:06 +0000

In my current position, we define continuous integration builds for all our projects in a Dockerfile. For us, Docker provides a set of guarantees that are perfect for CI.

Docker builds are reproducible.
Docker is available on every major OS.
Docker produces universal artifacts.

Builds are Reproducible Everywhere

A standard Jenkins build runs directly on the system. It's nothing against Jenkins, but if you are just building on a raw node, the underlying instance will have discrepancies. This could be security patch, library versions, or even OS versions. For us, a runner might be Ubuntu or OSX. This means out of date local packages cause build failures. Don't get me wrong, that's a worst case scenario. Worst case always happens in DevOps.

The ideal state of our build is "reproducible state". A successful build that is rerun should succeed with the same output. A build that changes between runs of the same git commit means someone somewhere messed up. Mostlikely, you aren't versioning build dependencies...I'm not saying we had this exact problem, but we had this exact problem.

The beauty of Docker builds, they are immutable unless you try really hard like not versioning. The steps in a Dockerfile run in the same order every time. Wonderful.

The biggest boon is the system that runs the build doesn't need to care what language or packages a build needs. The Docker build handles that via Dockerfile instead at the encompassing OS. There are no tricks to create virtual environments like in Python. No race conditions on which packages are installed by different builds. Everything is self contained and way faster than spinning up a fresh dedicated VM.

The glaring question one may ask, "Why not use the already built in Docker runner environments that CI solutions like Jenkins support?" Yes, that's a solution, but it doesn't solve another problem.

Run Builds Anywhere

Running a build anywhere is a dream chased by many a developer. If we guarantee that if the build works on a dev system will work in production, that's gold. Docker is a solution to the problem. A Docker build that works on Windows will work on a Mac. Sure, it's trickery if you look too hard at the man behind the curtain (the VM running Linux). Devs usually don't care about that. When a build and its tests succeed on their machine but fails on the build box, they are rightfully upset. So why not let them run the exact same process on their machine that the Jenkins box uses?

This portability provides robust builds and increased developer productivity. Jenkins pipeline files are opaque at best so we don't expect devs to learn the syntax. DevOps doesn't want to write a new Jenkins file for every project either. Rather, we encourage each team write their project's Dockerfile. Once the hump of Dockerphobia is surmounted, the teams are more productive since they don't wait on DevOps for their build changes. They don't even have to trigger their Jenkins build to see if it will fail. Less waste overall.

As a side anecdote, I explained this build system to friend of mine. He implemented it on a new service his team was working on while doing builds on AWS CodeBuild. For various reasons, CodeBuild wasn't the best solution, so he moved everything to CircleCI. The overall process took him 45 minutes, because all the build scripts were in Dockerfiles already. I'm not saying you couldn't move that quickly with normal build systems, but it's worth noting.

Docker as Artifacts

Dockers are a great universal artifact for build systems. Related to Run Builds Anywhere you can deploy Dockers practically everywhere. That's kind of the point of Dockers. What's more interesting is we can use Dockers as a storage for libraries.

We've experimented with this a bit over the last year. Every library is built using Docker as describe above. The resulting image is never used alone as docker itself is the storage for that artifact. Then when other Docker builds need a library we will import that upstream dependency using multi-stage Docker builds.

FROM cool-java-lib:2.0.0 as lib
FROM openjdk as builder

RUN mkdir /app
WORKDIR /app

COPY --from lib mavencache/* mavencache/*
COPY ./src/* /app/src

RUN mvn build

This isn't a perfect solution as it requires we still push the library to a Nexus repo for local development. Overall, this is not a recommendation. It's something to consider if you have problems with external library repos and only want to limit dependencies to a Docker repo at build time.

The Downsides

There are downsides to using Docker builds for CI. The big one is the use of Docker itself. Containers are great things, but they can be heavier than a cached library. This may not be true in some sense on the repository level since we can reuse layers. You can't do that in Maven repos. This caching requires a good knowledge of how Docker build layers work.

The intimate understanding of how to use Docker to build software is an entry barrier to developers. Dockers often seem like this magical tech that DevOps community peddles. Because of this, there is a lot more upfront work to get devs up to speed on Dockers but the tail is usually a lot better in our experience.

Reality

We've been using Docker builds like this for the better part of a year and it's made life a lot easier. Everyone has a much better idea of how projects build and exactly what went wrong. Is this for everyone? No, nothing is. This works for our use case as we rely heavily on container orchestration systems and have that expertise. In the cases we are not using containers to run a system, we do have to shoehorn this build style in. This is a minority of our services and the benefits are great for now. Like all things in DevOps, we will have a different philosophy in 6 months.