Forem: Jeremy Kreutzbender

How Release Uses Action Cable and Redux Toolkit

Jeremy Kreutzbender — Thu, 15 Jul 2021 15:19:25 +0000

Over the past few weeks at Release the Frontend Engineering team has started working on adding Redux to Release. We had been making use of React Context but felt that we were starting to stretch its capabilities. In some places we were having to add multiple providers to implement new features. After some research on the current state of Redux, we decided to go with Redux Toolkit and Redux Saga. Moving all our data into the Redux store and out of local state meant that we were going to have to change our approach with Action Cable and how we were going to receive the messages, store them, and display changes for the user.

Action Cable, Redux, and Release

Release uses Action Cable in a single direction, which is from the backend to the frontend. The frontend is a separate React application running as a Static Service Application, not a part of Rails. The backend will send messages to the frontend when the state of objects change or to stream logs of deployments and builds. Today I’m going to go through the thought process, including code snippets, of how we set up our Redux implementation for Action Cable when Release builds a Docker image. If you’re curious about how Release builds Docker images, read about we Cut Build Time In Half with Docker’s Buildx Kubernetes Driver.

Action Cable Setup

Let’s start off with how we set up the backend to send updates as a Build object progresses. We have two ActiveRecord models to consider in this scenario, Build, and Log. The Build class includes the aasm gem functionality to progress it through the lifecycle of actually creating a Docker build. The following is an extremely pared down version of our Build class, but has enough information to explain how we’re sending the Action Cable messages.

class Build < ApplicationRecord
  include AASM  
  include Logging

  has_many :logs

  aasm use_transactions: false do
    state :ready, initial: true
    state :running, after_enter: Proc.new { update_started_at; log_start }
    state :done, after_enter: Proc.new { set_duration; log_done }
    state :errored, after_enter: Proc.new { set_duration; log_error }

    event :start do
      transitions from: [:ready], to: :running
    end

    event :finish do
      transitions from: [:running], to: :done
    end

    event :error do
      transitions from: [:running], to: :errored
    end

  def log_start
    message = "Build starting for #{repository.name}!"
    log_it(:info, message, metadata: log_metadata)
  end

  def log_done
    message = "Build finished for #{repository.name}!"
    log_it(:info, message, metadata: log_metadata)
  end

  def log_error
    message = "Build errored for #{repository.name}!"
    log_it(:error, message, metadata: log_metadata)
  end

  def log_metadata
    {
      build_id: self.id, 
      aasm_state: self.aasm_state,
      started_at: self.started_at,
      duration: self.total_duration
    }
  end

  def logs_channel
    "build_channel_#{self.id}"
  end
end

Whenever the Build transitions its state, we create a Log record through the log_it method. A log level is supplied, along with the message, and metadata about the Build itself. That metadata is used by the frontend to make changes for the user as you’ll see when we go through the Redux code. log_it also sends the message to the logs_channel through Action Cable. Since that wasn’t defined above, let’s look at that now.

module Logging
  module Log
    def log_it(level, message, metadata: {})
      log_hash = {
        level: level,
        message: message.dup.force_encoding('UTF-8')
      }

      self.logs << ::Log.new(log_hash)

      payload = log_hash.merge(metadata)
      ActionCable.server.broadcast(logs_channel, payload)
    end
  end
end

There is not too much to it. We create the Log record and ensure the message is properly encoded. Then we combine the level, message, and supplied metadata to Action Cable and broadcast it. We use the log_it method with more classes than just Build and have found it makes for an easy and reliable way to store and send messages.

That takes care of our state transitions. The last piece needed to wrap up our backend setup is to create the BuildChannel.

class BuildChannel < ApplicationCable::Channel
  def subscribed
    Rails.logger.info "Subscribing to: build_channel_#{params['room']}"
    stream_from "build_channel_#{params['room']}"
  end
end

The method receives a room parameter to ensure we are sending messages about a specific Build and does not go to everyone. I like to have the logging message in there so that it is easy to tell in the Rails logs if the frontend has successfully connected to the channel. With all that covered, we’re ready to dive into the setup on the frontend to receive those messages!

Redux Setup

As you’ll recall we’re using Redux Toolkit and we’re not going to cover our entire setup with Toolkit, instead focusing only on the portions relevant to updating the Build when we receive an Action Cable message. From there we’ll go over a small wrapper component we made to handle receiving the Action Cable messages and tie it all together with a small demo component.

We’ll start off with the BuildsSlice.

import { createSlice } from "@reduxjs/toolkit";

import {
  handleBuildMessageReceived,
} from "./helpers/actionCable/builds";

const initialState = {
  activeBuild: undefined, // object
};

export const buildsSlice = createSlice({
  updateBuildFromMessage(state, action) {
    const message = action.payload;

    const build = state.activeBuild;
    const newBuild = handleBuildMessageReceived(build, message);

    return {
      ...state,
      activeBuild: newBuild,
    };
  },
})

export const {
  updateBuildFromMessage,
} = buildsSlice.actions;

export default buildsSlice.reducer;

You’ll notice that we import handleBuildMessageReceived from a file under helpers/actionCable. We wanted to separate out the code for the logic of updating the build from the slice itself so that our slice file does not grow too enormous. Other than that, the slice itself follows the suggested setup of a slice from the createSlice documentation.

Now we need to look at our handleBuildMessageReceived function.

const handleBuildMessageReceived = (build, message) => {
  const buildId = message["build_id"];
  const aasmState = message["aasm_state"];
  const duration = message["duration"];
  const startedAt = message["started_at"];
  const level = message["level"];
  const messageLog = message["message"];

  const logs = build.logs;

  if (build.id !== buildId) {
    return build;
  } else {
    const newLogLine = { level: level, message: messageLog };
    const newBuild = {
      ...build,
      logs: [...logs, newLogLine],
      aasm_state: aasmState || build.aasm_state,
      total_duration: duration || build.total_duration,
      started_at: startedAt || build.started_at,
    };
    return newBuild;
  }
};

export { handleBuildMessageReceived };

First a sanity check is done to ensure we didn’t somehow receive a message for a Build that we aren’t viewing. This shouldn’t happen because we open and close our Action Cable subscriptions when we enter and leave a page, but an extra check never hurts. Then we construct a new Build object by appending the new log line and adding the metadata. If the metadata fields are undefined, we’ll retain what the build variable already had.

We’re ready to receive messages so we need a component that will handle that for us. The ActionCableWrapper component is just that.

import React, { useEffect, useState } from "react";
import actionCable from "actioncable";

export default function ActionCableWrapper({ channel, room, onReceived }) {
  const [actionCableConsumer, setActionCableConsumer] = useState(undefined);

  useEffect(() => {
    if (!actionCableConsumer) {
      setActionCableConsumer(actionCable.createConsumer("ws://localhost:3000/cable"));
    } else {
      actionCableConsumer.subscriptions.create(
        { channel, room },
        {
          received: onReceived,
        }
      );
    }

    return () => {
      if (actionCableConsumer) {
        actionCableConsumer.disconnect();
      }
    };
  }, [actionCableConsumer]);

  return <></>;
}

This component will mount and check to see if actionCableConsumer is not undefined. However, if it is undefined, which it will be on the first pass through the useEffect, we will create a consumer through actionCable.createConsumer connecting to a /cable endpoint. "ws://localhost:3000/cable" is hard coded but the URL should come from an environment variable so the component works locally or in production. That consumer is set into the local state actionCableConsumer and the useEffect will trigger a second time.

In the second pass through, the else block is entered and a subscription is created with the passed in channel, room, and onReceived properties. The return function is set to call disconnect() if we have an actionCableConsumer set and will ensure that no web socket connections are left open if a user navigates away from the page. With that, we have a reusable component that will take care of our Action Cable needs throughout the application.

Pulling it all together, we can create a demo component that will display the state and logs and update whenever it receives a message.

import React from "react";
import { useDispatch, useSelector } from "react-redux";

import { Grid } from "@material-ui/core";

import ActionCableWrapper from "../ActionCableWrapper"; 

import { updateBuildFromMessage } from "redux/slices/builds";

export default function BuildDetailsCard(props) {
  const dispatch = useDispatch();
  const build = useSelector(state => state.builds.activeBuild);

  const handleMessageReceived = message => dispatch(updateBuildFromMessage(message));

  return (
    <>
      <ActionCableWrapper channel="BuildChannel" room={build.id} onReceived={handleMessageReceived} />
      <Grid container>
        <Grid item xs={3}>
          <div>
            <b>Repository Name:</b> {build.repository.name}
          </div>
          <div>
            <b>Commit Message:</b> {build.commit_message}
          </div>
          <div>
            <b>Commit SHA:</b> {build.commit_short}
          </div>
          <div>
            <b>State:</b> {build.aasm_state}
          </div>
        </Grid>
        <Grid
          item
          xs={9}
          style={{
            border: "2px",
            backgroundColor: "#343a40",
            fontSize: "0.9rem",
            fontFamily: "Monaco",
            color: "white",
            padding: 10,
          }}
        >
          {build.logs.map(log => (
            <div>{log.message}</div>
          ))}
        </Grid>
      </Grid>
    </>
  );
}

For demo purposes I probably went a little overboard with the styling, but I wanted to create something that resembles our actual application which you saw at the start of this post. The two things needed to power the page are the build, which is retrieved with useSelector and the handleMessageReceived function, which dispatches updateBuildFromMessage every time we receive a message through Action Cable. We supply the ”BuildChannel” and build.id as the channel and room to ActionCableWrapper along with handleMessageReceived as the onReceived function.

In the video below I’ll move the build through its different states and we’ll be able to see the frontend receive the messages, update the state, and add the logs to the screen.

Conclusion

That's a wrap on my adventure into how we set up our Action Cable integration with Redux Toolkit. There are tons of places in the application we’re going to be adding live updates too so that our users will always be up to date on the state of their application. I hope you enjoyed taking a peek inside some development work at Release. If you're interested in having an ephemeral environment created whenever we receive a Pull Request webhook from your Repository, head on over to the homepage and sign up! If you’d like to join our awesome team, check out our job listsings.

Webhook Authentication Learnings for GitHub, GitLab, and Bitbucket

Jeremy Kreutzbender — Thu, 03 Jun 2021 15:37:20 +0000

Webhook Authentication and Processing for Github, Gitlab, and Bitbucket

I was recently tasked with implementing Gitlab support for Release and to complete that task I needed to implement authentication for Gitlab and a way to handle their webhooks. I completed the authentication first and left the webhook implementation for another pull request as I wanted to refactor the way we were handling all webhooks.

As I started the webhook work, the state of how Release handled webhooks was that Github was using the github_webhook gem and Bitbucket was using some custom built code that lived in a Controller Concern. With the need to add a third client I wanted to align everything into a few classes that allow us to easily onboard more providers if the need ever arose.

As I finished up the work I thought it would be useful to share some things I found about the differences between the three providers and share some of the code that I wrote in case anyone else is trying to do a similar implementation. First I'll talk about a few things I came across and later, in the Technical Details section, I'll go over a few classes we're using in our Ruby on Rails project for handling the webhooks.

I enjoyed doing this work because it spanned a lot different aspects of our codebase, including designing a refactor that would allow for processing webhooks from three different sources, reading documentation and understanding three different APIs, and writing tests to ensure that as we move forward we shouldn't need to every worry about breaking our webhook processing.

Authentication Methods

Further down in the Technical Details section, I'll show some code for the Authenticator class so you'll see the implementation around this topic, but I wanted to touch on how each of the three providers handles authenticating the webhooks. We'll start with Github.

Github - With a Github App, when you create the App, you can supply a secret key which will be used as the basis for authenticating the webhooks for all repositories. Their approach is to combine the payload of the webhook and the secret to generate a hash. The generated hash will be passed as a header when the request is sent to you. The documentation gives an example in Ruby on how to generate your own version of the hash. Then, if the comparison of the two hashes matches, you know the authenticity of the request is valid. I would rate this as the most secure way of authenticating the webhooks because if the request were intercepted and decoded, the secret used to generate the hash is not present anywhere in the request. You can read Github's Securing your webhooks documentation for yourself if you want to learn more
Gitlab - Gitlab follows a similar approach to Github, except that a different webhook object must be created on Gitlab for each Repository (as opposed to the singular Github App). Each webhook installation can take in a secret as it is being created and that secret is sent with the webhook request in the headers. There is no generating of a hash like Github, the secret is simply added to the request. Due to the secret being sent in the header, we decided to generate a different secret for every Repository as we save the Repository in the database. This means that if a request were to be intercepted and decoded, at most a single Repository would be compromised. You can read Gitlab's Webhook documentation for yourself if you want to learn more.
Bitbucket Cloud - Bitbucket is set up in a similar fashion to Gitlab where we have to create a webhook object for each Repository. However, unlike Gitlab, Bitbucket Cloud does not offer a way to add a secret. I came across a JIRA Ticket that was created in 2017 outlining this omission of a way to secure the requests but it is currently still open. It is unfortunate that Bitbucket doesn't offer a way to authenticate the webhook requests as it would allow someone to potentially send requests with bad information. They do offer an alternative solution in their documentation about whitelisting specific IPs that the requests could come from, but my opinion is that they should implement adding the secret and at least follow in Gitlab's approach to send the secret in the headers. You can read Bitbucket Cloud's Manange webhooks documentation for yourself if you want to learn more.

The Action Is Separated From The Event

One aspect I really like with Github and Gitlab is that they differentiate their webhook request through an event and an action. The event is sent as a header in the request and an example would be pull_request (on Github) or merge_request (on Gitlab). There are many different things that can happen with a Pull Request though: it might be one of opened, closed, merged, reopened, and so on. Those different actions that could happen on the Pull Request are sent over in the payload as the key action with the value of the aforementioned states. From a coding perspective this event and action pattern allowed me to create a method, say process_pull_request and inside of that method, handle the many different actions that could occur in another method, say pull_request_opened. I found that designing the code this way allowed for a good abstraction and thorough unit testing of all the different action methods.

The outlier is Bitbucket which added the event and the action together in the header. For example, when a Pull Request is created, the header contains pullrequest:created, when closed pullrequest:rejected, and when merged pullrequest:fulfilled. When using Release for ephemeral environments, closing and merging a Pull Request are considered the same type of action: we will destroy that ephemeral environment. But since the header contains two different values, I had to implement two different methods: process_pullrequest_rejected and process_pullrequest_fulfilled which simply call another method. While it is a pretty minor inconvenience, I like the code pattern of the action and event separated compared to having them combined.

Technical Details

First and foremost I want to acknowledge the great work on the gitub_webhook gem as I used a good amount of what they had done to create the foundation for the Authenticator and Processor classes. What follows is the Ruby code I wrote to manage the webhooks from the three providers that Release currently supports.

The Authenticator

First up, we'll look at the Authenticator class. Its purpose is to authenticate the webhooks that we are receiving to ensure that they're valid. You'll see that there is an optional parameter for the Repository and it is optional because as I mentioned in the Authentication Methods above, for Github we have a single secret, while for Gitlab and Bitbucket a secret is generated for each Repository.

Aside from the initialization method, the class has a single public method, authenticate_request! which does as it is named. It will raise an error if the authenticity of the request cannot be validated otherwise the call will return. The expected_signature method follows the different providers implementations with Github needing to create a hash to compare, Gitlab needing only the secret, and Bitbucket currently using a random string due to not offering an authentication method.

module Webhooks
  class Authenticator
    class SignatureError < StandardError; end

    def initialize(request:, vcs_type:, repository: nil)
      @request = request
      @vcs_type = vcs_type
      @repository = repository
    end

    def authenticate_request!
      secret = client_secret(@vcs_type, @repository)
      request_signature = signature_header(@vcs_type, @request)
      expected_signature = expected_signature(@vcs_type, secret, @request)

      unless ActiveSupport::SecurityUtils.secure_compare(request_signature, expected_signature)
        raise SignatureError
      end
    end

    private

    def request_body(request)
      @request_body ||= (
        request.body.rewind
        request.body.read
      )
    end

    def signature_header(vcs_type, request)
      @signature_header ||= (
        case vcs_type
        when :github
          @request.headers['X-Hub-Signature-256']
        when :gitlab
          @request.headers['X-Gitlab-Token']
        when :bitbucket
          'bitbucket_cloud'
        end
      )
    end

    def expected_signature(vcs_type, secret, request)
      digest = OpenSSL::Digest.new('sha256')

      case vcs_type
      when :github
        "sha256=#{OpenSSL::HMAC.hexdigest(digest, secret, request_body(request))}"
      when :gitlab
        secret
      when :bitbucket
        'bitbucket_cloud'
      end
    end

    def client_secret(vcs_type, repository)
      case vcs_type
      when :github
        Clients::Github.webhook_secret
      when :gitlab, :bitbucket
        repository.webhook_secret
      end
    end

  end
end

The Processor

If the request is authenticated, then we need to process the payload that comes with the request and the Processor class does just that. It will look through the payload and try to find the associated Repository in our database, if that Repository cannot be found, then an error occurs. To determine what event occurred, we look through the different headers in the request and parse the value into Ruby method declaration form by replacing any non-word character with an underscore. Based on the provider who sent the request, a service object is initialized and then we attempt to call a process_ method. Some webhooks we receive are for things Release doesn't deal with, for example Github's issues webhooks, so we safely try the method as there may not be an implemented process_ method.

module Webhooks
  class Processor
    def initialize(request, vcs_type)
      @request = request
      @payload = json_body(request)
      @vcs_type = vcs_type
      @repository = repository_from_payload(@vcs_type, @payload)

      @webhook_service = webhook_service(@vcs_type, @payload, @repository)
    end

    def repository
      @repository
    end

    def process_webhook
      process_method = "process_#{event_method(@vcs_type, @request)}"
      @webhook_service.try(process_method)
    end

    private

    def json_body(request)
      payload = request.body.read
      ActiveSupport::HashWithIndifferentAccess.new(JSON.load(payload))
    end

    def repository_from_payload(vcs_type, payload)
      provider_repository_id = provider_repository_id(vcs_type, payload)
      if provider_repository_id
        Repository.find_by!(type: "Repositories::#{vcs_type.capitalize}", provider_repository_id: provider_repository_id)
      end
    rescue ActiveRecord::RecordNotFound => error
      # Re-Raise the error with info from the payload so we know what the repository is
      repository_info = payload.dig('repository', 'full_name')
      new_error = ActiveRecord::RecordNotFound.new(error.message + "Repository Info: #{repository_info}")
      new_error.set_backtrace(error.backtrace)
      raise new_error
    end

    def provider_repository_id(vcs_type, payload)
      case vcs_type
      when :github
        payload.dig('repository', 'id')
      when :gitlab
        payload.dig('project', 'id')
      when :bitbucket
        payload.dig('repository', 'uuid')
      else
        nil
      end
    end

    def event_method(vcs_type, request)
      @event_method ||= 
        (
          case vcs_type
          when :github
            request.headers['X-GitHub-Event']
          when :gitlab
            request.headers['X-Gitlab-Event']
          when :bitbucket
            request.headers['X-Event-Key']
          else
            nil
          end
        )&.downcase&.gsub(/\W/, '_')&.to_sym
    end

    def webhook_service(vcs_type, payload, repository)
      service_class = "Webhooks::#{vcs_type.to_s.capitalize}"
      service_class.constantize.new(payload, repository)
    end
  end
end

Github Webhook Service

The last method in Processor, webhook_service returns a service class that goes through our internal business logic of what we want to do with
the webhook. I'm going to provide a small snippet of the Github service when we receive a Pull Request webhook. If you recall, I mentioned this method in the The Action Is Separated From The Event section and how I liked this pattern of structuring the code. If someone else were to look at this code, I would hope they would find it easy to understand that anything to do with Github Pull Request webhooks happens inside of the process_pull_request method and the case statement handles all the different actions that can take place.

module Webhooks
  class Github
    def initialize(payload, repository)
      @payload = payload
      @action = @payload.dig('action')

      @repository = repository
    end

    def process_pull_request
      if @action.nil?
        error_message = "ERROR: Pull Request no action received, payload : #{@payload}, do nothing"
        Rails.logger.error(error_message)
      else            
        message = "Pull Request with action : *#{@action}*. Received Repository : #{@repository.name}."
        Rails.logger.info(message)

        case @action
        when 'opened', 'reopened'
          pull_request_opened
        when 'closed'
          pull_request_closed
        when 'labeled'
          pull_request_labeled
        else
          message = "Pull Request with action : *#{@action}*. Nothing to do for now."
          Rails.logger.info(message)
        end
      end
    end
  end
end

The Controller

The final piece to tie everything together is the controller. WebhooksController is the base class and each subclass implements only the vcs_type method. Our previous approach had custom code for each of the Webhooks::GithubController and Webhooks::BitbucketController. This meant that each required a ton of specific tests to ensure that we were processing all the different webhooks correctly. My refactored approach moved all that logic out of the controller and aimed for the smallest footprint possible to make testing as simple as possible.

There is only one route in the controller, which is a POST to create. I decided that since the Repository may be optional in the Authenticator that I will store it in the Processor and pass it into the Authenticator. Otherwise you can see that the public methods for each of the classes are called. If an error is raised by either, due to an unauthenticated webhook or possibly a webhook for a Repository we don't have in our database, we'll capture the error and log as much information as possible so that we can look into what went wrong.

class Webhooks::GithubController < WebhooksController
  vcs_type(:github)
end

class WebhooksController < ActionController::Base
  skip_before_action :verify_authenticity_token

  def self.vcs_type(vcs_name)
    define_method :vcs_type do
      vcs_name
    end
  end

  rescue_from StandardError do |error|
    payload = @webhook_service&.payload

    backtrace_cleaner = ActiveSupport::BacktraceCleaner.new
    cleaned_backtrace = backtrace_cleaner.clean(error.backtrace)

    error_message = "Error in #{self}! Message : #{error.message}\nPayload : #{payload}\nBacktrace : #{cleaned_backtrace.join("\n")}"
    Rails.logger.error(error_message)
    head :bad_request
  end

  def create
    processor = Webhooks::Processor.new(request, vcs_type)
    repository = processor.repository

    authenticator = Webhooks::Authenticator.new(request: request, vcs_type: vcs_type, repository: repository)
    authenticator.authenticate_request!

    processor.process_webhook

    head :ok
  end
end

Conclusion

That's a wrap on my stint in refactoring our webhook code to work with Github, Bitbucket, and Gitlab. If we ever have to add another provider I think it will be quite straightforward and I hope you enjoyed taking a peek inside some development work at Release. If you're interested in having an ephemeral environment created whenever we receive a Pull Request webhook from your Repository, head on over to the homepage and sign up!

Cutting Build Time In Half with Docker’s Buildx Kubernetes Driver

Jeremy Kreutzbender — Thu, 18 Feb 2021 18:22:25 +0000

At Release, environments are our main focus, but we can’t create environments without builds. Recently we undertook a project to revisit our build infrastructure and determine if it needed to be upgraded. Build times had become a big factor in our service delivery and we needed a way to improve our customers’ experiences. One of the main areas that we wanted to improve upon was the parallelism of building multiple docker images for a single application.

The title of the article already spoiled the solution, and the alternative ‘Release Did This One Thing To Cut Their Build Time In Half!’ didn’t quite fly with the rest of the company, but Docker’s new buildx project fit the bill. First, we’ll cover what our original infrastructure looked like and how long builds on an example project were taking. Then, we’ll describe the changes we made to use buildx and the speed increases we observed.

Let’s start off with a diagram of what our original infrastructure looked like.

As you can see, the requests for builds would flow into our main Rails application and then divvied out to the different builder instances through Sidekiq. The builder container is Ruby code that would authenticate to Github, clone the repository, check out the correct SHA, and then execute the docker build. Due to the way we built the authentication to pull the code from Github, a single builder container could only clone one repository at a time. Which meant that the container could only do a single build request at a time. We added threading in the Ruby code to be able to execute multiple docker build commands at a time, but the number of builder containers we had spun up limited our concurrent builds. While it’s not hard to horizontally scale with Kubernetes, we saw this authentication setup as a major bottleneck.

Another issue we encountered was that we had no mechanism for attempting to place builds on servers where they had been previously built, instead opting for grabbing the first free server. This meant there was very little chance to land on the same server and get the full benefit of Docker caching. While this isn’t a deal breaker for us, we still believed we could do better when creating the version of our build infrastructure. Enough of the theoretical, let’s actually build something!

Release Applications can contain many docker images and one of our favorite example repositories to showcase this is our fork of example-voting-app. Looking at the docker-compose we see that there are 3 different Docker images that we have to build, result, vote, and worker. Now that we have an understanding of Release’s original infrastructure and the application we want to build, let’s start up a fresh build and see the results.

NOTE I forked the awesome-release repo to my own Github, jer-k for the following results.

We can see that this brand new build with no cache hits took two minutes and 15 seconds to complete. Next, we want to make a few changes to ensure that each Docker image needs to be rebuilt. The changes are listed below.

git status
On branch release_builders
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   result/views/index.html
    modified:   vote/app.py
    modified:   worker/src/main/java/worker/Worker.java

For the purpose of this blog post, I ensured the following build ran on the same builder as the first and that we will have cache hits. As noted before, this wasn’t always the case in our production environment.

The caching helps and cuts 45 seconds off the build! The uncached build took almost twice as long as the second build with caching, but our assumption was that we could do a lot better (cached and uncached) with some new technology.

Enter Docker's Buildx Kubernetes Driver

One of the first things we wanted to solve was the concurrency issue and we set out to ensure that Docker itself was able to handle a larger workload. We came across the issue Concurrent "docker build" takes longer than sequential builds where people were describing what we feared; Docker slowed down when many builds were being run at the same time. Lucky for us, that issue was opened in 2014 and plenty of work had been done to resolve this issue. The final comment, by a member of the Docker team, was "Closing this. BuildKit is highly optimized for parallel workloads. If you see anything like this in buildkit or buildkit compared to legacy builder please report a new issue with a repro case." Thus we set out to learn more about BuildKit (the Github repository is located here). While researching, we came across buildx, which ended up having three key features we believed would resolve many of our issues. These three features were the bake command, the buildx kubernetes driver, and the ability for the Kubernetes driver to consistently send builds to the same server. Let’s cover each of these, first up the bake command.

buildx bake [OPTIONS] [TARGET...]
Bake is a high-level build command.

Each specified target will run in parallel as part of the build.

bake intrigued us because it seemed to be a built-in command for us to avoid using Ruby threading for our parallelism. bake takes an input of a file, which can either be in the form of a docker-compose, .json, or .hcl. We initially tested bake with the docker-compose from example-voting-app and we were blown away at how smoothly it built directly out of the box and how quickly it was able to build the three images! However, we opted to create our own .json file generator in Ruby, parsing our Application Template into an output. Here is our generated file for example-voting-app.

{
  "group": {
    "default": {
      "targets": [
        "vote",
        "result",
        "worker"
      ]
    }
  },
  "target": {
    "vote": {
      "context": "./vote",
      "tags": [
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/vote:latest",
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/vote:buildx-builders"
      ]
    },
    "result": {
      "context": "./result",
      "tags": [
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/result:latest",
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/result:buildx-builders"
      ]
    },
    "worker": {
      "context": "./worker",
      "tags": [
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/worker:latest",
        "<REDACTED>.dkr.ecr.us-west-2.amazonaws.com/jer-k/release-example-voting-app/worker:buildx-builders"
      ]
    }
  }
}

There are other inputs which can make their way into this file, such as build args, but since example-voting-app does not have any, they are omitted.

Next, we wanted to find more information on the Kubernetes driver and we found this blog post Kubernetes driver for Docker BuildX from the author of the Pull Request. We encourage you to read the latter as it covers getting up and running with the Kubernetes driver as well how the caching works, which is exactly what we needed. With that information in hand, we were able to start work on adding the buildx servers to our cluster. We created a generic way to deploy the servers into different clusters and adjust the number of replicas with the final command being

docker buildx create --name #{name} --driver kubernetes --driver-opt replicas=#{num_replicas},namespace=#{builder_namespace} --use

For us, we created a release-builder namespace with five replicas, in our development cluster. We can see the output by querying for the pods

kubectl get pods --namespace=release-builder
NAME                            READY   STATUS    RESTARTS   AGE
development0-86d99fcf46-26j9f   1/1     Running   0          6d10h
development0-86d99fcf46-5scpq   1/1     Running   0          6d13h
development0-86d99fcf46-jkk2b   1/1     Running   0          15d
development0-86d99fcf46-llkgq   1/1     Running   0          18d
development0-86d99fcf46-mr9jt   1/1     Running   0          20d

Since we have five replicas, we wanted to ensure that when we build applications, they end up on the same server so that we get the greatest amount of caching possible (distributed caching is a topic for another day). Luckily for us, buildx, with the Kubernetes driver, has an option for where to send the builds called loadbalance.

loadbalance=(sticky|random) - Load-balancing strategy. 
If set to "sticky", the pod is chosen using the hash of the context path. Defaults to "sticky"

The default sticky means that the builds should always end up on the same server due to the hashing (more detailed information on this is described in the aforementioned blog post). With all of that in place, we are ready to test out our new setup!

Using the same example-voting-app repository as before, I created a new branch buildx_builders and pointed the code to the buildx servers.

What we see is that this uncached build was more than twice as fast as the other uncached build and even faster than the cached build on the old infrastructure! But uncached builds should be a thing of the past with the sticky load balancing, so let’s make the same changes as the previous branch and see the results.

This build finished three times faster than the previous cached build! These types of speed increases are the reason we set out to redo our build infrastructure. The faster the builds complete, the faster we can create environments and help our customers deliver their products.

We’re still experimenting with buildx and learning as we go, but the initial results were more than enough for us to migrate our own production builds to the new infrastructure. We’re going to continue to blog about this topic as we learn more and scale so check back in with the Release blog in the future!