Forem: Mike Coutermarsh

How we use "ship small" to rapidly build new features at GitHub

Mike Coutermarsh — Wed, 01 Jan 2020 23:55:26 +0000

"Ship small! fast!" and "ship to learn!"

You've seen that shouted everywhere on Twitter. But what does it actually look like in practice?

I'm on the GitHub Actions team and we "ship small" every day to build new features quickly. Personally, this is my favorite way of building software. In this post I'll show you how we do it.

Let's build a new feature

In this post we will pretend we are building a new feature for a web app. It has a complex UI + backend and will take a small team roughly 6 weeks to build.

Project planning, define the minimal scope

To start, the lead engineer, product manager and eng manager collaborate to define minimal scope. We determine what parts are a MUST for this new feature to be valuable to users.

The lead engineer then does a break down of tasks and prioritizes them on a project board. Their job here isn't to be the person that figures out every technical detail. Their job is to find enough info to break the project up into small enough pieces people can get started with.

During planning we aren't spending hours hammering out small details. Perfect upfront planning can be the enemy of shipping. We know that as long as we have the minimal scope defined, we can always iterate and improve from there. Software is never complete.

There will still be things to figure out, but we don't need to wait to get started.

Start writing code

Getting started as soon as possible is important. I've never seen any software project go exactly to plan, as soon as code starts to be written, we know we will uncover new information that may change our approach.

We need to be flexible and roll with it as we learn more. As individuals find problems, they document them in new issues and it gets added to the project plan.

The first pull request: the boilerplate

Since this feature has a UI. One engineer will do the boilerplate setup for it. This is a blank page, that has a URL and is behind a feature flag. It's essentially a "hello, world" for the feature.

This should be a very quick task. They will open the pull request, tag everyone else working on the feature, and we get it shipped to production. Everyone on the team gets added to the feature flag, they can now see the blank page in production.

I cannot emphasize enough how important this first pull request is.

Once it's merged we've now made it much easier for others to build on top of it without running into each other. If your goal is to "ship small", I believe it's critical to start the project off that way as well. The first change is the example set for the changes to come after it. It's now culturally accepted to be shipping things to production
that aren't complete yet.

We can now have parallel work streams:

1 engineer can start working on the backend that feeds data to this page.
Another engineer can start working on the frontend.
The designer works on the UI. If they can code, they jump in and write markup + CSS.

We now have everyone able to contribute and we're able to ship partially completed work to production confidently (remember, it's behind a flag).

As small pieces of work are completed, everyone is opening pull requests, getting reviews from each other and merging to production. The shared context of the project is high.

If we need reviews from other teams (security for example), these small PR's are fairly low effort to review for outside teams. Making it likely we get feedback faster.

This is important because:

it allows people to work async (we are remote/different timezones).
each pull request is small and reviewed by everyone else. Making them easy to review and keeping everyone aware of progress/context
Project stakeholders can get feature flagged in and view production to see progress (and give feedback!)

Never get blocked, start ASAP

Often when building software we'll find ourselves in a situation where we believe "we need X from other person" before we can start. I've never found this to be 100% true.
I have seen software projects be stuck in a holding pattern where engineers are waiting on each other (this can burn days, or even weeks of time). We can generally always start something.

For example, maybe you need data from the API before starting the frontend. Stick it in a JSON file and fake the API. You're then unblocked and can get started. You'll then learn things about the data you need and will be able to provide this information to the developer who is working on the API. This will make their job easier and probably get you the API endpoint faster and exactly how you need it.

Production code is the best code

There's a reason we want to get changes into production as fast as possible.

When code is successfully merged and deployed to production, we've now eliminated a bunch of uncertainties about it. We know it passes CI, we know it won't bring the site down.

Even though it's behind a flag, there's always the possibility of something going wrong. Getting it into production increases our trust of it and makes it easier for the team
to build on it moving forward. Confidence is a huge driver of momentum. We need both if we're going to be able to build this feature quickly.

Feedback loop

As this project progresses, we are seeing changes in production daily. Once the feature is somewhat working, we have a good opportunity to start inviting more people into the process.
I like to share with people from security, documentation, support, sales, biz dev etc. Add them to the feature flag and ask them for feedback.

Remember how we started with a very basic project plan? At this point we have feedback coming in, we turn that feedback into new issues (tasks on the project board) and it feeds right
back into the development process.

We are now ITERATING on the feature and making it better. Iterating is easy for the team because we've been making small changes all along. People have a shared context
of how everything works, so contributing is easy. We have confidence because the majority of the code is in production.

Personally this is my favorite phase of every project. If things were setup correctly from the start, there should be a lot of momentum driving everything forward at this point.

This process of feedback -> code -> ship continues to loop until we have something we can ship to real users.

Shipping to real users

Once the team has accomplished the minimal scope, it's time to start getting this new feature out to real users.
The high level process we follow is "Staff ship" -> get feedback -> "General availability". There can be a lot of other details here (documentation, capacity, security, abuse) that depend on the feature.

The "Staff ship" is when we make the feature available to every employee and ask for feedback. Our company is large enough that this works really well for uncovering
potential problems. During this time we also watch instrumentation more closely for potential errors/performance problems. The ways people use our product is highly
variable and we can't anticipate everything.

"General availability" is when we make the feature available to everyone. We have tools available to let us roll it out slowly to a % of users. As we do this we continue to
monitor things closely. If we see serious problems we can always turn it off, fix the problem and then continue the roll out.

Principles

This was a pretty generalized view of how we work and the details will vary from project to project. But I think I can distill it down to a few important points.

Start by defining the MVP. Review and cut scope until it truly feels minimal.
Software is never done. We are OK with shipping things that aren't perfect. We can always ship another PR to improve it.
Always be unblocking. Make it easy for people to be contributing at all times (even if alone in a separate timezone).
The project plan starts minimal and evolves as we ship code. Invite others early to give feedback, incorporate that into the plan as the project progresses.

Some important points

you need to be able to deploy and rollback quickly (if you only deploy a couple times a week, this won't work well)
you need feature flags to gate code and ship only to specific users
this is a general example of how teams I've been on work (GitHub is a big company and individual teams do what works best for them)
I'm an engineer, this is an engineering centric view of the process

GitHub Actions - Deploy to Heroku

Mike Coutermarsh — Sat, 02 Nov 2019 18:58:57 +0000

Here's how to deploy to Heroku using GitHub Actions. Without adding any new dependencies.

First, you need a Heroku auth token. You can generate one on your machine using the following:

heroku authorizations:create

Copy the Token value and add it as a secret to your repository (Go to your repository Settings -> Secrets. Name it HEROKU_API_TOKEN)

Next, add this to your workflow.

- name: Deploy to Heroku
  env:
    HEROKU_API_TOKEN: ${{ secrets.HEROKU_API_TOKEN }}
    HEROKU_APP_NAME: "your-app-name-here"
  if: github.ref == 'refs/heads/master' && job.status == 'success'
  run: git push https://heroku:$HEROKU_API_TOKEN@git.heroku.com/$HEROKU_APP_NAME.git origin/master:master

This will only run for builds on the Master branch and if previous steps have worked properly.

Here's a full example for a Rails app. This runs your tests, and if successful, deploys to Heroku.

name: Ruby Test and Deploy
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Ruby
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.3
    - uses: actions/cache@v1
      with:
        path: vendor/bundle
        key: ${{ runner.os }}-gem-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: |
          ${{ runner.os }}-gem-
    - name: Install Bundler
      run: |
        gem install bundler
    - name: Install Gems
      run: bundle install --path vendor/bundle --jobs 4 --retry 3
    - name: Run Tests
      run: |
        bundle exec rake test
    - name: Deploy to Heroku
      env:
        HEROKU_API_TOKEN: ${{ secrets.HEROKU_API_TOKEN }}
        HEROKU_APP_NAME: "your-app-name-here"
      if: github.ref == 'refs/heads/master' && job.status == 'success'
      run: git push https://heroku:$HEROKU_API_TOKEN@git.heroku.com/$HEROKU_APP_NAME.git origin/master:master

Creating images with Ruby + HTML/CSS API

Mike Coutermarsh — Sun, 10 Mar 2019 21:17:01 +0000

Ever wonder how trendy websites like Product Hunt or Medium generate those fancy Twitter screenshots?

// Detect dark theme var iframe = document.getElementById('tweet-1104239882879156224-275'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1104239882879156224&theme=dark" }

Wonder no more!

Essentially, they render the HTML/CSS in a headless Chrome instance and take a screenshot. Sound complicated? Yes, it is. Especially "at scale."

Good thing there are API's for this.

I will show you how to do this with an API I built specifically for this purpose: HTML/CSS to Image API.

require "httparty"


auth = { username: 'user_id', password: 'api_key' }

html = "<div class=\"box\"><h3>Hello, world 😍</h3></div>"

css = ".box {
  border: 4px solid #8FB3E7;
  padding: 20px;
  color: white;
  font-size: 100px;
  width: 800px;
  height: 400px;
  font-family: 'Roboto';
  background-color: #8BC6EC;
  background-image: linear-gradient(135deg, #8BC6EC 0%, #9599E2 100%);
}"

image = HTTParty.post("https://hcti.io/v1/image",
                      body: { html: html, css: css },
                      basic_auth: auth)

# { url: https://hcti.io/v1/image/bfae7d68-86cc-4934-83ac-af3ba75a0d34 }

Done. We have an image ✨.

Rails + Caching

If you're creating these from Rails and already use memcached, here's a nice trick for ensuring you only create them once.

cache_key = "image/#{html}/#{css}"

image = Rails.cache.fetch(cache_key) do
  HTTParty.post("https://hcti.io/v1/image",
                      body: { html: html, css: css },
                      basic_auth: auth)
end

We use the html/css to generate the cache key. Don't worry about it being too long. Rails/Dalli automatically hashes the key for you, guaranteeing it will be unique and the correct length.

This way, if you send the exact same payload again, Rails will pull the URL from cache rather than regenerating it.

Another Rails tip

If you're generating these for all of your content, it may be temping to stick the API call into an after_create. I advise against that. It's always best to keep I/O to an absolute minimum when "in request". Even though the request may only take 30ms, that can add up if the endpoint is already doing a lot.

The solution, is using a background job.

So instead of making the call directly in an after_create. You can instead enqueue the background job using after_create. Then the image will be generated in the background. Keeping your Rails response times super quick.

Build something cool?

If you build something cool this, let me know so I can tweet it!!

Build a GitHub Action with Ruby

Mike Coutermarsh — Tue, 15 Jan 2019 16:04:11 +0000

Want to build a GitHub Action with everyones favorite language?

Here's a quick start for building one with Ruby. ✨

mscoutermarsh / ruby-action-recipe

Quick start recipe for building an Action with Ruby and the GitHub API.

This repo is out of date

📣 There is now a better way.

Take a look at setup-ruby for examples on how to use Ruby with Actions.

❤️

View on GitHub

Fancy gif

High traffic app that can’t take downtime - Data migration

Mike Coutermarsh — Sun, 13 Jan 2019 23:50:04 +0000

When you can't take downtime, the “trick” to data migration is: Think small changes.

New table
Update writes to go to old and new
Background job to migrate old data to new
Move reads to new table
Stop writing to old table

Each of these steps is a deploy.

Getting started with GitHub Actions? Try Actions Toolkit

Mike Coutermarsh — Sun, 13 Jan 2019 23:40:51 +0000

If you're writing your own Actions using Node.js. Save yourself some time and take a look at Actions Toolkit. Jason is collecting common tasks and extracting them to a single module you can use to build your Actions.

JasonEtco / actions-toolkit

🛠 A toolkit for building GitHub Actions in Node.js

GitHub Actions Toolkit

An opinionated toolkit for building GitHub Actions in Node.js
Usage • API • How to test your Action • FAQ

This toolkit is an opinionated alternative to (and wrapper around) the official toolkit. actions/toolkit makes many features optional in the interest of performance, so you may prefer to use it instead of this library.

Usage

Installation

$ npm install actions-toolkit

const { Toolkit } = require('actions-toolkit')
const tools = new Toolkit()

Bootstrap a new action

$ npx actions-toolkit my-cool-action

This will create a new folder my-cool-action with the following files:

├── Dockerfile
├── action.yml
├── index.js
├── index.test.js
└── package.json

API

Toolkit options

event (optional)

An optional list…

View on GitHub

Refactoring Tip: Use the View Object pattern in Rails for building a user profile page

Mike Coutermarsh — Sun, 14 May 2017 17:47:32 +0000

Medium.com’s user profile shows all of the users posts, their followers and what they have highlighted.

I’d imagine their data model works like this. They have a User model, a Post model, a FollowerAssociationModel and a Highlights model.

For their profile page, they bring in data from several different sources.

When building out a users profile page in your app, it might feel tempting to create an entirely new model for it. But I'm generally very wary of creating a model for something so non-specific. It can quickly become a general "throw everything here" model.

Use the View Object Pattern

The best way I have found to do this in Rails is to use the View Object Pattern (aka Presenter Object). You can read more about it here: 7 Patterns to Refactor Fat ActiveRecord Models (search view object).

First, you'd create a class that encapsulates all the data you want shown on the profile page.

Something like this:

class UserProfile
  attr_reader :user

  def initialize(user)
    @user = user
  end

  def tagline
    # example of some html conversion
    @tagline ||= ConvertTagsToLinks.run(user.tagline)
  end

  def posts
    @posts ||= user.posts.order_by_featured_date
  end

  def followers
    @follows ||= user.followers.ordered_by_popularity
  end

  ## ect...
end

(a bit of a contrived example, but hopefully you get the idea)

Then this makes it really easy to have one nice clean object to access in your controllers and views.

# some controller
def show
  @user = User.find(params[:id)
  @user_profile = UserProfile.new(@user)

  # render a view...
end

Then in your view, you are able to easily access your view object.

<p class="tagline"><%= @user_profile.tagline %></p>

Using this pattern keeps your abstractions very explicit. This pattern is easy to write tests for and helps keep User Profile code out of your models.

Give it a try and let me know how it works out for you.

My Top 3 Ruby on Rails Best Practices

Mike Coutermarsh — Wed, 29 Mar 2017 21:10:11 +0000

Here is what will give you the biggest bang for your effort.

Learn the presenter and service object patterns. This allows you to break down complex logic into small, self contained and easily tested classes. This makes writing/changing/reading code easy. Remember to namespace things and create clear separation between different objects. Learn more here: 7 Patterns to Refactor Fat ActiveRecord Models
Roughly follow Sandi Metz rules: Sandi Metz' Rules For Developers. No need to follow them exactly. But use them as a guideline. Small models/short methods. Methods should do 1 thing only. You can use something like Rubocop to force yourself/the whole team to follow these rules.
Don't be too clever. What I mean by this is: Keep it simple and follow patterns. Don't add complexity prematurely. You may think you need that fancy event handler/rabbitmq/etc Ruby gem - but it can probably wait. I like to follow the rule of “do I have 3 problems I can solve with this?” before I add in something non-standard (I think I stole this idea from Sandi Metz' book: POODR).

I hope this helps!

Hi, I'm Mike Coutermarsh

Mike Coutermarsh — Wed, 28 Sep 2016 22:22:45 +0000

I have been coding for [number] years.

You can find me on Twitter as @mscccc

I live in [city].

I work for [company]

I mostly program in these languages: [languages].

I am currently learning more about [topic].

Nice to meet you.