Forem: Patrick Blesi

SSL Cert Rotation with Runbook

Patrick Blesi — Thu, 07 Nov 2019 19:34:53 +0000

SSL cert rotation is a problem that plagues nearly every web developer. Some are fortunate to work with infrastructure where an automated solution is available. This eliminates the need for tedious SSL cert management. Others are fortunate to have a team dedicated to managing infrastructure needs, so they can toss this issue to another engineer.

Unfortunately, I do not fall into either of these two camps when it comes to SSL certificate management. I needed to develop my own certificate management solution to ensure new SSL certs got properly installed on my servers. This post covers how to use Braintree's Runbook to automate the rotation of SSL certificates.

My certificate infrastructure is based on Jamie Nguyen's OpenSSL Certificate Authority Tutorial. I store a root certificate authority and intermediate certificate authority on my local machine. From the intermediate CA, I issue certs for servers within my infrastructure. All servers in my infrastructure trust my intermediate CA, so they accept traffic from valid certs when connecting to other servers.

SSL certificate rotation is a process that's ripe for automation. It is a tedious, repetitive process. The process is performed infrequently enough that it is easy to forget how to do it. And it is mission critical; failing to properly rotate SSL certificates almost certainly results in an outage.

The automation process

Several iterations were required before I boiled the rotation process down to a single command entered on the command line. My first pass at rotating SSL certs involved a lot of trial and error. Ultimately, I nailed down all the steps required to successfully rotate SSL certificates in my infrastructure. I took diligent notes to be well-armed for my next inevitable encounter with SSL cert rotation.

On my second iteration, with notes in hand, I was able to successfully rotate certificates with only minor tweaks to my documented process. Nevertheless, having a handful of servers still made this a tedious process. Retyping the several-dozen commmand incantations over and over for each server is not how I like to spend my afternoons. Leveraging Runbook allowed me to bring this process down to single command for rotating my SSL certs. Below is the SSL cert rotation runbook I developed.

The runbook

The SSL Cert rotation process breaks down into three main steps. First, a new cert needs to be generated using the intermediate CA. Second, the new cert needs to be distributed to the target server. And finally, the service must be restarted to pick up the new certificate.

See Runbook's README.md for details on setting up and working with Runbook.

Setup

First, we must collect some input for our parameterized runbook. The two pieces of information we need are the host and service for the SSL certificate we are generating.

#!/usr/bin/env ruby
require "runbook"

host = ENV["HOST"] # e.x. ldap01.stg
raise "Error no host specified using HOST env var" unless host

service = ENV["SERVICE"] # e.x. slapd
raise "Error no service specified using SERVICE env var" unless service

local_user = ENV["USER"]
company_name = "patricks_pickles"
intermediate_ca_path = "/root/ca/intermediate"
local_git_dir = "/home/#{local_user}/dev/#{company_name}/pp-infrastructure"
local_cert_path = "modules/ldap/files"
local_cert_file = "#{host}_#{service}.cert.pem"
staging_suffix = "#{company_name}-staging.com"
prod_suffix = "#{company_name}.com"

We collect this info from the command line using environment variables. The reason we do not use Runbook's ask statement to collect this info is because these values are used prominently throughout the runbook. It makes for a cleaner implementation to collect these values on the command line and then parameterize our runbook with them before executing it.

Additionally, we initialize a number of other local variables that are used throughout our runbook.

All runbooks are initialized with a title. It is best-practice to provide a description of the runbook as well.

runbook = Runbook.book "Renew SSL Certs" do
  description <<-DESC
    This Runbook rotates SSL Certs.
  DESC

  layout [[:runbook, :commands]]

  section "Setup" do
    step { ruby_command { @env = host.split(".").last.to_sym } }
  end
end

This runbook has a layout, which implies the runbook manages multiple panes using tmux. The layout is a stacked layout where the runbook executes in the top pane and commands are sent to the bottom pane. The setup section sets the @env instance variable at runtime. This value is subsequently available in all other ruby_command blocks.

Certificate generation

After initial setup, the next runbook section encompasses all steps required to create the new certificate.

runbook = Runbook.book "Renew SSL Certs" do
  #...

  section "Create New Cert" do
    user "root"

    step "Backup and update index.txt" do
      capture %Q{ls #{intermediate_ca_path}/index.txt.old* | tail -n 1 | sed -E "s/.*([0-9]{2})/\\1/"}, into: :backup_num

      ruby_command do
        @backup_num = (backup_num.to_i + 1).to_s.rjust(2, "0")

        command "cp #{intermediate_ca_path}/index.txt #{intermediate_ca_path}/index.txt.old#{@backup_num}"
        command "cp #{intermediate_ca_path}/index.txt.attr #{intermediate_ca_path}/index.txt.attr.old#{@backup_num}"

        command %Q{sed -i "/#{host} #{service.upcase}/d" #{intermediate_ca_path}/index.txt}
      end
    end

    step "Generate new cert" do
      ruby_command do
        case env
        when :stg
          @expiration_days = 1035
        when :prod
          @expiration_days = 1095
        else
          raise "Unknown env: #{env}"
        end

        tmux_command "sudo openssl ca -config #{intermediate_ca_path}/openssl.cnf -extensions server_cert -days #{@expiration_days} -notext -md sha256 -in #{intermediate_ca_path}/csr/#{host}_#{service}.csr.pem -out #{intermediate_ca_path}/certs/#{local_cert_file}", :commands
        confirm "Have you generated the cert?"
      end

      command "sudo chmod 444 #{intermediate_ca_path}/certs/#{local_cert_file}"

      tmux_command "sudo openssl x509 -noout -text -in #{intermediate_ca_path}/certs/#{local_cert_file}", :commands
      confirm "Does the cert look correct?"

      tmux_command "sudo openssl verify -CAfile #{intermediate_ca_path}/certs/ca-chain.cert.pem #{intermediate_ca_path}/certs/#{local_cert_file}", :commands
      confirm "Is the cert valid?"
    end
  end
end

The user setter designates that all commands in this section exececute as the root user. Because no host is specified, commands are executed locally. confirm statements allow us to confirm everything is correct before moving on to the next step.

Because the case statement references the env instance variable, it must be wrapped in a ruby_command block so it is executed at runtime. Because the tmux_command references @expiration_days, it also must be defined in the ruby_command block.

Uploading the certificate

In this section we copy the newly-generated certificate to our version-controlled infrastructure management repository, upload the certificate to the server, and install the certificate.

runbook = Runbook.book "Renew SSL Certs" do
  #...

  section "Upload Cert" do
    step "Copy the cert to pp-infrastructure" do
      user "root"

      command "cp #{intermediate_ca_path}/certs/#{local_cert_file} #{local_git_dir}/#{local_cert_path}"
      command "chown #{local_user}:#{local_user} #{local_git_dir}/#{local_cert_path}/#{local_cert_file}"
    end

    step "Upload the cert" do
      server host

      upload "#{local_git_dir}/#{local_cert_path}/#{local_cert_file}", to: "/home/#{local_user}"
    end

    step "Install the cert" do
      server host
      user "root"

      command "mv ~#{local_user}/#{local_cert_file} /etc/ssl"
      command "chown root:ssl-cert /etc/ssl/#{local_cert_file}"
      command "chmod 444 /etc/ssl/#{local_cert_file}"

      command "cp /etc/ssl/#{local_cert_file} /etc/ssl/certs"
    end
  end
end

The server setter uses entries in my ~/.ssh/config file to resolve the long-form host name, port, and user to log in as for a given host string passed to ssh.

Restarting the service

Lastly, we restart the service, verify our new certificate is valid, and commit our certificate to version control.

runbook = Runbook.book "Renew SSL Certs" do
  #...

  section "Upload Cert" do
    #...

    step "Restart the service" do
      server host
      user "root"

      command "service #{service} restart"
    end

    step "Validate the service cert is valid" do
      ruby_command do
        case env
        when :stg
          @suffix = staging_suffix
        when :prod
          @suffix = prod_suffix
        else
          raise "Unknown env: #{env}"
        end

        tmux_command "ssh #{host}", :commands
        tmux_command "openssl s_client -connect #{host}.#{@suffix}:12345 -CApath /etc/ssl/certs", :commands

        confirm "Is the cert valid?"
      end
    end

    step "Commit cert changes" do
      path "#{local_git_dir}"

      command "git add #{local_cert_path}/#{local_cert_file}"
      command %Q{git commit -m "Update #{local_cert_file} certificate"}
      command "git push"
    end
  end
end

Including the cert validation step allows us to confirm that the new cert is working as expected so we don't encounter an issue when the old certificate expires.

With our runbook in hand, rotating SSL certs is as simple as invoking the following command:

HOST=ldap01.stg SERVICE=slapd runbook exec runbooks/renew_ssl_certs.rb

Conclusion

The full runbook for rotating SSL certs is available as a gist. This runbook likely won't meet your SSL rotation needs as is. But hopefully it can serve as a basis for automating your SSL cert rotation process, so you are no longer plagued with manually managing SSL certificates.

Runbook: A Ruby DSL for Gradual System Automation

Patrick Blesi — Wed, 14 Aug 2019 15:25:50 +0000

At Braintree, we like to write tools to automate our work. Our latest tool is Runbook, a Ruby DSL for gradually automating system operations.

I know what you’re thinking: Why build yet another tool to automate an engineer’s job? We already have bash scripts!

First, anyone who has tried writing a for-loop in bash will admit it's not intuitive (I have to look it up every time!). Second, even when scripting out solutions to common maintenance operations, there are often setup, teardown, and verification steps that are required to ensure the operation ran successfully. How many times have you run into issues forgetting to execute a setup or cleanup step that’s required for your maintenance script? How many times have you forgotten to verify that an operation has succeeded?

We can often mitigate these kinds of issues with good documentation. The problem with software documentation, as we know, is that it can become outdated over time if the maintainers neglect to update it.

How often have you scripted a maintenance operation only to have it become outdated and break six months later? Inevitably, you break out the editor and perform script surgery in an effort to recover from the failed state.

Runbook addresses these types of issues by providing a framework that tightly couples the documentation and code for an operation. It also allows you to progressively automate your operations, finding the right balance between full automation and human involvement.

The philosophy of Runbook is heavily aligned with Dan Slimmon's Do-nothing scripting and Atul Gawande's The Checklist Manifesto. It is designed to minimize Toil.

Runbook is not intended to replace more special-purpose automation solutions such as configuration management solutions (Puppet, Chef, Ansible, Salt), deployment solutions (Capistrano, Kubernetes, Docker Swarm), monitoring solutions (Nagios, Datadog), or local command execution (Rake tasks, Make). Instead Runbook is best used as a glue when needing to accomplish a task that cuts across these domains.

A simple runbook

A runbook outlines a list of steps required to perform an operation.

# restart_nginx.rb

Runbook.book "Restart Nginx" do
  description <<-DESC
This is a simple runbook to restart nginx
  DESC

  section "Restart Nginx" do
    step "Stop Nginx"
    step "Wait for requests to drain"
    step "Start Nginx"
  end
end

It can be compiled and used to generate a Markdown checklist or be interactively executed.

# Restart Nginx

This is a simple runbook to restart nginx

## 1. Restart Nginx

1. [] Stop Nginx

2. [] Wait for requests to drain

3. [] Start Nginx

Adding automation

Moving past this initial outline, one can start to build automation into their runbook.

# restart_nginx.rb

Runbook.book "Restart Nginx" do
  description <<-DESC
This is a simple runbook to restart nginx and
verify it starts successfully
  DESC

  section "Restart Nginx" do
    server "app01.prod"
    user "root"

    step "Stop Nginx" do
      note "Stopping Nginx..."
      command "service nginx stop"
      assert %q{service nginx status | grep "not running"}
    end

    step { wait 30 }

    step "Start Nginx" do
      note "Starting Nginx..."
      command "service nginx start"
      assert %q{service nginx status | grep "is running"}
      confirm "Nginx is taking traffic?"
      notice "Make sure to report why you restarted nginx"
    end
  end
end

Notice that this runbook includes the step confirm "Nginx is taking traffic?". You can easily put off scripting steps that are more difficult to automate by delegating that step to the person executing the runbook.

Features

Some of Runbook's features include:

SSH integration

Runbook integrates with SSH using SSHKit to provide support for executing commands on remote servers, downloading and uploading files, and capturing output from remotely executed commands. You can control the parallelization strategy for execution, executing in parallel, serially, or in groups.

Runbook.book "Restart Nginx" do
  section "Restart Services" do
    servers (0..50).map { |n| "app#{n.to_s.rjust(2, "0")}.prod" }
    parallelization(strategy: :groups, limit: 5, wait: 2)

    step "Restart services" do
      command "service nginx restart" 
    end
  end
end

The above example executes service nginx restart across app{01..50}.prod on five servers at a time, waiting 2 seconds between each execution.

Dynamic control flow

We designed Runbook's control flow to be dynamic; at any point you can skip steps, jump to any step (even a previous step), or exit.

Runbook saves its state between each step of the runbook, and it can restart from where it left off if an error occurs while executing the runbook. In fact, you can resume a stopped runbook at any point in its execution.

Noop and auto modes

Runbook provides both a noop and an auto mode. Noop mode allows you to verify the operations your runbook will run before you execute it. Auto mode will execute your runbook, requiring no human interaction. Any prompts you have added to your runbook will use the provided default values, or the execution will immediately fail if prompts exist without defaults.

Execution lifecycle hooks

Runbook provides support for before, around, and after execution hooks. You can alter and augment your runbook behavior by hooking into the execution of entities and statements in your runbook. Hooks can be used to provide a rich set of behavior such as timing the execution of steps of a runbook or the runbook as a whole, tracking the frequency of execution of a runbook, and notifying Slack when a runbook has completed.

Runbook::Runs::SSHKit.register_hook(
  :notify_slack_of_execution_time,
  :around,
  Runbook::Entities::Book
) do |object, metadata, block|
  start = Time.now
  block.call(object, metadata)
  duration = Time.now - start
  unless metadata[:noop]
    message = "Runbook #{object.title}: took #{duration} seconds to execute!"
    notify_slack(message)
  end
end

First-class tmux support

At Braintree we live on a steady diet of vim and tmux. Consequently, Runbook provides first-class support for executing commands within a tmux. When specifying your runbook, you can define a tmux layout. This flexible and intuitive interface allows you to send commands to panes by name.

Executing commands in separate panes is ideal for monitoring, commands that require user interaction, or commands that are prone to failure. You can then interact with the command directly, troubleshooting and resolving issues before continuing the runbook.

Runbook.book "Restart Nginx" do
  layout [[
    [{name: :top_left, runbook_pane: true}, :top_right],
    :middle,
    {name: :bottom, directory: "/var/log", command: "tail -Fn 100 nginx.log"},
  ]]

  section "Setup monitoring" do
    step do
      tmux_command "watch 'service nginx status'", :top_right
      tmux_command "vim /etc/nginx/nginx.conf", :middle   
    end
  end
end

Runbooks remember their tmux layouts between executions. If a runbook stops unexpectedly, it will connect to the existing tmux layout when resumed as long as the tmux panes have not been altered. Additionally, runbooks offer to automatically close their tmux panes when the runbook finishes executing.

Ruby commands

Runbook provides a ruby_command statement to dynamically define runbook statements and their arguments. You can, for example, hit a JSON endpoint to retrieve a list of servers and then execute a command on each of those servers. Because you are working in Ruby, you have access to all the parsing and processing capabilities it provides.

require 'json'

Runbook.book "Restart Old Services" do
  section "Restart week-old services" do  
    step do
      server "monitor01.prod"

      capture "curl localhost:9200/host_ages.json",  into: :host_ages

      ruby_command do |rb_cmd, metadata|
        one_week_ago = 1.week.ago
        old_hosts = JSON.parse(host_ages).select { |host| host["started"] < one_week_ago }
        old_host_names = old_hosts.map { |host| host["name"] }
        old_host_names.each do |name|
          command "shutdown -r now", ssh_config: {servers: [name], user: "root"}
        end
      end  
    end
  end
end

Generators

Runbook provides generators, similar to Rails, for generating runbooks, runbook extensions, and runbook-focused projects. You can even define your own generators for including team-specific customizations in your generated runbooks.

Adaptability

Runbook is designed to seamlessly integrate into existing infrastructure. It can be used as a Ruby library, a command line tool, or to create self-executable runbooks. Runbook adheres to universal interfaces such as the command line and ssh. Runbooks can be invoked via cron jobs and integrated into docker containers.

Further, Runbook is extensible so you can augment the DSL with your own statements and functionality. The below example aliases section to s in the Book DSL.

module MyRunbook::Extensions
  module Aliases
    module DSL
      def s(title, &block)
        section(title, &block)
      end
    end
  end

  Runbook::Entities::Book::DSL.prepend(Aliases::DSL)
end

This flexibility allows you to adapt Runbook to meet any use case you encounter.

Check it out

At Braintree, we use Runbook for automating our app deployment preflight checklists, on-call playbooks, system maintenance operations, SDK deployments, and more. We've found it to be instrumental in streamlining production operations, reducing human error, and increasing overall quality of life.

Check out Runbook on Github for more information on how you can use Runbook to streamline production operations and increase developer happiness!

This post was originally published on medium.