Forem: Nate Vick

Deploy Infrastructure With CDKTF

Nate Vick — Tue, 07 Sep 2021 20:19:54 +0000

CDK or Cloud Development Kit came out of AWS in 2018 as a way to write Infrastructure as Code in software languages used day-to-day by developers (JavaScript/TypeScript, Python, Java, C#, and soon Go). Since its release, a community has built up around it, and new flavors have arrived. AWS CDK, CDKTF, and CDK8s are all based on the same core, but compile to different formats. AWS CDK compiles to CloudFormation, while CDKTF compiles to Terraform compatible JSON, and CDK8s compiles to Kubernetes config.

In this post, we will walk through how to use CDKTF with DigitalOcean's Terraform provider. A Terraform provider is a "plugin" for Terraform to interact with remote systems. In this case, the provider is created and maintained by DigitalOcean. We will share a few examples of creating a Digital Ocean Project, a VPC, a Postgres Database, and a Droplet.

Prereqs and Install

To use cdktf you will need the following packages installed:

Terraform ≥ 0.12
Node.js ≥ 12.16
Yarn ≥ 1.21

To install cdktf, we will use npm install

$ npm install --global cdktf-cli

You will also need to create a personal access token on Digital Ocean if you would like to deploy this config.

Create and Initialize the example project

First let's make a directory and change into it.

$ mkdir cdk-do-example && cd cdk-do-example

Initialize the project with the init command. In general, I use the --local flag, so all state is stored locally.

$ cdktf init --template=typescript --local

You will be prompted for Project Name and Description but defaults are fine.

The last step of setting up the project is to add Terraform providers to the cdktf.json. Open the project in your editor and let's add the DigitalOcean provider to it as follows.

{
  "language": "typescript",
  "app": "npm run --silent compile && node main.js",
  "terraformProviders": [
    "digitalocean/digitalocean@~> 2.9"
  ],
  "terraformModules": [],
  "context": {
    "excludeStackIdFromLogicalIds": "true",
    "allowSepCharsInLogicalIds": "true"
  }
}

Install Dependencies

Run cdktf get to download the dependencies for using DigitalOcean with Typescript.

$ cdktf get
Generated typescript constructs in the output directory: .gen

Let's Write Some TypeScript

Open main.ts in your editor and let's start by creating a DO Project.

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })

    new Project(this, 'example-project', {
      name: 'Example Rails Project'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

You will create a new DigitaloceanProvider and pass in the environment variable assigned to your DO personal access token. Next, we create the Project, which has a required key of name.

Optionally, as a test of the code above, run cdktf deploy to deploy your CDKTF project. The cli will ask if you want to make the changes listed under Resources. Type 'yes', then once it finishes, you will see a green check next to the Resources it successfully created.

cli prompt:

successful deployment:

Now we will add a VPC, a Postgres Database, and a Droplet. Once those examples are in, we will tie it all together before deploying it again.

Create a VPC

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    new Vpc(this, 'example-vpc', {
      name: 'example-vpc',
      region: 'sfo3'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

*Note: If you do not have VPC in your account already this will become the default VPC which will not be deleted when cleaning up the CDKTF Project.

Create a Postgres Database

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc, DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    const postgres = new DatabaseCluster(this, 'example-postgres', {
      name: 'example-postgres',
      engine: 'pg',
      version: '13',
      size: 'db-s-1vcpu-1gb',
      region: 'sfo3',
      nodeCount: 1
    })

    new DatabaseUser(this, 'example-postgres-user', {
      clusterId: `${postgres.id}`,
      name: 'example'
    })

    new DatabaseDb(this, 'example-postgres-db', {
      clusterId: `${postgres.id}`,
      name: 'example-db'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

Notice we assigned the DatabaseCluster to a variable const postgres. We then use the variable to create the DatabaseUser and DatabaseDb on that cluster.

Create a Droplet

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc, DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })
    ...more code above

    new Droplet(this, 'example-droplet', {
      name: 'example-droplet',
      size: 's-1vcpu-1gb',
      region: 'sfo3',
      image: 'ubuntu-20-04-x64'
    })

  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

Let's Put It All Together

If you run cdktf deploy now, it would create everything, but nothing created would be put into the Digital Ocean project or the VPC we create. Let's do that now.

import { Construct } from 'constructs'
import { App, TerraformStack, Token } from 'cdktf'
import { DigitaloceanProvider, Droplet, Vpc} from './.gen/providers/digitalocean'
import { DatabaseCluster, DatabaseUser, DatabaseDb } from './.gen/providers/digitalocean'
import { Project, ProjectResources } from './.gen/providers/digitalocean'

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name)

    new DigitaloceanProvider(this, 'digitalocean', {
      token: Token.asString(process.env.DO_TOKEN) 
    })

    const project = new Project(this, 'example-project', {
      name: 'Example Rails Project'
    })

    const vpc = new Vpc(this, 'example-vpc', {
      name: 'example-vpc',
      region: 'sfo3'
    })

    const postgres = new DatabaseCluster(this, 'example-postgres', {
      name: 'example-postgres',
      engine: 'pg',
      version: '13',
      size: 'db-s-1vcpu-1gb',
      region: 'sfo3',
      nodeCount: 1,
      privateNetworkUuid: vpc.id
    })

    new DatabaseUser(this, 'example-postgres-user', {
      clusterId: `${postgres.id}`,
      name: 'example'
    })

    new DatabaseDb(this, 'example-postgres-db', {
      clusterId: `${postgres.id}`,
      name: 'example-db'
    })

    const droplet = new Droplet(this, 'example-droplet', {
      name: 'example-droplet',
      size: 's-1vcpu-1gb',
      region: 'sfo3',
      image: 'ubuntu-20-04-x64',
      vpcUuid: vpc.id
    })

    new ProjectResources(this, 'example-project-resources', {
      project: project.id,
      resources: [
        postgres.urn,
        droplet.urn
      ],
      dependsOn: [ postgres, droplet ]
    })
  }
}

const app = new App()
new MyStack(app, 'cdk-do-example')
app.synth()

We start by assigning the project, VPC, and droplet to variables. In the DatabaseCluster definition, we add privateNetworkUuid: [vpc.id](http://vpc.id) to place the database in our newly created VPC. Similarly, on the Droplet definition, we place it in the VPC, by adding vpcUuid: vpc.id.

Lastly, create a new ProjectResource to assign other resources to the Digital Ocean project. In this small example, we will assign the database and droplet to the project using the urn for each resource. We will wait to assign those until both are created using a Terraform helper dependsOn.

With all of that in place you can deploy again. The database and droplet creation take a bit, so be patient. 🙂 Once it has finished, check your Digital Ocean Dashboard to see everything created and ready for use.

*Make sure you run cdktf destroy to remove these resources from your account or you will be charged by Digital Ocean.*

Things To Know

CDKTF is still a young project, so it is changing fast, has limited documentation, and you can run into unclear errors.

There are few things that help with the limited documentation. You can read the provider documentation on the Terraform registry site and use it as a guide. Also, using a language like Typescript with VSCode there are a lot of code hints, hover info, and signature information. The gif below is an example of what is shown in VSCode when you hover on the problem. Note the missing properties for DropletConfig.

When you run into unclear errors, prepend CDKTF_LOG_LEVEL=debug to the deploy and/or destroy commands to get very verbose output.

Wrapping Up

In this post, we used CDKTF to create some basic example resources on Digital Ocean which gives you a good primer to build more complex infrastructure in a language of your choice. You can find the code from this post in this repo. If you would like to chat more about CDK or infrastructure as code you can ping me on Twitter @natron99.

Happy Ruby 3 Release Day!!!

Nate Vick — Fri, 25 Dec 2020 14:02:52 +0000

https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/

Also, Merry Christmas 🎄

Rails System Tests In Docker

Nate Vick — Tue, 28 Apr 2020 17:04:35 +0000

Originally posted on Hint's blog.

At Hint, we use Docker extensively. It is our development environment for all of our projects. On a recent greenfield project, we wanted to use Rails System Tests to validate the system from end to end.

In order to comfortably use System Tests inside of Docker we had to ask ourselves a few questions:

How do we use RSpec for System Tests?
How do we run the tests in headless mode in a modern browser?
Can we run the test in a non-headless browser for building and debugging efficiently?

Context

In the context of answering these questions, we are going to first need to profile the Rails project to which these answers apply. Our Rails app:

Uses Docker and Docker Compose.
Uses RSpec for general testing.
Has an entrypoint or startup script.

If some or none of the above apply to your app and you would like to learn more about our approach to Docker, take a look at this post: Dockerizing a Rails Project.

Prep Work

If your project was started before Rails 5.1, some codebase preparation is necessary. Beginning in Rails 5.1, the Rails core team integrated Capybara meaning Rails now properly handles all the painful parts of full system tests, e.g. database rollback. This means tools like database_cleaner are no longer needed. If it is present, remove database_cleaner from your Gemfile and remove any related config (typically found in spec/support/database_cleaner.rb , spec/spec_helper.rb , or spec/rails_helper.rb).

Dependency Installation

After that codebase prep has been completed, verify that RSpec ≥ 3.7 and the required system tests helper gems are installed.

group :development, :test do
  gem 'rspec-rails', '>= 3.7'
end

group :test do
  gem 'capybara', '>= 2.15'
  gem 'selenium-webdriver'
  # gem 'chromedriver-helper' don't leave this cruft in your Gemfile.:D
end

These are the default system test helper gems installed with rails new after Rails 5.1. We will not need chromedriver-helper since we will be using a separate container for headless Chrome with chromedriver.

Note: The configuration below has been tested on Mac and Linux, but not Windows.

Docker

Speaking of containers, let's add that service to the docker-compose.yml configuration file.

services:
  # other services...
  selenium:
    image: selenium/standalone-chrome

We have added the selenium service, which pulls down the latest selenium/standalone-chrome image. You may notice I have not mapped a port to the host. This service is for inter-container communication, so there is no reason to map a port. We will need to add an environment variable to the service (app in this case) that Rails/RSpec will be running on for setting a portion of the Selenium URL.

services:
  app:
    build: .
    command: bundle exec rails server -p 3000 -b '0.0.0.0'
    # ... more config ...
    ports:
      - "3000:3000"
      - "43447:43447"
    # ... more config ...
    environment:
      - SELENIUM_REMOTE_HOST=selenium

Capybara

We added a port mapping for Capybara as well: 43447:43447. Now let's add the Capybara config at spec/support/capybara.rb.

# frozen_string_literal: true

RSpec.configure do |config|
  headless = ENV.fetch('HEADLESS', true) != 'false'

  config.before(:each, type: :system) do
    driven_by :rack_test
  end

  config.before :each, type: :system, js: true do
    url = if headless
            "http://#{ENV['SELENIUM_REMOTE_HOST']}:4444/wd/hub"
          else
            'http://host.docker.internal:9515'
          end

    driven_by :selenium, using: :chrome, options: {
      browser:              :remote,
      url:                  url,
      desired_capabilities: :chrome
    }

    # Find Docker IP address
    Capybara.server_host = if headless
                             `/sbin/ip route|awk '/scope/ { print $9 }'`.strip
                           else
                             '0.0.0.0'
                           end
    Capybara.server_port = '43447'
    session_server       = Capybara.current_session.server
    Capybara.app_host    = "http://#{session_server.host}:#{session_server.port}"
  end

  config.after :each, type: :system, js: true do
    page.driver.browser.manage.logs.get(:browser).each do |log|
      case log.message
        when /This page includes a password or credit card input in a non-secure context/
          # Ignore this warning in tests
          next
        else
          message = "[#{log.level}] #{log.message}"
          raise message
      end
    end
  end
end

Let's break it down section by section.

headless = ENV.fetch('HEADLESS', true) != 'false'

We are using the headless variable to make some decisions later in the file. This variable allows us to run bundle exec rspec normally and run system tests against headless Chrome in the selenium container. Or we run HEADLESS=false bundle exec rspec and when a system test will attempt to connect to chromedriver running on the host machine.

config.before :each, type: :system do
  driven_by :rack_test
end

Our default driver for system tests will be rack_test. It is the fastest driver available because it does not involve starting up a browser. It also means we cannot test JavaScript while using it, which brings us to the next section.

config.before :each, type: :system, js: true do
  url = if headless
          "http://#{ENV['SELENIUM_REMOTE_HOST']}:4444/wd/hub"
        else
          'http://host.docker.internal:9515'
        end

  driven_by :selenium, using: :chrome, options: {
    browser:              :remote,
    url:                  url,
    desired_capabilities: :chrome
  }

  # ...more config...       
end

Any specs with js: true set will use this config. We set the url for Selenium to use depending on if we are running headless or not. Notice the special Docker domain we are setting the non-headless url to; it is a URL that points to the host machine. The special domain is currently only available on Mac and Windows, so we will need to handle that for Linux later.

We set our driver to :selenium with config options for browser, url, desired_capabilities .

config.before :each, type: :system, js: true do

  # ...selenium config...

  Capybara.server_host = if headless
                           `/sbin/ip route|awk '/scope/ { print $9 }'`.strip
                         else
                           '0.0.0.0'
                         end
  Capybara.server_port = '43447'
  session_server       = Capybara.current_session.server
  Capybara.app_host    = "http://#{session_server.host}:#{session_server.port}"
end

Here we set the Capybara.server_host address to the app container IP address if headless or 0.0.0.0 if not.

The last part of RSpec configuration is to require this config in spec/rails_helper.rb .

# frozen_string_literal: true

# This file is copied to spec/ when you run 'rails generate rspec:install'
require 'spec_helper'
ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
# Prevent database truncation if the environment is production
abort("The Rails environment is running in production mode!") if Rails.env.production?
require 'rspec/rails'
require 'support/capybara'

Next, we need to install ChromeDriver on the host machine. You will need to place it in a location in your $PATH. Once it is there, when you want to run non-headless system tests, you will need to start ChromeDriver chromedriver --whitelisted-ips in a new terminal session. Now on a Mac, you should be able to run headless or non-headless system tests. Those commands again are:

#HEADLESS
bundle exec rspec

#NON-HEADLESS
HEADLESS=false bundle exec rspec

Special Linux Config

There is one last step for Linux users because of the special host.docker.internal URL is not available. We need to add some config to the entrypoint or startup script to solve that issue.

: ${HOST_DOMAIN:="host.docker.internal"}
function check_host { ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1; }

# check if the docker host is running on mac or windows
if ! check_host; then
  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
  echo "$HOST_IP $HOST_DOMAIN" >> /etc/hosts
fi

We set an environment variable to the special Docker URL. We then create a function to check if the host responds to that URL. If it responds, we move on assuming we are running on Mac or Windows. If it does not respond, we assign the container's IP to an environment variable, then append a record to /etc/hosts. We are now all set to run system tests on Linux as well.

Bonus: CI Setup

Let's wrap this up with config to run system tests on Circle CI. We need to add SELENIUM_REMOTE_HOST and the Selenium Docker image to .circleci/config.yml

version: 2
jobs:
  build:
    parallelism: 1
    docker:
      - image: circleci/ruby:2.6.0-node
        environment:
          SELENIUM_REMOTE_HOST: localhost
      - image: selenium/standalone-chrome

Connect with me on Twitter(@natron99) to continue the conversation about Rails and Docker!

ActiveStorage & S3 Server-side Encryption

Nate Vick — Thu, 26 Mar 2020 20:46:46 +0000

Originally posted on Hint's blog.

TIL, it is possible to use S3 server-side encryption and ActiveStorage.

This commit to Rails in 2017 adds the ability but did not add documentation or an example of how to use the upload_options feature. Below is a vanilla S3 service config for ActiveStorage.

amazon:
  service: S3
  access_key_id: ACCESS_KEY_ID
  secret_access_key: SECRET_ACCESS_KEY
  region: us-east-1
  bucket: BUCKET

Here is a S3 service config using upload:

amazon:
  service: S3
  access_key_id: ACCESS_KEY_ID
  secret_access_key: SECRET_ACCESS_KEY
  region: us-east-1
  bucket: BUCKET
  upload: 
    server_side_encryption: 'aws:kms' # 'AES256'

The upload hash is passed to Aws::S3::Client#put_object(params = {}). One of the configuration options for put_object is :server_side_encryption (String). For more options checkout the Ruby SDK docs.

💡If you are using KMS keys, the bucket user will need the following policies:

"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:ReEncryptTo",
"kms:DescribeKey",
"kms:ReEncryptFrom"

To help other Rails devs, here is a PR to Rails to add the above example to the official guides.

Have a great day!

Happy Ruby Day!!!

Nate Vick — Wed, 25 Dec 2019 17:26:53 +0000

Ruby 2.7.0 Released

Introducing:

Pattern Matching
REPL (irb) improvement
Compaction GC
Separation of positional and keyword arguments and more!

What are you doing this weekend?

Nate Vick — Sun, 22 Dec 2019 16:32:27 +0000

I went for a drive yesterday around Mt St Helens. I couldn’t see the mountain, but I saw a bunch of waterfalls.

Guide: Rails Development with Docker

Nate Vick — Tue, 17 Dec 2019 20:18:32 +0000

Originally posted on Hint's blog.

As a software consultancy, we switch between many projects throughout the year. A critical factor in delivering value is the ease at which we are able to move between projects.

Over the years, we have used many tools to manage dependencies needed to run and develop our clients' projects. The problem with most tools has been the ability to have consistent, reproducible development environments across our team. About two years ago, we discovered that Docker was a viable option for building consistent development environments. Since then, we continue to iterate on our configuration as we learn new ways to handle the complexity of the projects while simplifying the setup process for our team.

In this guide, we will cover the basics of our Docker development environment for Rails.

Getting started

If you would like to follow along, install Docker CE and create, clone, or have a working Rails app.

We will be using a combination of Dockerfiles, Docker Compose, and bash scripts throughout this guide, so let's make a place for most of those files to live.
Start by creating a docker folder in the root of the Rails project. Here we will store Dockerfiles and bash scripts to be referenced from the docker-compose.yml file we will be creating later.

Inside the docker folder, create another folder named ruby. Inside the newly created ruby folder, create a file named Dockerfile. This file will contain commands to build a custom image for your Rails app.

Hello Dockerfile

  ARG RUBY_VERSION=2.6
  FROM ruby:$RUBY_VERSION
  ARG DEBIAN_FRONTEND=noninteractive

In this block, we set the RUBY_VERSION and DEBIAN_FRONTEND build arguments and specify the docker image we will use as our base image.

The first ARG sets a default value for RUBY_VERSION, but passing in a value from the command line or a docker-compose file will override it, as you'll see later in the guide. An ARG defined before FROM is only available for use in FROM.

FROM in a Dockerfile is the base image for building our image and the start of the build stage. In our case, we are using the official Ruby image from Docker Hub, defaulting to Ruby 2.6.

The next ARG is used to set the build stage shell to noninteractive mode, which is the general expectation during the build process. We don't want this environment variable to carry over to when we are using the images in our development environment, which is why we are using ARG instead of ENV.

Base Software Install

  ARG NODE_VERSION=11
  RUN curl -sL https://deb.nodesource.com/setup_$NODE_VERSION.x | bash -
  RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

  RUN apt-get update && apt-get install -y \
    build-essential \
    nodejs \
    yarn \
    locales \
    git \
    netcat \
    vim \
    sudo

Here we are setting up the necessary software and tools for running a modern Rails app. Like in the previous section, we are setting a default for our NODE_VERSION build argument.

The next two RUN lines set up the defined version of the node apt repository and the latest stable yarn apt repo. Since Webpacker started being officially supported in Rails 5.1, it is important we have a recent version of node and yarn available in the image we will be running Rails on.

The third RUN will look familiar if you have used any Debian based OS. It updates the apt repositories, which is important since we just installed two new ones, and then it installs our base software and tools.

I want to point out netcat and sudo specifically. netcat is a networking tool we will use to verify the other services are up when we are bringing up our Rails app through Docker Compose. We install sudo since by default it is not installed on the Debian based Docker images, and we will be using a non-root user in our Docker image.

Non-root User

  ARG UID
  ENV UID $UID
  ARG GID
  ENV GID $GID
  ARG USER=ruby
  ENV USER $USER

  RUN groupadd -g $GID $USER && \
      useradd -u $UID -g $USER -m $USER && \
      usermod -p "*" $USER && \
      usermod -aG sudo $USER && \
      echo "$USER ALL=NOPASSWD: ALL" >> /etc/sudoers.d/50-$USER

Docker containers generally use the root user, which is not inherently bad, but is problematic for file permissions in a development environment. Our solution for this is to create a non-root user and pass in our UID, GID, and username as build arguments.

If we pass in our UID and GID, all files created or modified by the user in the container will share the same permissions as our user on the host machine.

You will notice here that we use the build arguments to set the environment variable (via ENV) since we will also want these variables available when we bring up the container.

In the RUN instruction we add a standard Linux user/group and then we add the new user to the sudoers file with no password. This gives us all the benefits of running as root while keeping file permissions correct.

Ruby, RubyGems, and Bundler Defaults

  ENV LANG C.UTF-8

  ENV BUNDLE_PATH /gems
  ENV BUNDLE_HOME /gems

  ARG BUNDLE_JOBS=20
  ENV BUNDLE_JOBS $BUNDLE_JOBS
  ARG BUNDLE_RETRY=5
  ENV BUNDLE_RETRY $BUNDLE_RETRY

  ENV GEM_HOME /gems
  ENV GEM_PATH /gems

  ENV PATH /gems/bin:$PATH

Explicitly setting the LANG environment variable specifies the fallback locale setting for the image. UTF-8 is a sane fallback and is what locale defaults to when it's working properly.

We will be using a volume with Compose, so we need to point RubyGems and Bundler to where that volume will mount in the file system. We also set the gem executables in the path and set some defaults for Bundler, which are configurable via build arguments.

Optional Software Install

  #-----------------
  # Postgres Client:
  #-----------------
  ARG INSTALL_PG_CLIENT=false

  RUN if [ "$INSTALL_PG_CLIENT" = true ]; then \
      apt-get install -y postgresql-client \
  ;fi

Here is an example of how to set up optional software installs in the Dockerfile. In this scenario, postgresql-client will not be installed by default, but will be installed if we pass a build argument set to true (INSTALL_PG_CLIENT=true).

Dockerfile Final Touches

  RUN mkdir -p "$GEM_HOME" && chown $USER:$USER "$GEM_HOME"
  RUN mkdir -p /app && chown $USER:$USER /app

  WORKDIR /app

  RUN mkdir -p node_modules && chown $USER:$USER node_modules
  RUN mkdir -p public/packs && chown $USER:$USER public/packs
  RUN mkdir -p tmp/cache && chown $USER:$USER tmp/cache

  USER $USER

  RUN gem install bundler

To wrap up the Dockerfile, we create and set permissions on needed directories that were referenced previously in the file or that we will be setting up as volumes with Docker Compose.

We call USER $USER here at the end of the file, which will set the user for the image when you boot it as a container.

The last RUN command installs Bundler, which may not be required depending on the version of Ruby.

Next, we'll take a look at our docker-compose.yml file.

Docker Compose

Docker's definition of Compose is a great place to start.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services. Then, with a single command, you create and start all the services from your configuration.

That is what we are going to be doing in this section of the guide, adding support for a multi-container Docker application.

The first thing we will need is a docker-compose.yml file at the root of the Rails app. If you would like to copy and paste the following YAML into a docker-compose.yml file, I'll breakdown its parts below.

  version: '3.7'

  services:
    rails:
      build:
        context: ./docker/ruby
        args:
          - RUBY_VERSION=2.6
          - BUNDLE_JOBS=15
          - BUNDLE_RETRY=2
          - NODE_VERSION=12
          - INSTALL_PG_CLIENT=true
          - UID=500
          - GID=500
      environment:
        - DATABASE_USER=postgres
        - DATABASE_HOST=postgres
      command: bundle exec rails server -p 3000 -b '0.0.0.0'
      entrypoint: docker/ruby/entrypoint.sh
      volumes:
        - .:/app:cached
        - gems:/gems
        - node_modules:/app/node_modules
        - packs:/app/public/packs
        - rails_cache:/app/tmp/cache
      ports: 
        - "3000:3000"
      user: ruby
      tty: true
      stdin_open: true
      depends_on:
        - postgres
    postgres:
      image: postgres:11
      environment:
        - POSTGRES_HOST_AUTH_METHOD=trust
      volumes:
        - postgres:/var/lib/postgresql/data

  volumes:
    gems:
    postgres:
    node_modules:
    packs:
    rails_cache:

In this file, we are defining two services: the rails service, which our Rails app will run in, and the postgres service, which will accommodate PostgreSQL. The names for these services are arbitrary and could easily be foo and bar, but since the service names are used for building, starting, stopping, and networking, I would recommend naming them close to the actual service they are running.

We also are setting our Compose file compatibility to version: '3.7' which is the latest at the time of this writing.

The App Service Broken Down

  version: '3.7'

  services:
    rails:
      build:
        context: ./docker/ruby
        args:
          - RUBY_VERSION=2.6
          - BUNDLE_JOBS=15
          - BUNDLE_RETRY=2
          - NODE_VERSION=12
          - INSTALL_PG_CLIENT=true
          - UID=500
          - GID=500

The first part of the rails service is to set up to build the image from the Dockerfile we created earlier. Once built, it uses the local image from that point forward. We set the context, which is the directory path where our Dockerfile is stored. The args: key specifies the build arguments we set up in the Dockerfile earlier in the guide. Notice that we are overriding some of our earlier defaults here.

  services:
    rails:
      . . .
      environment:
        - DATABASE_USER=postgres
        - DATABASE_HOST=postgres
      command: bundle exec rails server -p 3000 -b '0.0.0.0'
      entrypoint: docker/ruby/entrypoint.sh
      volumes:
        - .:/app:cached
        - gems:/gems
        - node_modules:/app/node_modules
        - packs:/app/public/packs
        - rails_cache:/app/tmp/cache
      ports: 
        - "3000:3000"

In the next part, we set up environment variables for connecting to the postgres service, which will require updating your database.ymlexample file to take advantage of these variables.

We also set the default command that will run when docker-compose up runs, which in this case starts the Rails server on port 3000 and binds it to all IP addresses.

Next, we point to our entrypoint script, which we'll revisit after breaking down the rest of the Compose file.

After the entrypoint, we set up volumes.

The first volume mounts the current directory (.) to /app in the container. The mapping for this one is HOST:CONTAINER. If you are on Mac, I would recommend using .:/app:cached since file sharing on Docker for Mac is CPU bound. By setting :cached on the mount point it allows files to be out of sync with the host being authoritative. Here are more details about file sharing performance on Docker for Mac. In our testing :cached has been the best option for speed vs. tradeoffs.

The next volumes are named volumes. Creating them happens when we bring up the containers for the first time and then they are persistent from up and down. These volumes are native to the Docker environment, so they operate at native speeds. The persistence and speed are why we have chosen to use them for gem, node_modules, packs, and rails_cache storage; otherwise, we would have to reinstall both every time we bring our environment back up. Lastly, there is a top-level volumes key at the very bottom of the docker-compose.yml file, which is where they are defined.

The last key in this section is ports. Ports only need to be defined to access the containers from the host otherwise container to container communication happens on the internal Docker network generally using service names.

The mapping "3000:3000" allows us to connect to the Rails server at [localhost:3000](http://localhost:3000). The mapping is HOST:CONTAINER just like the volume mount, and it is recommended to pass them in as strings because YAML parses numbers in xx:yy as base-60.

  services:
    rails:
      . . .
      user: ruby
      tty: true
      stdin_open: true
      depends_on:
        - postgres

This last section explicitly sets the user to ruby, which was set up in our Dockerfile above.

Use the next two keys: tty and stdin_open, for debugging with binding.pry while hitting the Rails server. Check out this gist for more info.

One thing to call out from that gist is to use ctrl-p + ctrl-q to detach from the Docker container and leave it in a running state. The depends_on key is used to declare other containers that are required to start with the service. Currently, it is only dependent on postgres. No other health checks or validations are run; we will handle those in the entrypoint.

The Postgres Service

  services:
    rails:
      . . .
    postgres:
      image: postgres:11
      environment:
        - POSTGRES_HOST_AUTH_METHOD=trust
      volumes:
        - postgres:/var/lib/postgresql/data

We define our postgres service next with pretty minimal configuration compared to what we went through for the rails service.

The first key, image, will check locally first, then download the official postgres image from Docker Hub with a matching tag if needed. In this case, it will pull in 11.5. I chose 11 for this example because that is now the default on Heroku, but there are lots of image options available on Docker Hub.

We are using a named volume here as well for our postgres data.

And, that wraps up the Compose file.

Enter The Entrypoint

There are a few requirements for starting a Rails server, e.g., the database running and accepting connections. We use the entrypoint, which is a bash script, to fulfill those requirements. Looking back at the compose file we should create this bash script at docker/ruby/entrypoint.sh and make it executable.

  #! /bin/bash
  set -e

  : ${APP_PATH:="/app"}
  : ${APP_TEMP_PATH:="$APP_PATH/tmp"}
  : ${APP_SETUP_LOCK:="$APP_TEMP_PATH/setup.lock"}
  : ${APP_SETUP_WAIT:="5"}

  # 1: Define the functions to lock and unlock our app container's setup
  # processes:
  function lock_setup { mkdir -p $APP_TEMP_PATH && touch $APP_SETUP_LOCK; }
  function unlock_setup { rm -rf $APP_SETUP_LOCK; }
  function wait_setup { echo "Waiting for app setup to finish..."; sleep $APP_SETUP_WAIT; }

  # 2: 'Unlock' the setup process if the script exits prematurely:
  trap unlock_setup HUP INT QUIT KILL TERM EXIT

  # 3: Wait for postgres to come up
  echo "DB is not ready, sleeping..."
  until nc -vz postgres 5432 &>/dev/null; do
    sleep 1
  done
  echo "DB is ready, starting Rails."

  # 4: Specify a default command, in case it wasn't issued:
  if [ -z "$1" ]; then set -- bundle exec rails server -p 3000 -b 0.0.0.0 "$@"; fi

  # 5: Run the checks only if the app code is executed:
  if [[ "$3" = "rails" ]]
  then
    # Clean up any orphaned lock file
    unlock_setup
    # 6: Wait until the setup 'lock' file no longer exists:
    while [ -f $APP_SETUP_LOCK ]; do wait_setup; done

    # 7: 'Lock' the setup process, to prevent a race condition when the
    # project's app containers will try to install gems and set up the
    # database concurrently:
    lock_setup
    # 8: Check if dependencies need to be installed and install them
    bundle install

    yarn install
    # 9: Run migrations or set up the database if it doesn't exist
    # Rails >= 6
    bundle exec rails db:prepare
    # Rails < 6
    # bundle exec rake db:migrate 2>/dev/null || bundle exec rake db:setup

    # 10: 'Unlock' the setup process:
    unlock_setup

    # 11: If the command to execute is 'rails server', then we must remove any
    # pid file present. Suddenly killing and removing app containers might leave
    # this file, and prevent rails from starting-up if present:
    if [[ "$4" = "s" || "$4" = "server" ]]; then rm -rf /app/tmp/pids/server.pid; fi
  fi
  # 12: Replace the shell with the given command:
  exec "$@"

I will only call out a few things specific to working with Rails from this file. Below, we are using nc(netcat) to verify that postgres is up before running any database commands. nc doesn't just ping the server; it is checking that the service is responding on a specific port. This check runs every second until it boots. Note: We are using the service name postgres to connect to the container.

  echo "DB is not ready, sleeping..."
  until nc -vz postgres 5432 &>/dev/null; do
    sleep 1
  done
  echo "DB is ready, starting Rails."

Next, we run our bundle and yarn install commands to make sure we have the latest dependencies. Once both of those have run, we will use the new Rails 6 db:prepare method to either set up the database or run migrations. This can be handled in many different ways but my preference is to use Rails tools. (Note: the comment is how you would do it on Rails 5 or older)

  bundle install

  yarn install
    # Rails >= 6
    bundle exec rails db:prepare
    # Rails < 6
    # bundle exec rake db:migrate 2>/dev/null || bundle exec rake db:setup

Lastly, we check for any dangling PID files left from killing and removing the app containers. If the PID files are left our rails server command will fail.

  if [[ "$4" = "s" || "$4" = "server" ]]; then rm -rf /app/tmp/pids/server.pid; fi

Start It Up

With all of this in place, we run docker-compose up, which will build and start our containers. It is also a great time to get coffee because building the app container will take some time. Once everything is built and up, point your browser at localhost:3000 and your homepage, or if this is a new app, it will display the default Rails homepage. If you need to run specs or any other Rails command, running docker-compose exec app bash in a separate terminal will launch a Bash session on the running app container.

Wrapping Up

We now have a base-level Rails Docker development environment. Here is a gist of all the three files referenced in this guide. Now that you have an understanding of how to set up a Docker development environment, check out Railsdock! It is a CLI tool we are working on that will generate most of this config for you. PR's appreciated!

In the next part of the series, we will set up more services and use Compose features to share config between similar services.

Favorite Non-Technical Activities

Nate Vick — Sat, 02 Nov 2019 14:56:23 +0000

What are your favorite non-technical activities?

React Concurrency Mode Released(Experimentally)

Nate Vick — Sat, 02 Nov 2019 14:00:57 +0000

Share your initial thoughts!

Rails 6 Released!!!

Nate Vick — Fri, 16 Aug 2019 20:26:34 +0000

What are you looking forward to the most?

https://weblog.rubyonrails.org/2019/8/15/Rails-6-0-final-release/

https://dev.to/hint/rails-6-upgrade-best-practices-5934

What are your favorite business tools that free you up to focus on development?

Nate Vick — Sun, 04 Aug 2019 15:49:50 +0000

I Updated Rails. Now Everything's on Fire.

Nate Vick — Mon, 03 Jun 2019 20:32:48 +0000

Originally posted on Hint's blog.

Following up on my best practices post, I have a prime example of one common issue that surfaces when upgrading a Rails app.

I decided to spike on upgrading a small internal app to Rails 6.0.0.rc1. The test suite is green on Rails 5.2.3 and Ruby 2.6.2, so I bumped the Rails version in the Gemfile.

source '<https://rubygems.org>'
git_source(:github) { |repo| "<https://github.com/#{repo}.git>" }

ruby '2.6.2'

gem 'rails', '6.0.0.rc1'

In the terminal, I ran bundle update rails and it updated without issue (I mentioned this is a small app, right?). Even on small apps, this should make you raise an eyebrow.

Skeptically, I ran the test suite.

Errors all the way down.

An error occurred while loading ./spec/services/step_processor_spec.rb.
Failure/Error: require File.expand_path('../../config/environment', __FILE__)

TypeError:
  Rails::SourceAnnotationExtractor is not a class/module
# /gems/gems/haml-rails-1.0.0/lib/haml-rails.rb:49:in `block in <class:Railtie>'
# ...more output...

Finished in 0.00027 seconds (files took 1.41 seconds to load)
0 examples, 0 failures, 17 errors occurred outside of examples

Based on the error, let's take a look at haml-rails and SourceAnnotationExtractor in Rails to see what changed and if there is a newer version of haml-rails.

But wait. You may ask: "Why not try upgrading the gem and move on?". Taking the time to understand the root cause of a problem provides confidence in the solution and clear direction to resolve broader symptoms throughout the codebase.

Taking the time to understand the root cause of a problem provides confidence in the solution and clear direction to resolve broader symptoms throughout the codebase.

Here is line 49 in lib/haml-rails.rb:

SourceAnnotationExtractor::Annotation.register_extensions('haml') do |tag|

Sure enough, there is a change in Rails 6 that moves SourceAnnotationExtractor to Rails::SourceAnnotationExtractor (PR #32065). The top-level class has been deprecated, but the deprecation warning does not apply to the sub-classes which raise the TypeError.

Knowing the source of the error, we can check if there is a newer version of haml-rails compatible with Rails 6. Again, you will find value by not just upgrading the gem, but understanding the changes in the newer versions. With haml-rails, there is a newer version that has a fix for the issue. By bumping up to that version, I'm able to run the test suite without errors and get back to the upgrade at hand.

Schedule a quick call with me to see how our expertise can keep your team delivering features during your Rails upgrade.