Forem: Alex Musayev

Sane Defaults

Alex Musayev — Fri, 28 Feb 2020 22:32:57 +0000

This post explains one of the practices we use to improve web service configuration management. As an example, we will use a Ruby on Rails application running on the Kubernetes environment. However, the general concept of sane defaults is technology-agnostic and entirely applicable to different programming languages and frameworks.

Imagine medium to large scale web service maintained by a team of 5 engineers. The production environment constitutes a distributed system of Rails servers, auxiliary microservices, and data storage. Everything is running on a Kubernetes cluster. As usual, besides the production, there is also a staging configuration — very similar, but not identical. It is smaller and cheaper. And there are also local development environments typically containing a minimal task-specific subset of services.

To maintain multi-environment Rails application, we need it to be configurable. Most common approach here is to follow The Twelve-Factor App methodology and store configuration in the system environment variables.

Say, we use dotenv gem to set some arbitrary configuration parameters for development environment:

# .env
MAIN_DOMAIN=app.local
REDIS_URL=redis://127.0.0.1:6379/0
AVATAR_SIZE=100
IMGPROXY_URL=https://imgproxy.local
SECRETS_IMGPROXY_KEY=key
SECRETS_IMGPROXY_SALT=salt

Now when we can address configuration values like so: ENV.fetch('MAIN_DOMAIN'). Dead-straight, cheap and easy? Sort of. Let's take a closer look at the potential shortcomings related to environment variables-based configuration in a mature Rails project.

Downsides of using environment variables

Configuration bloat. A more extensive project configuration may include hundreds of parameters. This fact itself is healthy — big configuration is better than hardcoding magic values all over the codebase, anyway. But you need to expect the config to require some gardening to prevent duplication, obsoletion, and other potential issues.

Each change in the configuration require manual effort. If your project uses dotenv on the development environment, each team member will have to maintain an up to date copy of the .env file with all the necessary configuration variables.

.env file may include individual settings — like a personal S3 bucket name you use for testing — and not supposed to be committed to the version control system. Therefore, it is not possible to automate config updates propagation amongst the team members.

Application startup errors. Typically it is a good practice to check mandatory configuration parameters during the application initialization stage and raise an error if something is missing (note #fetch method call in the above example). I will skip a detailed explanation here, assuming you already familiar with the fail-fast principle in system design. But in the practical context of this article, this validation means each configuration change may cause a hiccup in every team member's working process. Nobody likes it.

Fragile tests. Sometimes you need to use your configuration during testing, directly or indirectly. That means your test suite execution result depends on the system it is running on. A successful test run on a developer's machine does not guarantee the same result on CI, even in a containerized environment. And the reasons for the failure could be much less evident than the wrong DATABASE_URL.

Sane defaults

Our experience of maintaining configuration for larger web services shows that a significant part of configuration parameters don't change too often. Based on this observation, it is possible to mitigate most of the issues listed above by complementing configuration parameters with default values:

# const/defaults.rb

module Defaults
  AVATAR_SIZE = 100
  MAIN_DOMAIN = 'amplifr.local'
  IMGPROXY_MAX_SRC_RESOLUTION = 9000
end

Rails autoload mechanism will make Defaults module resolvable from everywhere in your project, making it possible to use the constants as a safe fallback each time you need the corresponding configuration parameter:

ENV.fetch('AVATAR_SIZE') { Defaults::AVATAR_SIZE }

Default constants make it safe to remove related environment variables from .env, or Dockerfile, ordocker-compose.yml, or whatever you use to keep your configuration. Configuration changes will automatically propagate over the version control system, preventing some of the application startup errors. And from now on, it is easy to keep an environment-agnostic test suite (just keep using Defaults::* values instead of the ENV).

Furthermore, I'd like to highlight that sane defaults plays well with the Twelve-Factor App methodology since environment variables always have higher priority over the constants.

There are some limitations, though: not every configuration parameter could have a default. Each default value should match the following conditions:

A sane default supposes to remain identical for all code execution environments, including Rails environments, individual developer machines, server instances, CI, etc.
Default values should never be secret. In other words, defaults do not apply to API keys or other configuration parameters that you don't want to keep in the repository.
A default does not change often or ever.

Making it easy to use

We solved the issues of the environment variables-based configuration, but ~~at what cost?~~ accessing configuration params with fetch calls is bulky and cumbersome:

ENV.fetch('AVATAR_SIZE', Defaults::AVATAR_SIZE)

Also, it would be a bad idea to rely on the developer to remember about the defaults each time they use a related configuration. And what about required parameter validation during the application initialization?

Luckily it is easy to solve all of these problems with config gem. It allows to define structured configuration with YAML files, similar to Rails' database.yml:

# config/settings.yml
---
main_domain: <%= ENV.fetch('MAIN_DOMAIN') { Defaults::MAIN_DOMAIN } %>
redis_url: <%= ENV.fetch('REDIS_URL') { Defaults::REDIS_URL } %>
avatar_size: <%= ENV.fetch('AVATAR_SIZE') { Defaults::AVATAR_SIZE } %>

imgproxy:
  base_url: <%= ENV.fetch('IMGPROXY_URL') %>
  key: <%= ENV.fetch('SECRETS_IMGPROXY_KEY') %>
  salt: <%= ENV.fetch('SECRETS_IMGPROXY_SALT') %>
  max_src_resolution: <%= ENV.fetch('IMGPROXY_MAX_SRC_RESOLUTION') { Defaults::IMGPROXY_MAX_SRC_RESOLUTION } %>

Being added to your Gemfile, config gem builds a globally available settings object during Rails application startup, providing a set of concise accessors to the configuration params, like Settings.main_domain or Settings.imgproxy.base_url.

Summary

Here is a list of benefits you gain from complementing environment-based configuration with sane defaults:

Prevent configuration bloat.
Automate shared configs propagation over developer machines.
Prevent startup errors caused by the lack of shared configuration params.
More stable tests.

Bundling Custom Assets with Sprockets

Alex Musayev — Fri, 28 Feb 2020 16:08:05 +0000

This article explains how to process custom asset files with Rails Asset Pipeline by customizing Sprockets configuration.

Let's consider an example case of Rails application that uses client-side Mustache templates rendering. What would be an efficient way to access the templates from JavaScript?

There are several approaches here. The basic one is to nest the templates directly into the HTML view code. Like this:

<script type="text/template" id="post-template">
  <h2>{{ title }}</h2>
  <p>{{ body }}</p>
</script>

But this is far from ideal:

Most probably, there will be more templates, and you will need to share some of them between different views. Therefore it makes sense to keep them organized, instead of scattering templates all over the HTML.
Nesting a template is an unnecessary manual operation. It is better to have templates available from JavaScript right away, instead of relying on the fact that you won't forget to add one.
Sometimes nesting a template within <script> element breaks syntax highlighting, therefore keeping templates in individual *.mustache files usually works better. However, this depends on your text editor configuration.

To solve these issues, let's make Rails Asset Pipeline generate a JSON object within the JavaScript bundle, and populate this object from templates directory at app/assets/templates. For the consumer, it will look like this:

window.Templates = {
  post: "<h2>{{ title }}</h2><p>{{ body }}</p>",
  comment: "..."
}

This way, the templates will be preloaded with the shared bundle, and remain available from every view that is using it.

Step 1. Register new MIME type

In a typical Rails application, custom Sprockets configuration should be defined under config.assets.configure block within the Application class. So the examples below are based on the assumption that a block like this already exists in your config/application.rb:

module MyApplication
  class Application < Rails::Application
    config.assets.configure do |env|

      // Custom Sprockets configuration should go here

    end
  end
end

MIME type registration is straightforward. Just define an association between a new type, and the list of file extensions you plan to process:

env.register_mime_type('text/mustache', extensions: ['.mustache'])

Step 2. Register new assets transformer

The most important part here is the callback. It is a callable object that converts asset file content from source_type to something of target_type. The callback should return a hash with processed content in :data key:

source_type = 'text/mustache'
target_type = 'application/javascript'

callback = -> (input) { { data: '// HELLO ${input[:name]}' } }

env.register_transformer(source_type, target_type, callback)

Lambda expression could be unsuitable for real-life situations. A service class, defined somewhere outside Rails configuration, will be a better choice. But for now, let's finish the boilerplate first. A real transformer example will follow.

Step 3. Enable custom file type processing

To achieve that, add a regular expression to the precompile array in the assets initializer (config/initializers/assets.rb):

Rails.application.config.assets.precompile += [/\.(?:mustache)\z/]

Step 4. Bundle transformed assets

After the new transformer is registered, both require and require_tree directives will play well with the custom assets type.

If you keep Mustache templates under app/assets/templates, adding this line to app/assets/javascripts/application.js will inject transformer callback output for each *.mustache file:

//= require_tree ../templates

Dummy lambda-transformer from the previous step will replace this line with a list of commented file names like this:

// HELLO template_file_name
// HELLO another_template_file_name
// ...

To see it working, don't forget to restart Rails server and make sure to purge the assets cache.

And here is an actual transformer example, that generates JavaScript object from Mustache template files:

class MustacheTransformer
  def self.call(input)
    key = input[:name]
    body = JSON.dump(input[:data])
    obj = 'window.Templates'
    { data: "#{obj} = #{obj} || {}; #{obj}['#{key}'] = #{body}" }
  end
end

Besides :data, the input object contains several other extension fields, that could be valuable during content transformation. Check out the official reference for the full list.

The case of Mustache templates elaborated in this article is just a concrete example of extending Sprockets. Besides templates, it is possible to bundle any other assets with arbitrary transformation logic that suit your needs.

Peace ✌️

References:

Original discussion on GoRails forum
Extending Sprockets, official guide
Sprockets::Transformers API reference
Precompiling assets with Rails Assets Pipeline
Complete list of MIME types
handlebars_assets gem: a ready-to-use replacement for particular case of embedding templates into Rails assets bundle.

You probably don't need a version manager for Ruby

Alex Musayev — Fri, 28 Feb 2020 15:57:30 +0000

This manual explains how to install a recent stable version of Ruby on Linux, without using rbenv or alternative version managers. My goal was to find a fast and reliable approach for provisioning expendable virtual machines for Rails development environment.

Basic assumptions:

Installation speed matters.
Only one version of Ruby is required system-wide.
We're using Ubuntu, or any other Debian-based Linux distribution.

Step 1. Install prebuilt Ruby from Brightbox

Notice the dev package that should be installed in addition to the primary one. It is required to build native extensions for Ruby gems.

sudo apt-get install software-properties-common
sudo apt-add-repository ppa:brightbox/ruby-ng
sudo apt-get update
sudo apt-get install ruby2.6 ruby2.6-dev

Brightbox manual proposes a tool called ruby-switch that can help to switch between multiple Rubies on the same system. Since there will be only one of them, this step is unnecessary.

Step 2. Make gem work without sudo

By default, gem will try to install new gems to the system folder (e.g. /var/lib/gems/2.6.0), which is no good. Ruby version managers override this path with something under user home directory. But the same operation could be done manually. To permanently set target directory to the user home, and these lines to your ~/.bashrc:

export GEM_HOME=$HOME/.gem/ruby/2.6.0
export PATH=$HOME/.gem/ruby/2.6.0/bin:$PATH

Here is the most important thing you need to know about Ruby version managers to understand what exactly they do in system configuration:

"RubyGems' default local repository can be overridden with the GEM_PATH and GEM_HOME environment variables. GEM_HOME sets the default repository to install into. GEM_PATH allows multiple local repositories to be searched for gems" — rubygems.org

Step 3. Run a quick ad-hoc test

Log into the shell, and execute these commands to ensure gem and ruby binaries are available, gem home path is configured correctly, and native extensions could be built:

cd
gem install bundler rails
rails new testapp

Or just execute gem env to see the paths without installing anything.

Here is a full Vagrantfile to provision new Linux VM with Ruby development environment:

dreikanter / vagrant-rails

Vagrant configuration for Rails development environment

Will install:

Ubuntu Server (19.04 LTS)
Ruby (via ruby-install)
PostgreSQL
Redis
ElasticSearch
NodeJS
Yarn

Set up:

brew cask install virtualbox vagrant
vagrant up

Rails app setup after first log in (assuming Vagrant file is in you project root):

cd /app
bundle install
yarn install
bundle exec rails db:drop db:create db:migrate db:seed --trace

Running Rails app server:

rails s

Running webpack-dev-server:

bin/webpack-dev-server

View on GitHub

Peace ✌️