Forem: Dmitry Salahutdinov

Setup web-page screenshots within a minute

Dmitry Salahutdinov — Mon, 06 Apr 2020 09:40:43 +0000

For many years PhantomJS is used to generate a web-page screenshot. As we have a Ruby web-application on Kubebernetes, it brought some inconveniences: we had to install NodeJs and PhantomJS rights into the main Ruby Docker image:

# Phantomjs is installed via npm to avoid HUGE dependencies such as QT and others.
RUN npm install phantomjs-prebuilt \
  && ln -s /node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs /usr/local/bin/phantomjs \
  && rm -rf /tmp/phantomjs

COPY ./fonts/Helvetica.ttf /usr/share/fonts/truetype/Helvetica.ttf

It causes all the application instances, like web, sidekiq, and others, had these low-level dependencies installed even if they did not need to generate screenshots.

For some time PhantomJS are not supported anymore. The new alternative is to use Google Chrome within the "headless" mode, a great example of doing it from the command line:

google-chrome --headless --window-size=1180,768 --screenshot=/tmp/screenshot.png https://amplifr.com/en

We need to get google-chrome installed to a machine, so using this command rights from the ruby code doesn't change anything fundamentally.

That is why I decided to extract it into separated HTTP based micro-service for capturing web-page and returning an image.

Here it is https://github.com/dsalahutdinov/screenshot
It could be touched with curl:

curl --output screenshot.png http://screenshot.local/screenshot\?width\=1180\&height\=768\&url\=https://amplifr.com/en

Or with browser as well:

Internals

Internally it is simple HTTP-wrapper for running Chrome and return resulting image file. Nothing more on it. Screenshot tool is Kubernetes-ready, and you can get it up in a minute by running kubectl or using predefined helm-chart.

Use Helm Chart

The predefined helm-chart is located here. This repo also works as a helm-repository on top of Github pages. Add the repository to your helm settings:

$ helm repo add screenshot-helm https://dsalahutdinov.github.io/screenshot-helm/
"screenshot-helm" has been added to your repositories

And deploy the chart into current Kubernetes cluster:

$ helm install screenshot-helm/screenshot --name screenshot --namespace screenshot

NAME:   screenshot
LAST DEPLOYED: Sat Apr  4 16:59:02 2020
NAMESPACE: screenshot
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME         AGE
screenshot  1s

==> v1/Pod(related)
NAME                      AGE
screenshot-76948b-rrq77  1s

==> v1/Service
NAME         AGE
screenshot  1s

Since then, service will run as "internal" without any ingress resources. You can access it within the cluster by its internal DNS name, [release].[namespace].kubernetes.local, or with just short name [release].[namespace]:

curl --output screenshot.png \
     http://screenshot.screenshot/screenshot/?width=1180&height=768&url=https://amplifr.com/en

Run on Minikube

Here is the manual for running screenshot service into Minikube local cluster with using "external" domain name screenshot.local:

# starts minikube
minikube start

# gets minikube ip address and adds it as screenshot.local domain address
echo "$(minikube ip) screenshot.local" | sudo tee -a /etc/hosts

# adds helm repository
$ helm repo add screenshot-helm 

# installs screenshot service with the enabled ingress
$ helm install screenshot-helm/screenshot \
  --name screenshot \
  --namespace screenshot \
  --set ingress.enabled=True \
  --set ingress.hosts\[0\].host=screenshot.local \
  --set ingress.hosts\[0\].paths\[0\]=/screenshot

NAME:   screenshot
LAST DEPLOYED: Mon Apr  6 10:51:54 2020
NAMESPACE: screenshot
STATUS: DEPLOYED

RESOURCES:
==> v1/Deployment
NAME        AGE
screenshot  0s

==> v1/Pod(related)
NAME                         AGE
screenshot-58dd688598-ml9nw  0s

==> v1/Service
NAME        AGE
screenshot  0s

==> v1/ServiceAccount
NAME        AGE
screenshot  0s

==> v1beta1/Ingress
NAME        AGE
screenshot  0s


NOTES:
1. Get the application URL by running these commands:
  http://screenshot.local/screenshot

After the start, service will be available to access with amplifr.local domain. Other supported options of helm-chart, like resources, tls, affinity you can find here.

Usage

Using ruby is very straightforward:

options = {
  height: 1024, width: 768, url: 'https://amplifr.com/en'
}
response = Faraday.new(
  'http://screenshot.screenshot/screenshot?' + options.map { |k, v| "#{k}=#{v}" }.join("&")
).get

File.open('screenshot.png', "wb") {  |f| f.write(response.body) }

Conclusion

The screenshot tool is straightforward and solves the simple, practical task - web-page screenshot generation. You can run into a Kubernetes cluster in a minute if you would need a web-page screenshot generation one day.

Monitoring external services with Prometheus and Grafana

Dmitry Salahutdinov — Mon, 23 Mar 2020 05:07:21 +0000

Preface

Time ago, web-applications depended only on self-written code or libraries. It was easy to diagnose any problem, plainly investigating how an application work.

Today, modern web-applications tend to have many external dependencies, such as microservices or external SaaS. It makes your system depend not only on the internal factors but on the reliability of external services as well. When something fails, it's essential to localize a problem and find out which side caused it.

That is why monitoring of external services call is crucial.

I work for Amplifr, the social media automation platform. Our Ruby core application depends on many external APIs: Facebook, Instagram, LinkedIn, Pinterest, Bitly, Stripe, Amplitude, Intercom, and many others.
We also are integrated with many self-hosted micro-services, such as imgproxy for resizing images, browser screenshot generator, social services bots, and other internal domain-specific services.

When an application has so many integrations, it's handy to have a quick traffic details overview:

how many requests application does
how much time consumes an average request
how many of the attempts fail

That was my second point to set monitoring up and write my own solution.

Metrics

As most APIs work over HTTP, we will monitor the HTTP requests.

Meet my gem yabeda-http_requests. It is the new one of the Yabeda monitoring solution by Evil Martians. If you are not familiar with it, here is the great article to start.

In a nutshell, Yabeda - is the open-source Ruby framework for collection metrics of Ruby processes (Rack/Rails, Sidekiq,p Puma) and exporting them into Prometheus (or any other adapter).

yabeda-http_requiests wraps all the external HTTP requests with the help of sniffer gem. Sniffer patches all popular HTTP libraries and logs performing requests. By using middleware, yabeda-http_requests gem calculates and stores metrics. Then metrics could be exported with yabeda-prometheus gem.

To get started with it you need:

to add the gem into your dependencies (Gemfile)
set up metrics exporter for Prometheus
tune up Prometheus to scrape the metrics endpoint
import predefined dashboard into your Grafana config

You could find all the details yabeda-http_requests at the README.md.

Let's have a look at providing metrics & charts.

What to monitor?

The first release of yabeda-http_requests includes some basics:

RPM

The basic metric is RPM (request per minute). It shows how many HTTP request application does.

Formally, yabeda-http_requests collect a http_request_count as counter. It is separated by quest host, port and method:



# HELP http_response_total A counter of the total number
# of external HTTP responses.
http_response_total{host="twitter.com",port="443",method="GET",status="200"} 131.0
http_response_total{host="dev.to",port="443",method="GET",status="200"} 131.0

Than, using rate function:



sum by (host) (rate(http_response_total[1m]) )

renders the RPM chart:

It is easy to regroup the same chart by status:

❗ "RPM by status" chart uses another metric, which is called http_response_total which logs the HTTP responses.

Request duration

When you want to investigate which one of the dependent components causes a slow-down, there is a histogram duration data and the corresponding chart. It shows how many times consumes the slowest request:

Example Application

This section is for practical approach lovers. Here is the example of web-application. It shows how all things work together: web-application, metrics collector, exporter, and Prometheus & Grafana.

The example has a docker-compose config for automatic setup. Just run the following:



$ git clone https://github.com/yabeda-rb/yabeda-http_requests
$ cd yabeda-http_requests/example
$ docker-compose up

This command runs the rack-application, Sidekiq, Prometheus, and Grafana.
Rack application adds random web scrapping jobs to the sidekiq. Navigate to http://localhost:9292 to enqueue some jobs.

"Sidekiq instance" runs background scrapping tasks, collects metrics, and exposes an endpoint with data for the Prometheus.
Here is the basic sidekiq configuration, showing how the sidekiq process run its web-server:



# sidekiq config
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL') }

  # configures all the Yabeda metrics
  ::Yabeda.configure!

  # expose 9394 port with /metrics path for prometheus
  ::Yabeda::Prometheus::Exporter.start_metrics_server!
end

Navigate to http://localhost:9394/metrics to see metrics endpoint, and you will see the metrics values formatted as text:



# TYPE http_request_total counter
# HELP http_request_total A counter of the total number of external HTTP requests.
http_request_total{host="twitter.com",port="443",method="GET"} 16922.0
http_request_total{host="dev.to",port="443",method="GET"} 17083.0
http_request_total{host="amplifr.com",port="443",method="GET"} 451.0
http_request_total{host="www.sitepoint.com",port="443",method="GET"} 428.0
# TYPE http_response_total counter
# HELP http_response_total A counter of the total number of external HTTP responses.
http_response_total{host="twitter.com",port="443",method="GET",status="200"} 16910.0
http_response_total{host="dev.to",port="443",method="GET",status="200"} 17080.0
http_response_total{host="amplifr.com",port="443",method="GET",status="301"} 451.0
http_response_total{host="www.sitepoint.com",port="443",method="GET",status="200"} 428.0
http_response_total{host="twitter.com",port="443",method="GET",status="503"} 1.0
# TYPE http_response_duration_milliseconds histogram
# HELP http_response_duration_milliseconds A histogram of the response                                                duration (milliseconds).
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="0.5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="1"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="2.5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="5"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="10"} 0.0
http_response_duration_milliseconds_bucket{host="twitter.com",port="443",method="GET",status="200",le="25"} 0.0
...

Prometheus scrapes http://sidekiq:9394/metrics periodically and stores the metrics into its database.

Here is the overview diagram of how it works together:

Grafana visualizes the data. Navigate to http://localhost:3000/d/OGd-oEXWz/yabeda-external-http-requests. Use admin/foobar to login and see the template dashboard:

❗You could import prepared Grafana dashboard here.

Conclusion

The monitoring of external HTTP calls is essential. It is handy to an overview of your outgoing application traffic. It also helps to diagnose slow-down issues of dependent services.

Yabeda framework and the yabeda-http_requests let to quickly set up metrics and visualize them using a predefined Grafana dashboard template.

If you have any other ideas, please do not hesitate to contribute: by writing an Issue, sending PR, or leaving the comment.

Thank you for reading.

Kubernetes-native development: sync files with Ksync

Dmitry Salahutdinov — Mon, 27 Jan 2020 20:42:50 +0000

Previous post I started to set up continuous deployment for the development process. Skaffold is a handy tool for doing that.

Let's go deeper this time and imagine the next steps for making up a full-featured "local" development environment with Kubernetes. Running ruby code outside of the local machine keeps it clear and doesn't impose any resource limits.

It is the essence of the idea "Kubernetes-native development": to run all the code at a remote or local Kubernetes cluster.

The issue I try to investigate here is file synchronization. If we edit code locally and run it somewhere remotely, we could get sources out of sync. Unequal sources could cause unexpected results.

For local development with Docker it is common to use Docker volumes. The same idea could be applied here in terms of Kubernetes Persistent Volumes. I failed to use persistent volume, mapping them to the Minikube host machine. And map to local Mac directory onto the Minikube host machine.
Even if it worked well, there are other issues with that approach:

a cluster could be remote. It's a big deal to map the local directory into the remote cluster host machine
having additional persistent volume manifests only for developer's needs is obsessive

I decided to synchronize files instead and try out the ksync tool for Kubernetes.

Ksync speeds up developers who build applications for Kubernetes. It transparently updates containers running on the cluster from your local checkout. This enables developers to use their favorite IDEs, such as Atom or Sublime Text to work from inside a cluster instead of from outside it. There is no reason to wait minutes to test code changes when you can see the results in seconds.

If want to do something like docker run -v /foo:/bar with Kubernetes, ksync is for you!

Let's set up file sync for already known Skaffold Ruby example. We'll need the same tooling as in the previous article: kubectl, minikube, skaffold, and installed ksync.

Install ksync

Find the installation details here, or just run for MacOS:

$ curl https://ksync.github.io/gimme-that/gimme.sh | bash

Setup ksync

Then we have to initialize ksync.

$ ksync init

This command initializes ksync and installs the server component on your cluster. One part of ksync works as the entity in the Kubernetes cluster and has to be set up. See more details about ksync architecture.

Check out the Ruby example sources:

$ git clone git@github.com:GoogleContainerTools/skaffold.git
$ cd examples/ruby

We have to enable ksync file watch with running the watch command:

$ ksync watch

Enable watching changes within ($pwd)/backend folder and sync them back and forth with /app/ directory of running containers:

We will use the specific label app.kubernetes.io/ksync: 'true' presence to determine the scope of pod container to track changes with:

$ ksync create --selector app.kubernetes.io/ksync=true --reload=false -n default $(pwd)/backend /app/

Ruby app sample

Change the k8s/deployment.yaml to add the specific tag app.kubernetes.io/ksync: 'true' to pod's containers.

apiVersion: v1
kind: Service
metadata:
  name: ruby
spec:
  ports:
  - port: 9292
    targetPort: 9292
  type: LoadBalancer
  selector:
    app: ruby
--------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ruby
spec:
  selector:
    matchLabels:
      app: ruby
  template:
    metadata:
      labels:
        app: ruby
        app.kubernetes.io/ksync: 'true'
    spec:
      containers:
      - name: ruby
        image: ruby-example
        ports:
        - containerPort: 9292
        env:
          - name: RACK_ENV
            value: "development"

And deploy the application:

$ skaffold run --tail
Generating tags...
 - ruby-example -> ruby-example:v1.1.0-118-g92f37b200-dirty
Checking cache...
 - ruby-example: Found Locally
Tags used in deployment:
 - ruby-example -> ruby-example:de8f4f77b01e508f5d862a9ef84d9f33a50d03a88225b0e1b26ddb04d948cded
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
 - service/ruby created
 - deployment.apps/ruby created
[ruby-66d8d5758-g6m9r ruby] Puma starting in single mode...
[ruby-66d8d5758-g6m9r ruby] * Version 4.3.1 (ruby 2.7.0-p0), codename: Mysterious Traveller
[ruby-66d8d5758-g6m9r ruby] * Min threads: 0, max threads: 16
[ruby-66d8d5758-g6m9r ruby] * Environment: development
[ruby-66d8d5758-g6m9r ruby] * Listening on tcp://0.0.0.0:9292
[ruby-66d8d5758-g6m9r ruby] Use Ctrl-C to stop

This time we won't run skaffold in development mode to prevent the doubled file sync by ksync and skaffold concurrently.

File sync in action

Any changes made on local files are automatically synchronized with remote containers:

$ echo "gem 'oj'" >> backend/Gemfile

The similar ksync watch output means that file is synchronized:

INFO[0008] folder sync running                           pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] updating                                      pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] update complete                               pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0012] updating                                      pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0012] update complete                               pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0080] updating                                      pod=ruby-console-69674c499f-97m79 spec=massive-koala
INFO[0080] updating                                      pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0080] update complete                               pod=ruby-66d8d5758-g6m9r spec=massive-koala
INFO[0080] update complete                               pod=ruby-console-69674c499f-97m79 spec=massive-koala

Let check the container's Gemfile to see if changes are applied:

$ kubectl exec -it $(kubectl get pods --selector app.kubernetes.io/ksync=true | tail -n 1 | awk '{print $1}') --  cat Gemfile
source 'https://rubygems.org'

gem 'rack'
gem 'rack-unreloader' # for dynamic reloading
gem 'puma'
gem 'oj'

Then we could get inside the pod and run bundle to update Gemfile.lock dependencies lock-file:

$ kubectl exec -it $(kubectl get pods --selector app.kubernetes.io/ksync=true | tail -n 1 | awk '{print $1}') -- bundle

Using bundler 2.1.2
Using nio4r 2.5.2
Using oj 3.10.1
Using puma 4.3.1
Using rack 2.1.1
Using rack-unreloader 1.7.0
Bundle complete! 4 Gemfile dependencies, 6 gems now installed.

After bundle had finished and updated the lock-file, we got it synchronized back with a local machine!

Conclusion

Perfect, we set up the two-way file synchronization and get it working with ksync. From now, any source file updates made automatically by ruby-code, get back to a local machine.

The sync would also work well for generated files, such as rails g migration or even for initial rails project generation with rails new my_project.

Ruby conferences to attend in 2020

Dmitry Salahutdinov — Sat, 25 Jan 2020 19:02:46 +0000

Do you think Ruby is dying?
Meanwhile, Rubyists get the sold-out at more than 20 conferences around the world.

While looking for, I made a list of Ruby conferences you can attend in 2020. Here it is:

Birmingham On Rails

Date: January 31
Location: 🇺🇸USA, Birmingham, AL

Birmingham on Rails is a forward thinking Ruby on Rails conference with the purpose of building a network of software professionals and educating them on the latest developments and uses of the Rails platform.

RubyFuza

Date: February 6-7
Location: 🇿🇦South Africa, Cape Town
Follow: @rubyfuza

Rubyfuza is the longest run Ruby conference in South Africa. 2020 will be the 9th event held in Cape Town This year we will be opening the conference up!

ParisRB Conf

Date: February 18-19
Location: 🇫🇷France, Paris
Follow: @parisrb

Meet fellow rubyists from around the world, and expand your Ruby horizons!

The year 2020 marks the 25th Anniversary of the first release of Ruby and we are lucky (and filled with gratitude) to welcome Ruby creator Yukihiro Matsumoto who will help us open ParisRB with a Keynote speech.

RubyConf AU

Date: February 20-21
Location: 🇦🇺Australia, Melbourne
Follow: @rubyconf_au

With a mixture of local and international speakers, there will be two day of talks. We'll cover topics that interest, excite, and delight Rubyists and the Ruby-curious of all levels. There will also be social events hosted where you will get the opportunity to meet like-minded people!

Wroclove.rb

Date: March 20-22
Location: 🇵🇱Poland, Wroclaw
Follow: @wrocloverb

The main goal of the conference is to help Ruby professionals become better at what they do.

wroclove.rb serves best to people who are already fluent with Ruby. The ideas presented at our conference are very advanced. They are meant to inspire the way we write code, the way we help our clients and users. It’s not uncommon for us to present you ideas that come from the Java and .NET worlds, we believe that their ideas can have a great influence on our community. There are good  reasons, why our conference is called “the best Java conference in the Ruby world”.

BRUG.by

Date: March 21
Location: 🇧🇾Belarus, Minks
Follow: @ByRoR

THE INTERNATIONAL TECH CONFERENCE FOR RUBY ENTHUSIASTS IN MINSK.

During the conference, we want to focus on the topic of Engineering Happiness. What gets us satisﬁed with our work? What processes, tools, industry news make us happy? What does an engineer need to be happy? What tips and tricks do companies use to increase developers’ happiness? All in all, is happiness really so important?

If you have something to say on that subject, we are looking forward to your 40-minute presentation (Keynote, PPT or any other) with slides or a live coding session. The QA block will also take 20 minutes.

RubyDay

Date: April 2
Location: 🇮🇹Verona, Italy
Follow: @rubyday

RubyDay 2020 is the 8th edition of the Italian Ruby conference, organized by GrUSP, organizers of events like phpday and jsday. Event is international and all sessions will be in English.

The event’s goal is to allow all Rubyists to meet and share experience while having fun and networking in an enjoyable context.

RubyWine

Date: April 4
Location: 🇲🇩Moldova, Chișinău
Follow: @RubyMeditation

Being first of its kind in Moldova, Ruby Wine is intended to promote Ruby programming language among country and fortify existing Ruby community. It will bring together IT professionals for knowledge sharing and let them explore new insights of Ruby's possibilities. This event is a perfect mix of top-notch talks by talented Ruby developers and amazing Moldovan atmosphere.

RubyKaigi

Date: April 9-11
Location: 🇯🇵Japan, Matsumoto
Follow: @rubykaigi

RubyKaigi is an annual conference held in Japan for Ruby programmers that was held for the first time in 2006.

RubyConfBy

Date: April 18
Location: 🇧🇾Belarus, Minks
Follow: @RubyConfBY

The 6th large technological Ruby, Rails and related technologies conference. Leaders from Eastern+Western Europe and USA will gather to discuss technological aspects of development with Ruby.

RubyConfIndia

Date: April 25-26
Location: 🇮🇳India, Goa
Follow: @rubyconfindia

RubyConf India is a global event complementing other RubyConf events across the world.

This year our event is a 2 day, single track event focused on Ruby Language, Framework and Tools.

RailsConf

Date: May 5-7
Location: 🇺🇸USA, Portland
Follow: @railsconf

RailsConf is the world’s largest gathering of Rails developers, brought together to further discussion and learning about building, managing, and testing Rails applications. With a specific focus on Rails, conference topics can range from new users to administration to advanced techniques.v

BalkanRuby

Date: May 15-26
Location: 🇧🇬Bulgaria, Sofia
Follow: @BalkanRuby

Balkan Ruby's aim is to bring world-class talks to the Balkan area and allow the local community to interact with the wider Ruby community from Europe and around the world.

Saint P Ruby Conf

Date: June 6-7
Location: 🇷🇺Russia, Saint Petersburg
Follow: @saintpruby

Ruby community based in Saint Petersburg, Russia. We do meetups, drinkups and conferences.

Ruby Unconf

Date: June 6-7
Location: 🇩🇪Germany, Hamburg
Follow: @RubyUnconfEU

What exactly is an Unconference? It is an Barcamp style event. It works like this: Everybody can propose a talk. Everybody votes for the talks they want to see. The most voted talks will be held. You decide what will happen!

RedDotRubyConference

Date: June? (is not know yet)
Location: 🇸🇬 Singapore
Follow: @reddotrubyconf

According to that tweet the conference will get back this year.

BrightonRuby

Date: July 3
Location: 🇬🇧UK, Brighton
Follow: @brightonruby

It’s a friendly, inclusive, single day, single track event for Rubyists and the Ruby-curious.

We’re nice. The speakers are ace.

RubyConf Kenya

Date: July 23-25
Location: 🇰🇪Kenia, Nairobi
Follow: @NairubyKE

Nairobi Ruby community driven conference.

Euruko

Date: August 21-22
Location: 🇫🇮Finland, Helsinki
Follow: @euruko

Euruko is a Ruby conference organised annually, each year in a different European city. Join us in Helsinki, Finland on 21st–22nd August 2020 for two days of Ruby.

GrillRB

Date: unknown, end of August
Location: 🇵🇱Poland, Wroclaw
Follow: @grill_rb

GrillRB is Polish outdoor single-track community-driven conference fully dedicated to Ruby on Rails framework and Ruby language. During the two day event we are going to discuss the topics which are the most crucial for Ruby community, such as: best practices, development process and the future of Ruby. Come and exchange the knowledge and experience, talk over projects with other Rubyists in relaxing & enjoyable environment.

RubyC

Date: September 12-13
Location: 🇺🇦Ukraine, Kyiv
Follow: @rubyc_eu

RubyC is a European conference devoted to Ruby, Rails and other related technologies.

Every year we invite hundreds of Ruby enthusiasts and developers from various programming groups to exchange knowledge, discuss the latest news, learn from one another, and generally have a wonderful time in Kyiv, Ukraine.

RubyRussia

Date: unknown, end of September
Location: 🇷🇺Russia, Moscow
Follow: @railsclub_ru

The conference is devoted to all aspects of Ruby and Rails development. We are the largest Ruby focused pro conference in Russia.
Ruby Russia is a unique place where you can socialize with colleagues, receive and share experience, meet old friends and learn something new.

RubyConf Thailand

Date: October 16 — 17
Location: 🇹🇭Thailand, Bangkok
Follow: @rubyconfth

RubyConf TH is a community and volunteer-powered event. We also depend on the awesomeness of the whole Ruby community to contribute to a terrific experience. So help us to make it a success!

RubyConf

Date: November 18-20
Location: 🇺🇸USA, Nashville
Follow: @rubyconf

RubyConf is the world’s largest and longest-running gathering of Ruby enthusiasts, practitioners, and companies.

Please, let me know if I missed any event.

Continuous deployment Ruby application to Minikube with Google's Skaffold

Dmitry Salahutdinov — Tue, 21 Jan 2020 18:44:39 +0000

Kubernetes has made it very easy to deploy and scale applications to the cloud than ever. Still, the development process has not evolved at the same speed.

Today, most developers try to either run parts of the infrastructure locally with docker or test these integrations directly in the cluster via CI jobs or the "docker build, docker push, kubectl apply" cycle. Manual deployment is painful and incredibly slow.

In this article, we will see how to set up continuous deployment of simple Ruby application to Minikube to reduce the development cycle time using the Google's Skaffold tool.

Skaffold handles the workflow for building/pushing Docker images and deploying your application. It helps to reuse your actual application production configuration (Kubernetes manifests, Helm chart, Docker Compose config) for the local continuous development cycle. Skaffold also supports file sync to prevent the full container rebuild/redeploy.

Preparation

First of all, let's install the required tooling.

Install Minikube

Here are the commands to run for Mac, more details could be found here:

$ brew install minikube
$ minikube version
minikube version: v1.5.2
commit: 792dbf92a1de583fcee76f8791cff12e0c9440ad-dirty
$ minikube start

$ eval $(minikube docker-env)
$ kubectl config use-context minikube

The line eval $(minikube docker-env) set the docker client to use Minikube's Docker engine. It allows us to avoid docker image pushing and simplifies the full redeploy cycle.

We've also changed to actual kubectl context to minikube.

Install Skaffold

Here are the documentation details of Skaffold installing, for running on Mac just use:

$ brew install skaffold

Ruby/Rack application

We will use the simplest Rack-application from the Skaffold's github:

$ git clone git@github.com:GoogleContainerTools/skaffold.git
$ cd examples/ruby

Skaffold dev

Just run skaffold dev and you will see similar output:

$ skaffold dev
Listing files to watch...
 - ruby-example
Generating tags...
 - ruby-example -> ruby-example:v1.1.0-118-g92f37b200-dirty
Checking cache...
 - ruby-example: Not found. Building
Found [minikube] context, using local docker daemon.
Building [ruby-example]...
Sending build context to Docker daemon  6.144kB
Step 1/6 : FROM ruby:2.7
 ---> fb53c5f433da
Step 2/6 : WORKDIR /app
 ---> Using cache
 ---> 0c2d04fabca0
Step 3/6 : ADD Gemfile* ./
 ---> Using cache
 ---> 06d6623561fc
Step 4/6 : RUN bundle install
 ---> Using cache
 ---> 9bbcf218dda5
Step 5/6 : ADD . ./
 ---> Using cache
 ---> 478cca6b9c4f
Step 6/6 : CMD ["bundle","exec","puma"]
 ---> Running in 434b1923c060
 ---> 988d9e9e7bf4
Successfully built 988d9e9e7bf4
Successfully tagged ruby-example:v1.1.0-118-g92f37b200-dirty
Tags used in deployment:
 - ruby-example -> ruby-example:988d9e9e7bf4b44beb62a8aaa7104cdf407a8523db0bdc7fe331600d374ba274
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
 - service/ruby created
 - deployment.apps/ruby created
Watching for changes...
[ruby-75c5b89895-rd2lb ruby] Puma starting in single mode...
[ruby-75c5b89895-rd2lb ruby] * Version 4.3.1 (ruby 2.7.0-p0), codename: Mysterious Traveller
[ruby-75c5b89895-rd2lb ruby] * Min threads: 0, max threads: 16
[ruby-75c5b89895-rd2lb ruby] * Environment: development
[ruby-75c5b89895-rd2lb ruby] * Listening on tcp://0.0.0.0:9292
[ruby-75c5b89895-rd2lb ruby] Use Ctrl-C to stop

At that time Skaffold tool has built the Docker image, tagged it with a specific tag, and deployed application to Minikube cluster. Then it started to wait for the source file changes.

The result could be seen here:

$ curl -s $(minikube service ruby --url)
Hello Skaffold!

Hot reload

As configured at the Skaffold config - it watches for *.rb files and reloads all the changes rights into the container sources. The sample Rack application also supports for reloading with help of the rack-unreloader ruby-gem.

If we change the app.rb file to return other string, like

class App
  def self.call(env)
    [ 200, {"Content-Type" => "text/html"}, ["Hello Skaffold!!!"]]
  end
end

We will see skaffold sync files into the container:

Syncing 1 files for ruby-example:988d9e9e7bf4b44beb62a8aaa7104cdf407a8523db0bdc7fe331600d374ba274
Watching for changes...

As the Rack application supports file reloading, it will now produce the different output:

$ curl -s $(minikube service ruby --url)
Hello Skaffold!!!

Automatic redeploy

If we change others files, such as backend/Dockerfile, Skaffold will autmatically rebuild the Docker image, and redeploy the application:

echo "gem 'oj'" >> backend/Gemfile

We'll see full redeploy:

Generating tags...
 - ruby-example -> ruby-example:v1.1.0-118-g92f37b200-dirty
Checking cache...
 - ruby-example: Not found. Building
Found [minikube] context, using local docker daemon.
Building [ruby-example]...
Sending build context to Docker daemon  6.144kB
Step 1/6 : FROM ruby:2.7
 ---> fb53c5f433da
Step 2/6 : WORKDIR /app
 ---> Using cache
 ---> 0c2d04fabca0
Step 3/6 : ADD Gemfile* ./
 ---> Using cache
 ---> 7b4671e7469d
Step 4/6 : RUN bundle install
 ---> Using cache
 ---> 0331311c04f3
Step 5/6 : ADD . ./
 ---> 67e2cf138ed3
Step 6/6 : CMD ["bundle","exec","puma"]
 ---> Running in fa04b480dd8f
 ---> 31f2e1f54283
Successfully built 31f2e1f54283
Successfully tagged ruby-example:v1.1.0-118-g92f37b200-dirty
Tags used in deployment:
 - ruby-example -> ruby-example:31f2e1f5428397d5d4aa1f2d2328ff859e01d970f304f2f9dbb9f4ba8d0c8968
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
 - deployment.apps/ruby configured
Watching for changes...
[ruby-75c5b89895-rd2lb ruby] - Gracefully stopping, waiting for requests to finish
[ruby-75c5b89895-rd2lb ruby] === puma shutdown: 2020-01-20 11:49:02 +0000 ===
[ruby-75c5b89895-rd2lb ruby] - Goodbye!
[ruby-5b74868996-ckg24 ruby] Puma starting in single mode...
[ruby-5b74868996-ckg24 ruby] * Version 4.3.1 (ruby 2.7.0-p0), codename: Mysterious Traveller
[ruby-5b74868996-ckg24 ruby] * Min threads: 0, max threads: 16
[ruby-5b74868996-ckg24 ruby] * Environment: development
[ruby-5b74868996-ckg24 ruby] * Listening on tcp://0.0.0.0:9292
[ruby-5b74868996-ckg24 ruby] Use Ctrl-C to stop

Change the app.rb then to return json string:

require 'oj'

class App
  def self.call(env)
    json = { "message" => "hello" }
    [ 200, {"Content-Type" => "text/html"}, [Oj.dump(json)]]
  end
end

After the app.rb file synchronization, the app would return:

$ curl -s $(minikube service ruby --url)
{"message":"hello"}

Features

File sync and automatic rebuild/redeploy are the core.

But, Skaffold has many essential features for Kubernetes-native development:

policy-based image tagging
resource port-forwarding
supports multiple profiles
supports Helm, Kustomize, Docker Compose manifests

Conclusion

Skaffold is simple and lightweight tool, easy to get up and running for local development! It automates deployments, but for comfortable development I like it to have any other features, like backward file sync.

Dockerize the multi-services application for local development

Dmitry Salahutdinov — Mon, 15 Apr 2019 11:40:13 +0000

As nowadays many complex web-applications run on production containerized, we keep developing them in 'old-school' way, installing the Postgresql, Redis, Ruby and other components on the local development machine.

It is getting harder to maintain the process of development, especially when the system becomes heterogeneous and expands into a large number of services, running with various versions of the dependent components. It becomes especially actual when the dependent component's versions are varied.

In this article, I am going to review the local development containerization with the example of the Amplifr, the project I work on. With the help of docker-compose and docker networks, it's easy and efficient.

As all the infrastructure are containerized, and are managed with Kubernetes on production, we will attend the setting up local development only, following the one principle - the convenience of the development process.

Benefits of the local containerization

No need to install all the components, such as database engines, language interpreters on the local machine. It keeps local machine clean.
natural support of the different environments, g.e. running the services with different versions of Ruby, of Postgresql on the local machine

Project overview

However Amplifr's backend is running on Rails, the project also has the complicated frontend, serving by the standalone Node.js server and the Logux web-socket server, and other helper-services, written on Node.js, Ruby, and Golang.

The following picture shows the simplified architecture of the project:

I am going to quickly review some components of the overall system.

Backend server

The backend is the classic Rails-application, doing all the business logic and performing many background jobs with Sidekiq.

Frontend server

The frontend is the only public HTTP entry-point for the overall application. It serves the frontend assets and proxies other requests to the Rails backend.
The backend is also integrated back with frontend server for sharing some data, like browsers.json file for proper rendering of HTML.

Logux server

The Logux is the server exposing the web-socket port, holding the bidirectional connection with the client's browsers. To perform the business logic, it has two ways of HTTP integration with the backend. It allows us to keep all the business logic in Rails-backend, and sending back the notifications from backend by hitting Logux with HTTP.

"Link shortener"

The link shortener is the specific web-service, written with Golang. It aims for shortening a link, expanding them and manages the overall statistics about link expansions.

"Preview" service

The preview is the public service, used from the client browsers to render the OpenGraph representation of any link. It has public http endpoint only.

Other components

Shortener - is the standalone service for shorting the url and keeping analytics data about link expansion. It is written with Golang. It has the external public endpoint to expand the shorted links, and internal endpoint to short the links while publication social content within backend's background jobs.

And some other internal services, such as telegram and facebook bot, which have the backend only integrations.

Components dependents

Most of the components are by itself the complex web-services, depending on underlying components, such as Postgres, Redis, and other services low-level system services.

Containarization

💡We will containerize each service separately with the Docker Compose. It is a tool for defining and running multi-container Docker applications, making it easy to start just by running up all the services together with only one command:

docker-compose up

💡To make the services to integrate we will use the docker networks, that allows any docker containers to communicate with each other. We will use only one internal docker network for all the components for simplicity. Being more accurate a reader will be able to set up the individual network for every service dependents and for every group of the connectivity.

Dockerize Ruby Backend

Here we have the standard stack: Postgres, Redis, Rails web-server and the Sidekiq background. For all of them, we'll define the services in docker-compose.yaml.

Here are the key points:

for the Postgres and Redis, we will define the persistent volumes to save the data between the runs
we are not going to copy the Ruby source code into the container, instead of this - we will mount the Rails application source-code to the /app folder
we also will define the persistent storage for the bundle and other stuff to increase the next time starts
we will define the amplifr_internal network and add the interacting containers to that network
the application should be ready to be configured with the environments variables, which we are going to set up in docker-compose file
we will define the base application service in YAML file and then will use the Anchors and aliases of the YAML syntax not to repeat yourself.

❗Keep in mind, that this configuration differs from the way of building docker image for production, where all the source code and all the dependency bundles are copied inside the docker image, to let it be all-sufficient and not to have external dependencies!

Here is the full gist with all the config, but let me pay attention to the main points:

Describe the base-service to inherit from it

services:
  app: &app
    build:
      context: .
      dockerfile: Dockerfile.dev
      args:
        PG_VERSION: '9.6'
    image: amplifr-dev:0.1.0
    volumes:
      - .:/app:cached
      - bundle:/bundle
    environment:
      # environment settings
      - BUNDLE_PATH=/bundle
      - BUNDLE_CONFIG=/app/.bundle/config
      - RAILS_ENV=${RAILS_ENV:-development}

      - DATABASE_URL=postgresql://postgres@postgres/amplifr_${RAILS_ENV}
      - REDIS_URL=redis://redis:6379/

      # service integrations
      - FRONTEND_URL=https://frontend-server:3001/
      - LOGUX_URL=http://logux-server:31338
    depends_on:
      - postgres
      - redis
    tmpfs:
      - /tmp

The base service's container will be build from the Dockerfile.dev with the arguments - the Postgres version. All other Ruby based images will inherit the base. Here is the service inheritance diagram:

We also define the mapping of the current folder to the container's /app directory and mount the docker volume for the bundles. It prevents every time dependencies installation.

We also define two groups of the environments variables:
1) system variables, such as BUNDLE_PATH, REDIS_URL and DATABASE_URL URLs.
2) dependent services internal url for integration:
FRONTEND_URL - is the internal endpoint of the frontend server to get the supported browserslist.
LOGUX_URL - is the internal Logux HTTP endpoint for sending action from Rails-app to Logux.

Describe the 'runner'

The runner service is for running maintaining commands, such as rake tasks, or generators in Rails-environment. It is console-oriented service, so we have to go set up stdin_open and tty options, which corresponds to the -i and --t option of docker and enable bash shell for the container start:

services:
  runner:
    <<: *backend
    stdin_open: true
    tty: true
    command: /bin/bash

We can use it in this way:

docker-compose run runner bundle exec rake db:create

# or run container and any command within the container
docker-compose run runner

Compose the server

Define the web-server. The critical point here is that we define additional docker network internal and adds the web-server to it giving the backend-server alias to the container host in this network. So the web container will be accessible with the backend-server network name.

services:
  server:
    <<: *app
    command: bundle exec thin start
    networks:
      default:
      internal:
        aliases:
          - backend-server
    ports:
      - '3000:3000'

networks:
  internal:

Compose the Sidekiq

Easy, it just runs the sidekiq and inherits the base service:

services:
  sidekiq:
    <<: *app
    command: sidekiq

Compose Redis and Postgres

  postgres:
    image: postgres:9.6
    volumes:
      - postgres:/var/lib/postgresql/data
    ports:
      - 5432

  redis:
    image: redis:3.2-alpine
    volumes:
      - redis:/data
    ports:
      - 6379

volumes:
  postgres:
  redis:

The main point here is that we mount the volumes for the container's paths, where the data is stored. It persists the data between runs.

Dockerfile

We would not dive deep into writing the Dockefile. You can find it here. Just notice, that it inherites from the standard ruby image, some required components such as Postgresql client and some other binaries to build the bundle.

Usage

The usage is quite easy:

docker-compose run runner ./bin/setup # runs the bin/setup in docker
docker-compose run runner bundle exec rake db:drop # runs rake task
docker-compose up server # get the web-server running
docker-compose up -d # runs all the services (web, sidekiq)
docker-compose up rails db # runs the postgres client

Docker Compose also allows to specify the service dependencies and get the dependent service up if it is needed for the running service, g.e. Sidekiq requires the Redis and Postgres services to work correctly, that is why we define them in the depends_on section of the service.

And here is the service dependency diagram, showing how the services run:

Summary

We've got the Rails-application running locally for the development. It works the same way as the local: persists the database, runs the rake task. Also the commands like rails db, rails c works well within a container.

The main advantage is that we can change the Postgres version or the Ruby version easily by changing one line, then rebuild image and try to run with the new environment.

Dockerize Node.js (frontend server)

The primary key points here are:

use the base official node docker images without any tuning
add the server service to the amplifr_internal network
define the BACKEND_URL environment variable to map to the internal docker path of the backend service.
mount the mode_modules volume for the Node.js modules install path

version: '3.4'

services:
  app: &app
    image: node:11
    working_dir: /app
    environment:
      - NODE_ENV=development
      - BACKEND_URL=http://backend-server:3000
    volumes:
      - .:/app:cached
      - node_modules:/app/node_modules

  runner:
    <<: *app
    command: /bin/bash
    stdin_open: true
    tty: true

  server:
    <<: *app
    command: bash -c "yarn cache clean && yarn install && yarn start"
    networks:
      default:
      amplifr_internal:
        aliases:
          - frontend-server
    ports:
      - "3001:3001"

networks:
  amplifr_internal:
    external: true

volumes:
  node_modules:

Usage

The frontend server is now easy to start, by running:

docker-compose up server

But it needs the backend to start first because frontend service refers to the internal network, which gets up while starting up the backend.

Dockerize the Logux server

In any simple case, Logux server has any databases dependencies and could be configured the same way as frontend. The only one difference, that Logux service has its environment variables, to set up the interaction with integrated services.

docker-compose up server # runs the server

Dockerizing Golang (link shortener web service)

The main idea is also the same:

use the set up docker image with Golang, mount the application source code there and run it with the go run interpreter.
share the service with docker networks for integrate with Ruby backend

Our web-service has the Postgres and Redis dependencies. Lets start describing from the Dockerfile, overall config sample can be found here:

FROM golang:1.11

ARG MIGRATE_VERSION=4.0.2

# install postgres client for local development
RUN apt-get update && apt-get install -y postgresql-client

# install dep tool to ensuring dependencies
RUN go get -u github.com/golang/dep/cmd/dep

# install migrate cli for running database migrations
ADD https://github.com/golang-migrate/migrate/releases/download/v${MIGRATE_VERSION}/migrate.linux-amd64.tar.gz /tmp
RUN tar -xzf /tmp/migrate.linux-amd64.tar.gz -C /usr/local/bin && mv /usr/local/bin/migrate.linux-amd64 /usr/local/bin/migrate

ENV APP ${GOPATH}/src/github.com/evilmartians/ampgs
WORKDIR ${APP}

Here are a couple of interesting details:

we install postgres-client for local development image. It simplifies the access to the database, whenever you need it: docker-compose run runner "psql $DATABASE_URL". The same we have at the Ruby backend dockerization
we install the dep tool to install and ensure all the dependencies: docker-compose run runner dep ensure
we install the migration tool to the image, to allow do database migrations right from the docker container: docker-compose run runner "migrate -source file://migrations/ -database ${DATABASE_URL} up"

‼️ The most of those tool we do not need for the production environment docker image, because it will contain only compiled binary.

We will use the same way of dockerizing to Golang service, as the Ruby service:

extract the base app service and the special runner service for run the maintenance tasks
add the Postgres and Redis dependencies with persistible data volumes

Here are the significant parts of the docker-compose.yml file:

services:
  # base service definition
  app: &app
    image: ampgs:0.3.1-development
    build:
      context: .
      dockerfile: docker/development/Dockerfile
    environment:
      REDIS_URL: redis://redis:6379/6
      DATABASE_URL: postgres://postgres:postgres@postgres:5432/ampgs
    volumes:
      - .:/go/src/github.com/evilmartians/ampgs
    depends_on:
      - redis
      - postgres

  runner:
    <<: *app

  web:
    <<: *app
    command: "go run ampgs.go"
    ports:
      - '8000:8000'
    networks:
      default:
      amplifr_internal:
        aliases:
          - ampgs-server

Wrap up

Docker-compose is the powerful tool to simplify the managing of the complex services.
Let me review the main principles for local development dockerization in context of using docker compose:

mount the source code as the folder to the container instead of rebuilding docker image with the copy of source code. It helps a lot of time for every local restart
use the docker networks to craft the communication between services. It helps to test all the services together, but keeps their environments separately.
services get to know of each other by providing the environments variables to the docker container with the docker-compose

That's it. Thanks for reading!

Monitoring Puma web server with Prometheus and Grafana

Dmitry Salahutdinov — Tue, 02 Apr 2019 10:48:27 +0000

Introduction

Localizing web application performance problems and response latency could be tricky in the projects with complex infrastructure.

And having monitoring for all the services is highly crucial.

Sometimes performance degradation might be induced one step ahead of the main application because of the lack of the web server capacity. As the most popular web server for running Ruby web applications is Puma, let me explain how to implement and tune up the simple monitoring for it.

Puma control application

Puma has a built-in control application for managing web-server and asking it's internal statistics. Control application actives by the following code, called from Puma configuration file puma/config.rb:



activate_control_app('tcp://127.0.01:9000', auth_token: 'top_secret')
# or without any params, just activate_control_app

For serving it, Puma runs the separate web-server instance with the specific rack backend application.

This backend returns the worker statistics data in JSON format:



# GET /stats?secret=top_secret
{
  "workers"=>2,
  "phase"=>0,
  "booted_workers"=>2,
  "old_workers"=>0,
  "worker_status"=> [
    {
      "pid"=>13,
      "index"=>0,
      "phase"=>0,
      "booted"=>true,
      "last_checkin"=>"2019-03-31T13:04:28Z",
      "last_status"=>{
        "backlog"=>0, "running"=>5, "pool_capacity"=>5, "max_threads"=>5
       }
    },
    {
      "pid"=>17,
      "index"=>1,
      "phase"=>0,
      "booted"=>true,
      "last_checkin"=>"2019-03-31T13:04:28Z",
      "last_status"=>{
        "backlog"=>0, "running"=>5, "pool_capacity"=>5, "max_threads"=>5
      }
    }
  ]
}

The exact answer schema depends on the Puma configuration: when it is in clustered mode (has more than one worker), the output describes each worker.
If Puma is in non-clustered, the result describes only the single worker with rapid output:



{
  "backlog"=>0,
  "running"=>5,
  "pool_capacity"=>4,
  "max_threads"=>5
}

Anyway, there are meaningful metrics for monitoring purposes, such as:

max_threads - preconfigured maximum number of worker threads
running - the number of running threads (spawned threads) for any Puma worker
pool_capacity - the number of requests that the server is capable of taking right now. More details are here.
backlog - the number of connections in that worker's "todo" set waiting for a worker thread

Using them, we can automate the monitoring system, which checks the values periodically any Puma cluster and show metrics in dynamic, such as shown below:

Decreasing pool_capacity means raising the load of the server. It is the starting point of rising the request processing time latency by capacity issues.

Yabeda framework

For the Ruby world, we have the extendable framework for collecting and exporting metrics, which is called Yabeda.

It provides a simple DSL for describing the metrics and fetching their values with a simple lambda function.

For now, Yabeda framework provides solutions for monitoring Rails and Sidekiq out of the box.

Prometheus

Monitoring considers periodically storing the metric values for future analysis. And one of the most popular and suitable solutions for that is Prometheus. As Prometheus implements the "HTTP pull model," it expects the monitorable subject to expose some endpoint with the metrics value in the specific format.

Yabeda framework allows exporting metrics with the help of Prometheus Exporter.

Yabeda for Puma

Now I am going to introduce one more new monitoring solution of the Yabeda family - puma monitoring plugin.

It just needs to load the yabeda-puma-plugin gem and to configure Puma web server with following lines in puma/config.rb file:



activate_control_app
plugin :yabeda

That's it. After the Puma web server start, the plugin will do all the job for collection the metrics.

Get things together

Here is the overall architecture of the Puma monitoring solution:

It gets all the metrics from Puma control application statistics and consolidates them to the Yabeda framework. Values could be exported by Prometheus rack-middleware, serving the /metrics path of the web application and providing metrics values in prometheus-friendly format. Here is the sample response of metrics endpoint for Puma, configured with two workers:



GET /metrics

puma_backlog{index="0"} 0
puma_backlog{index="1"} 0
puma_running{index="0"} 5
puma_running{index="1"} 5
puma_pool_capacity{index="0"} 1
puma_pool_capacity{index="1"} 5
puma_max_threads{index="0"} 5
puma_max_threads{index="1"} 5
puma_workers 2
puma_booted_workers 2
puma_old_workers 0

Visualization

Depending on your needs, the data could be visualized in many ways; here is the example of basic summarized metrics values:

This diagram shows the overall metrics values for all the Puma workers. Also, indicators could be displayed separately for all the workers, or all the Puma cluster instances.

"Application busy" metric

Looking at all the raw Puma metrics might be not visually comfortable to make some quick overview of the system in general. More suitable way if to calculate the composite metric, describing the overall workload of the web-server in percentage. Let call it "Application busy" or just busy-metric. Formula evaluates the percentage of overall workload:



(1 - pool_capacity / max_threads) * 100

It turns out to have the only chart instead of several:

The busy-metric looks to be more informative to overview the health of the system. It shows the actual workload of overall Puma cluster in a more friendly way. When busy-metrics sticks up, it means that the application is under high load, and it probably needs to tune up the Puma web server.

Busy-metric allows to determine the problem state easily, but for specific incident investigation, raw metrics might be more helpful and advisable.

Metrics playground

Yabeda framework supply the example project with all the monitoring infrastructure set up for monitoring the Sidekiq, Rails, and Puma. It is easy to set it up with docker-compose.

Wrapup

Setting up the monitoring infrastructure makes to build more stable and maintainable software, and sleep calmly at night.
Monitoring is made easy with Yabeda framework.

Check out the yabeda-puma-plugin for getting ready to monitor Puma!

Outdated browser detection with Browserslist

Dmitry Salahutdinov — Tue, 20 Nov 2018 19:43:22 +0000

The standard way to configure target browsers with Node.js is Browserslist. It is possible to add the following:

{
  "browserslist": [
    "last 2 version",
    "not dead"
  ]
}

to the package.json or the .browserslistrc config file:

# Browsers that we support
last 2 version
not dead

Those two similar examples mean that the target browsers are two last version and the browser is not dead.

This config is used by many front-end tools, such as Autoprefixer, Babel and many others.

But in this article I am going to write about the Browserslist Useragent frontend tool for finding if a given user agent string satisfies a Browserslist browsers:

Install the browserslist-useragent:

npm install browserslist-useragent

and you can determine by User-Agent string if you browser matches:

const { matchesUA } = require('browserslist-useragent')

matchesUA(userAgentString, options)

// with browserslist config inferred
matchesUA('Mozilla/5.0 (Windows NT 10.0; rv:54.0) Gecko/20100101 Firefox/54.0')
//returns boolean

// with explicit browserslist
matchesUA('Mozilla/5.0 (Windows NT 10.0; rv:54.0) Gecko/20100101 Firefox/54.0', { browsers: ['Firefox > 53']})
// returns true

Imaging we have the .browserslistrc config file like this:

last 2 versions
not IE 11
not ExplorerMobile 11
not last 1 OperaMini version
not OperaMobile 12.1
not dead

We can get the array of detailed browsers rules with the help of browserslist:

const browserslist = require('browserslist')
const fs = require('fs')

fs.writeFileSync('./browsers.json', JSON.stringify(browserslist()))

For the sample above it will produce the json file with:

[
  "and_chr 67",
  "and_ff 60",
  "and_qq 1.2",
  "and_uc 11.8",
  "android 67",
  "android 4.4.3-4.4.4",
  "baidu 7.12",
  "chrome 69",
  "chrome 68",
  "edge 17",
  "edge 16",
  "firefox 62",
  "firefox 61",
  "ios_saf 11.3-11.4",
  "ios_saf 11.0-11.2",
  "op_mob 46",
  "opera 55",
  "opera 54",
  "safari 11.1",
  "safari 11",
  "samsung 7.2",
  "samsung 6.2"
]

That is the way to determine the browsers matchings with Node.js.

Why do we need to check the browsers version on both sides: backend and frontend?
In the case of your modern javascript frontend would not be loaded on the old browser - we can still use the backend rendering to write some HTML notifying user about the issue:

❗This HTML block would work in any browser no matter how old it is.

And if your backend is written with Ruby - use can still use the port of the original tool to the Ruby - browserslist-useragent gem. It works the same way its Node.js version - recognises the family and the version from the User-Agent header string and matches it with the browserslist-rules produced by the Browserslists tool.

Single project

The usage is straightforward - it just needs you to generate the browsers.json file before.


class ApplicationController
  def supported_browser?
    @browsers ||= JSON.parse(Rails.root.join("browsers.json").read)
    matcher = BrowserslistUseragent::Match.new(@browsers, request.user_agent)
    matcher.browser? && matcher.version?(allow_higher: true)
  end
  helper_method :supported_browser?
end

Then add this code to you Rails-application layout template:

- if !supported_browser?
  .div 
div( style: "position: fixed; bottom: 0; right: 0; padding: 8px 10px; background: #e9502f; color: white; width: 100%; z-index: 10; text-align: center;" )
    .div
      = t('unsupported_browser')

❗This old-fashioned style is deliberately chosen: 'style'-attributes will work mostly everywhere!

Here it is. It will work well for the Rails projects where all the frontend and backend live together as one solid project.

Separated frontend and backend projects

If you have separated projects for Ruby backend and Node.js frontend, you will prefer to get browsers.json over HTTP. You will need to do the following:

serve the /browsers.json path to render the browserslist output by putting it to the public folder:

fs.writeFileSync(
  path.join(__dirname, 'public', 'browsers.json'),
  JSON.stringify(browserslist(undefined, { path: path.join(__dirname, '..') }))
)

get in over HTTP in the ruby-backend code:

browsers = JSON.parse(Faraday.get('http://frontend-domain.local/browsers.json').body)
matcher = BrowserslistUseragent::Match.new(browsers, request.user_agent)
matcher.browser? && matcher.version?(allow_higher: true)

Or use the faraday-http-cache to cache the results of the http request. It will force to make one request per the Rails application instance only:

# caches http response locally with etag
http_client = Faraday.new do |builder|
  builder.use Faraday::HttpCache, store: Rails.cache
  builder.adapter Faraday.default_adapter
end

browsers = JSON.parse(
  http_client.get('http://frontend-domain.local/browsers.json').body
)
...

That's it. This solution will use one browserslist.rc config in the frontend repository, which will automatically be shared over the backend.

More details abort the browserslist_useragent gem you will find here.

Thanks for reading!

Purify code with data integrity. Part 1: defaults and not-nulls

Dmitry Salahutdinov — Mon, 19 Nov 2018 18:06:47 +0000

Working with the data is the crucial point of any application. ActiveRecord makes this process easier for Rails developer. It allows managing the data without getting deep into the underlying database. Sometimes it causes some code issues we would not have if pay more attention to the database structuring and maintaining data integrity.

Data integrity?

Data integrity is the process of maintaining the accuracy and consistency of the data.

In ActiveRecord context I mean, that data integrity - is the set of the principles for keeping an underlying database in accuracy state by paying more attention to its schema design.

In this article, I am going to review some well-known data integrity techniques which are very helpful in improving the codebase but sometimes are undervalued for some reasons. We will see how easy they are to integrate and how they can help to write more clear code.

Default values

Have you seen the code like this?

# Example 1
order = Order.new(state: 'new')

# Example 2
class Order < ApplicationRecord
  validates :state, presence: true

  def before_validation
    self.state ||= 'new'
  end
end

There is the same problem in both examples: the lack of default value preset. It causes for writing initialization "by hand" in a non-centralised way using one of the following options:

1) 👎to set the value "in place" (example one). This solution is fast and dirty. The main disadvantage is implicitness of the "correct" default value: someone who is not familiar with the project would not know the proposal default for setting up in the future or expect the value should already present. It may cause the errors.

The second point is that such "code snippets" spread all over the codebase and make it hard to maintain because of duplication and decentralization.

2) 👎to set the default values in the "pre-save" model callback
(example two from the top).But here we also have implicitness. Where should I find the correct initial value before_save, before_commit, before_validation? The answer depends on the imagination of the author 😊.

The second problem is that the default value is not initialized in the new model (example two), and even we set the default values before saving, we still have it uninitialized in the new model object:

order = Order.new
order.state
=> nil
order.save
order.state
=> 'new'

👍The best option is to preset the default value in the database schema and let ActiveRecord work well by design:

create_table :orders do |t|
  ...
  t.integer :state, default:'new'
end

# default value work by-default :)
Order.new.state
=> 'new'

It allows to have a new object with the column set by default. ActiveRecord will read the metadata of your table and preset the default values for any new model of the User type.

Setting up the database

The simple case is to set the default values while creating the new table. It is easy and non-blocking operation:

create_table :order do |t|
  ...
  t.integer :state, default: 'new'
end

Otherwise adding a column with default to the existing table is more complicated, because it will enforce setting the default value for all the rows. It will lock read/write operation for a while in the high-loaded databases.

If you are lucky to be on the edge and run you application on Postgres 11 - things become much simpler. PostgreSQL 11 does not touch the existing row, but updates them 'lazily'.

If you are not on Postgres 11, the safe way to set up default value constraint consists of two simple steps:
1) creating the nullable column without setting default value
2) changing the default value of the column

  add_column :orders, :state, :text
  change_column_default :orders, :state, 'new'

After that all new rows will have the default values and active record will use those metadata to automatically preset the default. All the existing rows in data base will still have the NULL value and it is needed to fill them "by hands" to start to prettify the ruby code. Then you can rely on the presence of the column for every row.

❗The main point is to maintain compatibility: change the database presets first and ship it. Then cut off the default value initialization out of Ruby-code, when you don't have NULL-values in the column anymore.

Not-Null Constraints

Have you ever faced code like this?

class ImageUploadService
  def call
    ...
    image.upload_tries_count ||= 0
    image.upload_tries_count += 1
  end
end

The issue is similar to the lack of default value. When a column is nullable - web couldn't be sure that there is no nil value. It enforces to write nil-checking logic every time we have to use the column.

If we guarantee that column does not have the nil value, the code will become much simpler:

image.upload_tries_count += 1

And here is the solution:

A not-null constraint specifies merely that a column must not assume the null value.

Create a new table and with not-null column

Set the default options and the column will have the value set by default:

create_table :user do |t|
  ...
  t.integer :points_count, null: false, default: 0
end

Adding not-null column to existing table

This article describes in detail the safe way to add the new column with default, but here are the summary steps:

Fill all the null-values in the database with some value
Set column not-null (by migration)
Cut off the "in-place" model initialization code

❗The better way to fill values of the overloaded table is to update with batches (every batch in its transaction) and to stretch in time.

The standard way to automate things in Ruby world is Rake-tasks, and here is the sample of rake-task to update users table users with batches:

desc 'Fill user points count'
namespace :migrations do
  task fill_user_points_count: :environment do
    total_updated = 0
    loop do
      updated_rows = User
        .where(points_count: nil).limit(10_000)
        .update_all(points_count: 0).to_i
      break if updated_rows.zero?

      total_updated += updated_rows
      puts "updated #{total_updated} users"
    end
  end
end

I recommend using rake-task instead of updating columns in migration, because of it
gives more control to run it on production (you can interrupt process at any time if something goes wrong). And here is some tricky migration to automate setting the default value for staging environment and the other developer machine:

class FillOrderSourceAndSetNotNull < ActiveRecord::Migration[5.0]
  disable_ddl_transaction!

  def up
    if Rails.env.production?
      puts "Run bundle exec migrations:fill_user_points_count in production"
    else
      Rake::Task['migrations:fill_user_points_count'].invoke
    end
  end
end

This migration disables transactional mode for the migration and updates users with batches of 10 000 rows. It allows to not to have the large table lock if the table updates in one transaction.

After this step - we can easily set table not-null:

change_column_null :users, :points_count, false

and deploy changes to production.
The last step is to cut out nil-checking code like this:

user.points_count ||= 0

Wrap up

Following this two simple rules: 'set the default value' and 'set column not null' are very useful to write more clear code and avoid many of the problems.

Thanks for reading! See you at the next topic.

Additional links:

Concerns testing with anonymous controller

Dmitry Salahutdinov — Wed, 10 Oct 2018 08:41:48 +0000

Once you have a controller concern like this:

module BillingErrorRescuer
  extend ActiveSupport::Concern

  included do
    rescue_from Billing::LimitExceeded, with: :rescue_billing_limits

    def rescue_billing_limits
      render status: 402, json: Billing::PlanPresenter.new(@account).to_json
    end
  end
end

The idea is to test is separately in anonymous controller:

require 'spec_helper'

describe BillingErrorRescuer, type: :controller do
  controller do
    include BillingErrorRescuer

    def action
      @account = FactoryBot.create :account
      raise Billing::LimitExceeded
    end
  end

  before do
    routes.draw { get :action, to: "anonymous#action" }
    get :action
  end

  let(:account) { create :account, state: :trial }

  it 'renders json with billing data' do
    expect(response.body).to match_json_schema('controllers/billing_error_rescuer')
  end
end

What does this test do?

Enables testing controller features in rspec
Defines anonymous controller to include concern extension
Draw the route for the action
Calls the action with the default controller pipeline

Thats it!
Thanks rspec-rails gem, that we have almost everything out of the box!

Stressless pull-request watching with Google Mail

Dmitry Salahutdinov — Tue, 02 Oct 2018 17:12:16 +0000

Let's share the experience of watching repositories, doing review and reacting to the messages.

I am going to review some common tips of organising pull-request watching with Google Mail, that work for me.

Why not Slack

Slack is integrated merely with GitHub but has no grouping, no priorities. Messages go in a timeline feed, so comments of different pull-requests alternate and intermix.

Google Mail benefits

Google Mail is very smart and helpful in organising email processing and have many features to be productive. We are going review some of them.

Messages chains

Google Mail groups all the incoming messages into the chains and puts all the GitHub’s comments of one pull-request in one. No matter how many unread messages of one pull-request you have - they all appear as one unread chain (with the number of incomes).

❗ Chaining breaks after renaming pull-request, and all the new comments with getting in the new group. That is why do not recommend to name pull-request correctly for the first time.

Labels

Labels allow organising all the emails in groups. They work like folders, but you can add multiple labels to a message. We will use them to specify the message types, e.g. “For review”, “My team”, “OSS” and so on.

❗ Labels are flexible, and you could reconfigure them at any time when you will need to reorganise your flow.

Filters

Google Mail uses customisable rules to preprocess incoming messages. This is the excellent way to classify emails and apply the source-specific label to them.

Prioritisation

Simple "prioritisation" could be maid manual by arranging labels decreasing the urgency of the messages you have to react.

Snoozing

Snoozing is the new feature. It is handy in the context of pull-requests watching because it allows postponing the incoming mail immediately when you are not able to react at the moment.

Now we are going to tune up those features to improve the organisation of watching and reviewing pull-requests process.

Google Mail features in action

1. Labels

First of all, we have to think about repositories you are watching and the urgency of incoming data. Here is my sample. You could use it to start and make it more proper to your case:

Review - holds the pull-requests from where someone is asking my review with the GitHub's "Reviewers" tool.

Backend - is used to put messages from the backend team. Here might be some pull-requests which are in work now (WIP) and probably no review has been requested yet. As I am in the back-end team, it is essential to keep track of events of my team.

Frontend labels used to mark frontend team's pull-requests. It is less important to watch those pull-requests but lets me observe what is happening in the frontend part of Amplifr.

Team: here are all the pull-request of the all Amplifr working project repositories. Most of the are auxiliary and don't change too often.

Used gems. At the backend projects, written with Ruby on Rails, we use many Ruby gems. And it is handy to follow all the updates, that is why I watch some repositories like twitter gem.
But this updates are not urgent and could be viewed at the free time.

OSS - this label is for following pull-requests of the open source projects I contribute to or like to track updates.

GITHUB - I use this label to mark all the messages from GitHub to overview the total count.

Filters

These filters work for me:

Matches: from:(notifications@github.com)
Do this: Apply label "GITHUB", Never send it to Spam

Matches: from:(noreply@github.com) subject:(Github Subscribed)
Do this: Mark as read, Delete it

Matches: from:(notifications@github.com) to:(review_requested@noreply.github.com)
Do this: Apply label "1) Review", Mark it as important

Matches: to:(evilmartians/amplifr <amplifr@noreply.github.com>)
Do this: Apply label "2) Backend", Mark it as important

Matches: to:(evilmartians/amplifr-front <amplifr@noreply.github.com>)
Do this: Apply label "3) Frontend"

Matches: (to:(gazay/gon) OR to:(sferik/twitter) OR to:(philnash/bitly) OR :to:(wilddima/logux_rails)
Do this: Apply label "5) Used gems"

❗ You could download the base filters from my Gist, import them into Google Mail, and start tuning them for your flow.

After that, we can see all the messages grouped by the urgency of them. A quick view of the label's table let you see how many tasks are pending.

Google Mail Checker

"GMail Checker" is the excellent browser plugin for checking the mail and for getting notified about new messages.

The main features, I use, are:

1) "Do not disturb" for the full-screen mode.

This option is beneficial not to lose concentration when you are coding in the VIM or browsing something in full screen.

2) Notify for the specified labels email:

This feature lets me react fast for the most crucial messages.

The mail checker also has many features, which would be very useful to improve usage of the mail.

Mental setup

When automation works well, you only need to view comments starting from to most urgent and react to them. And here is the most critical part for human factor come out.

Imaging the cause, you open the new email, read the comments, go to the GitHub, and then you look aside and switch to other activity, forgetting about this message. In this case, you miss the sight because you have the letter read in Google Mail but without no reaction.

❗The primary mental setup - is to review incoming mail intently and always make a reaction:

directly, with approving pull-request, doing the review or commenting
implicitly, with snoozing the message (for the more suitable time)
or with marking the email "unread" to leave it for the next session, you will work on the email.

Wrap up

Spending one hour for organising incoming emails let to save more time in the future, It:

makes email browsing smarter by using prioritisation and labels
shorten the reaction time for the most urgent messages
gives the flexibility - all the settings could be changed at any time for any situation to perform maximum

Thanks for reading.

How to migrate monolith to the scary new version of Rails

Dmitry Salahutdinov — Tue, 25 Sep 2018 19:14:08 +0000

Migration to the new version of Rails always looks scary. Thinking about the upgrade raises many questions, especially if your application is an older monolith. The most critical points are:

upgrading without blocking feature development
affecting as fewer users as possible
making sure all logic works correctly on the new version of Rails

In this article, we will review some common techniques and tips which let Amplifr’s team have the painless incremental upgrade to the latest version of Rails.

❗The main tip - is to upgrade incrementally only to the latest patch version of Rails at once, because most of the bugs have already been solved. And this choice minimises breaking changes you need to deal with each upgrade.

We had done three upgrades steps: 4.2.0 => 5.0.7 , 5.0.7 => 5.1.6, 5.1.6 => 5.2.0.

Here are the standard steps of one Rails version upgrade cycle and delivering it to production:

deprecations: clean up all deprecations, which block the next Rails version update
dual boot: make application runnable with current and next version of Rails
green tests: make tests pass for both Rails (matrix build)
deploy → monitor → (revert) → fix: switch to the new version of Rails, monitor application for the new failures and bugs, revert if bugs are critical and you need time to fix them
keep doing point number four until everything is ok

Repeat until you are on the top Rails version 😁

Deprecations

Deprecations say that something can break when a third party dependency gets updated. As we are going to upgrade Rails - we have to fix the blockers for the next version of Rails.

The main problem is to separate the target version blockers from fixing them first. Hiding the future version deprecations by using this dangerous and ugly-hack worked for us:

    if Rails::VERSION::MAJOR == 5
      # temporary mute test params deprecations
      ActiveSupport::Deprecation.behavior = lambda do |msg, stack|
        unless msg =~ /Using positional arguments in functional tests has been deprecated/
          ActiveSupport::Deprecation::DEFAULT_BEHAVIORS[:stderr].call(msg, stack)
        end
      end
    end

❗It is better to fix all deprecations at once if it’s possible, it will save a lot of time in the future

Dual boot

The idea is to support both versions of Rails at the same time. Keeping application runnable on current Rails version guarantees that you will be able to run the application on it if something goes wrong. Making application runnable on the next Rails version will allow you to have continuous upgrade.

Make your application able to run with both versions of Rails depending on environment variable RAILS_NEXT:
This command should boot current Rails version

    bundle exec rails c

And this will boot application with the new version of Rails:

    RAILS_NEXT=1 bundle exec rails c

The way to do it - is to hack bundler a little bit by putting this code at the beginning of Gemfile:

module Bundler::SharedHelpers
  def default_lockfile=(path)
    @default_lockfile = path
  end

  def default_lockfile
    @default_lockfile ||= Pathname.new("#{default_gemfile}.lock")
  end
end

module ::Kernel
  def rails_next?
    ENV["RAILS_NEXT"] == '1'
  end
end

if rails_next?
  Bundler::SharedHelpers.default_lockfile =
    Pathname.new("#{Bundler::SharedHelpers.default_gemfile}_next.lock")

  class Bundler::Dsl
    unless method_defined?(:to_definition_unpatched)
      alias_method :to_definition_unpatched, 
                   :to_definition
    end

    def to_definition(_bad_lockfile, unlock)
      to_definition_unpatched(Bundler::SharedHelpers.default_lockfile, unlock)
    end
  end
end

gem 'rails' #relax dependency

This code does a couple of things:

Defines the rails_next? method in Kernel class, so this method is available to use everywhere: in Gemfile, in your ruby application’s code and so on
Defines singleton-method to hold the current bundler’s lock-file name
Reset the lock-file name to Gemfile_next.lock if rails_next? is activated

❗Note, that we use one Gemfile and two lock-files for every version of Rails. It is much easier to maintain.

Dependencies

First of all, relax Rails dependencies in Gemfile to enable higher versions of Rails:
If you have rails version locked like this:

gem 'rails', '~> 4.2.3'

Unlock it.

gem 'rails'

It allows Gemfile to be compatible with all versions of Rails. Current and next rails version will be locked in the corresponding lock -files.

Then copy Gemfile.lock to Gemfile_next.lock and try to update rails on rails_next:

    $ cp Gemfile.lock Gemfile_next.lock
    # run bundle for fixing dependencies matedata in original Gemfile
    $ bundle
    # run bundle on rails_next to update rails
    $ RAILS_NEXT=1 bundle update rails

Here you will get bundler working hard to resolve dependencies and probably fail in the middle. Failure means that some of the dependencies are not allowed to work with the newer version of Rails.
Here are some scenarios:

Gem has current version locked and not supported. And it has the newer version to support the newer version of Rails. We have to update gem if only the rails_next? works:

    if rails_next?
      gem 'devise'
    else
      gem 'devise', git: 'https://github.com/plataformatec/devise', branch: '3-stable'
    end

Gem has Rails version locked from above: rails < 5.0 but probably works with newer version. You need to unlock dependencies and give it a try. Try to clone this gem from GitHub locally, relax dependency of Rails in gemspec file and plug gem locally. If it works - great, move to the next blocker, if not - try other solutions.
Gem has Rails locked, because of incompatibility and does not work with newer version. You need to unlock dependencies, understand how gem works and where is the problem, then fix all the conflicts and enable tests for the new version of rails in this gem.

Dealing with lasts cases - you will have to update gem source code. First of all, you will fix the problem in your fork. Despite this way looks very simple, fast and easy - there are many reasons for avoiding keeping it in forks:

Authors (or a group of authors) maintain their gems in a centralised way. Nobody is interested in supporting your fork 😢 Your fork is your problem!
Forking let you fix the problem fast, without design work and tests, but it is just postponement of the problem. You will pay back when upgrading the fork from the remote source in future
Fix a problem in a centralised way (by sending pull-request) helps other developers to deal with the same problem. Feel free to contribute by submitting the pull-request!

❗The better way to fix gem’s source code - is to send pull-request to the gem’s repository.

Unfortunately, some gems do not have anybody to maintain because of many reasons. And this is the signal for you to inspect your dependencies and get rid of unmaintainable ones in favour of more popular.

❗Note that you have to update all gems twice to support two version of Rails, and commit both Gemfile.lock and Gemfile_next.lock to keep them similar.

    bundle update some-gem
    RAILS_NEXT=1 bundle update some-gem

Runnable app

When all dependencies are okay, and your bundler successfully resolved for the rails_next - you are ready to proceed to the next step - check if the application is runnable.

You could try to run Rails console or local web-server, but it would be better to start from running the tests:

    RAILS_NEXT=1 bundle exec rspec

With a high probability tests would not start or even fail. The main problem here is the incompatibilities of your code or some gems’ code with new Rails version. So only one tip: go deeper and fix it.

❗ One trick we used - is to write Rails version dependent code using rails_next? method temporarily:

    project.run_callbacks(:commit) if rails_next?

It helps to write the version-specific code:

    if rails_next?
      redirect_to user_omniauth_authorize_path(:instagram)
    else
      redirect_to user_instagram_omniauth_authorize_path
    end

Green tests

Add the extra build to you CI for running tests with the next Rails version. If you are running on TravisCI, it might look like this:

    matrix:
      include:
        - env: RAILS_NEXT=0
        - env: RAILS_NEXT=1

If your next Rails version tests not pass yet - you can mark them optional like this:

    matrix:
      include:
        - env: RAILS_NEXT=0
        - env: RAILS_NEXT=1
      allow_failures:
        - env: RAILS_NEXT=1

This approach will let you merge matrix-build to the main branch without affecting feature-development, and you’ll be able to work on tests in parallel.

Finally, you need to make your tests green for both versions.

❗Getting upgrade without having tests - is not a very good idea 🙂, because it’s impossible to ensure crucial application parts work on the next version.

Deploy & Monitor

It time to rollout!

If you have staging - deploy there first and check the main functionality by hands. It minimises risks and may give a chance to catch some bugs.

Running in production is more effective. It makes the real users test application within the reals cases. The main idea is to deploy the new Rails version by “a little bit” and keeping in mind revert strategy, that would help you not to affect users too much or too long.

Depending on your infrastructure you may prefer to use one of the rollout strategies, or combine them:

serving specific endpoints with new Rails version application instance, starting from low-loaded to high-loaded, or starting from non-critical.
serving the percentage of overall load, and then run to increase the rate.
rollout the application at the low-loaded time (if you have those), e.g. at night-time
rollout 100% and keep calm 🙂

We decided to roll out the all Amplifr with new Rails version, because it is the most progressive way, and rolling back to the previous Rails version took a couple of minutes for us.

Once you’ve rolled out the application (or it’s part) and let real users face it - it is essential to keep monitor errors.
There are two general ways to get to know about problems:

automatic monitor with some tools like Honeybadger, Rollbar, Sentry, NewRelic. Having it - is a must! They are useful to be notified about causes of issues and bugs online with Slack, email or other ways
real users are critically helpful for small teams like Amplifr. No matter how much you’ve tested the application and how high test-coverage you have – it’s likely that you missed few bugs, especially in business logic. Always be kind and patient with your users, and they will behave the same way when you need their help.🙂

If something goes wrong - revert by changing the value of RAILS_NEXT environment variable. The exact way depends on the infrastructure you have:

    export RAILS_NEXT=0

❗Rolling back does not mean failure, so do not worry. It gives you the time to fix newly found errors while the application runs stable on the previous version of Rails. Just keep it up!

We have several rollbacks for fixing the critical bugs in Amplifr.

❗After the successful upgrade, let the application to work on the new version of Rails for a long-term to minimise risks and fix all the bugs.

Cleaning up

After the successful upgrade, we have clean up the source code and cut off the support of the previous Rails.

Make the application run with the new version of Rails by default. You have to copy next lock-file content and commit it: cat Gemfile_next.lock > Gemfile.lock git commit ...
Deploy the application and switch RAILS_NEXT 1 → 0
Cut off the Gemfile_next.lock: git rm Gemfile_next.lock git commit
Remove all the version dependent code; you can easily find it: grep -ir "rails_next\?" ./
Remove Bunder's hacks from the Gemfile

We haven’t removed the dual boot code and kept running on the non-updated lock-file for a couple of days. 🤦‍♂️ Be careful with it. 🙂

❗If you are going to keep upgrading - run from the begging with the next version.

Stay on edge

If you are on the last stable version, you might prefer to stop on it. Another option is to keep upgrading with the master branch as next version of Rails.

    gem 'rails', git: 'git@github.com/rails/rails'

This way is not single-valued but probably is the progressive one. Most of the gems do not support the newer version, and you will have to contribute there. It could be dangerous and make you catch bugs of the unstable features in Rails code.
But from the other hand - you will keep on the wave with the new version of Rails could make your upgrade painless and comfortable in the future, and be helpful by contributing to Ruby’s gems source codes.

Conclusion

Although upgrade Rails version might look like a tremendous amount of work, it is essential to be on the latest version, because of this reasons:

it is more stable, fast
it has new features
it lets you use the latest versions of gem’s

Doing upgrades iteratively, step by step, and keeping in mind revert strategy can make your upgrade less painful.