Forem: Rafi

Ruby on Rails pattern: Service Objects

Rafi — Mon, 07 Dec 2020 21:11:24 +0000

Ruby on Rails is an excellent framework for developing large scale applications. It is battle tested for more than a decade and still continues to grow. Many companies have built large scale applications with this like Github or Shopify. However, since it is a monolithic framework, it can grow rapidly and needs proper organization to keep things sane. There are a lot of talks about fat controllers and skinny models. We won't go in that direction. Instead, we will talk about the solution today.

Service Objects

One such pattern to solve fat controllers in Rails is service objects. The name doesn't do justice to what it really is. It is basically plain ruby objects that can offload much of the workload from the controllers. So let's get started and see how we can work with it

Preparation

We will build a simple API only Ruby on Rails application for this guide. Let's generate our app with

rails new rails-service \
  --skip-action-text \
  --skip-active-storage \
  --skip-javascript \
  --skip-spring -T \
  --skip-turbolinks \
  --skip-sprockets \
  --skip-test \
  --api

We see we skip some of the goodies Rails comes bundled with since we do not need it for this demo. We would not go to explain the migrations or models here, you can check them out in the repo. Instead, let's see the architecture and jump right into a fat controller.

Architecture

Imagine, we are building a SaaS service where a user can register and subscribe to a certain product. So, we would be handling multiple procedures when the user signs up. So let's see what are they

Create user entry with user's data
Retrieve the product, he signed up for
Create a subscription with that product and add an expiry date when the subscription will expire
Assign a dedicated support person for the user
We would also update our metrics table with the new revenue that we got from the user. This is necessary for time series aggregation for the investors
We would also send him a welcome email about his new subscription

So, a lot happens when a user signs up. Let's see how it looks like in a traditional Rails controller

# app/controllers/v1/user_controller.rb

module V1
  class UserController < ApplicationController
    def create
      begin
        ActiveRecord::Base.transaction do 
          # Create User
          prev_user = User.find_by email: params[:email]
          raise StandardError, 'User already exists' if prev_user.present?

          user = User.create(name: params[:name], email: params[:email], pass: params[:pass])

          # Create Subscription
          product = Product.find_by name: params[:product_name]
          raise StandardError, "Product doesn't exist" unless product.present?

          sub = Subscription.create(product_id: product[:id], user_id: user[:id], expires_at: Time.now + 30.day)

          # Assign support person
          support = Support.find_by name: 'Jessica'
          raise StandardError, "Couldn't assign a support person"  unless support.present?

          user.support_id = support[:id]
          user.save

          # Update Metrics
          Metric.create(user_count: 1, revenue: product.price)

          # Send welcome email
          UserMailer.with(user: user).welcome_email.deliver_later
          render json: { user: user, subscription: sub }
        end
      rescue StandardError => e
        render json: { error: e }
      end
    end
  end
end

Here, we are creating a user, then creating a subscription for that user by retrieving the product, assigning a dedicated support person, updating metrics, and sending welcome email. We are definitely doing a lot here. This also violates single responsibility principle of SOLID design pattern. What if we need the same behavior of updating metrics in another controller? We would be duplicating that block of code. Let's fix this

Service Objects

Service objects are Plain Old Ruby Objects (PORO) that are designed to execute one single action in your domain logic and do it well. It follows the single responsibility principle strongly. We should divide all the tasks in the controller into these service objects.

To host these objects inside our Rails application, we would create a services directory. Let's create our first service.

# app/services/create_user_service.rb

class CreateUserService
  attr_reader :name, :email, :pass

  def initialize(name, email, pass)
    @name = name
    @email = email
    @pass = pass
  end

  def call
    prev_user = User.find_by email: @email
    raise StandardError, 'User already exists' if prev_user.present?

    User.create(name: @name, email: @email, pass: @pass)
  end
end

This is a simple Ruby class where we send it a few parameters when initializing and then look for the previous user, if not found we create a new user. That's it. Now let's call it from controller. We would use out v2 namespace controller for this

# app/controllers/v2/user_controller.rb

module V2
  class UserController < ApplicationController
    def create
      begin
        ActiveRecord::Base.transaction do
          # Create User
          user = CreateUserService.new(params[:name], params[:email], params[:pass]).call
          render json: { user: user }
        end
      rescue StandardError => e
        render json: { error: e }
      end
    end
  end
end

Nice and simple. All the logic of user creation is abstracted away from the controller and we get a nice clean service that we call to handle it. The signature of invoking the service still looks a bit non-intuitive. Let's improve that.

# app/services/application_service.rb

class ApplicationService
  def self.call(*args, &block)
    new(*args, &block).call
  end
end

We are creating a base class ApplicationService that our service objects will inherit from. This class method creates a new instance of the class with the arguments that are passed into it and calls the call method on the instance. Let's see the usage to clear the confusion.

Let's inherit from the class first.

# app/services/create_user_service.rb

class CreateUserService < ApplicationService

...

Nothing else would change from CreateUserService class. Let's see how we can invoke CreateUserService service now.

# app/controllers/v2/user_controller.rb

user = CreateUserService.call(params[:name], params[:email], params[:pass])

We have shortened our service call since we don't need to call new on it anymore. Looks much cleaner. Now that we have an understanding of how service objects can help us organize our code, let's quickly refactor our controller into other services.

# app/services/get_product_service.rb

class GetProductService < ApplicationService
  attr_reader :name

  def initialize(name)
    @name = name
  end

  def call
    product = Product.find_by name: @name
    raise StandardError, "Product doesn't exist" unless product.present?

    product
  end
end


# app/services/create_subscription_service.rb

class CreateSubscriptionService < ApplicationService
  attr_reader :product_id, :user_id

  def initialize(product_id, user_id)
    @product_id = product_id
    @user_id = user_id
  end

  def call
    Subscription.create(product_id: @product_id, user_id: @user_id, expires_at: Time.now + 30.day)
  end
end


# app/services/assign_support_service.rb

class AssignSupportService < ApplicationService
  attr_reader :user, :support_name

  def initialize(user, support_name)
    @user = user
    @support_name = support_name
  end

  def call
    support = Support.find_by name: @support_name
    raise StandardError, "Couldn't assign a support person" unless support.present?

    @user.support_id = support[:id]
    @user.save
  end
end


# app/services/update_metrics_service.rb

class UpdateMetricsService < ApplicationService
  attr_reader :product

  def initialize(product)
    @product = product
  end

  def call
    Metric.create(user_count: 1, revenue: @product.price)
  end
end


# app/services/welcome_email_service.rb

class WelcomeEmailService < ApplicationService
  attr_reader :user

  def initialize(user)
    @user = user
  end

  def call
    UserMailer.with(user: @user).welcome_email.deliver_later
  end
end

We have created five more services, each handling one unit of work. Let's refactor our controller to reflect the changes

# app/controllers/v2/user_controller.rb

module V2
  class UserController < ApplicationController
    def create
      begin
        ActiveRecord::Base.transaction do
          # Create User
          user = CreateUserService.call(params[:name], params[:email], params[:pass])
          # Get Product
          product = GetProductService.call(params[:product_name])
          # Create Subscription
          sub = CreateSubscriptionService.call(product[:id], user[:id])
          # Assign support person
          AssignSupportService.call(user, 'Jessica')
          # Update Metrics
          UpdateMetricsService.call(product)
          # Send welcome email
          WelcomeEmailService.call(user)
          render json: { user: user, subscription: sub }
        end
      rescue StandardError => e
        render json: { error: e }
      end
    end
  end
end

This looks much cleaner. We have refactored out the fat controller into a more manageable one.

Some Tips

Like every other pattern, we need to also keep our service objects in check and follow some practices so that even if these services grow in numbers, we can continue working at the same pace

Group Similar Service Objects

When we would have a lot more services, we need to group these objects for sanity. For example, we can group everything related to user handling in user group and everything related to subscription management in subscription group.

services
├── application_service.rb
└── subscription_manager
      ├── create_subscription_service.rb
      └── expire_subscription_service.rb

Follow One Naming Convention

There are many ways you could name your services, like UserCreator, SubscriptionManager or plain commands like CreateUser, ManageSubscriotion as such. I like calling appending service at the tail so that I know for sure it is a service and avoid naming collision with let's say migration classes.

Single Responsibility Principle

Make sure to follow this with service objects, it should always be doing one thing and one thing only. When it does multiple things, better to break them down into other services

Conclusion

Service objects are an essential pattern to keep our business logic outside controllers. This would reduce a lot of complexity from the codebase. There are some gems that take inspiration from this approach. One of these gems are Interactor gem. We will explore how this would help us in a later article. Til then, stay tuned

Project Link: https://github.com/Joker666/rails-service-demo

Track Errors in Ruby on Rails Application with Sentry

Rafi — Thu, 26 Nov 2020 19:51:19 +0000

Project Link: https://github.com/Joker666/sentry-rails

Error tracking is one of the most important parts of modern software development. When an app goes to production, it goes out from your development environment to users or customers for who various cases can arise that might not be handled in the application. And enter crashes, errors, unusual behaviors. Unless we track these, we do not have full visibility into the performance of the software.

Rails is great at recovering from crashes automatically. But the error remains buried in logs. There are a lot of error tracking software and SaaS businesses that can integrate with Rails. Most popular ones are Rollbar, Sentry, Bugsnag etc. All of them have free tiers that handle a generous amount of events and retain data for 1 month to 3 months. For heavy workload, it's better to opt for paid tiers. But Sentry is amazing because it has its core software open sourced that anyone can deploy and scale. Plus, it has a very nice UI. We are going to choose Sentry for this tutorial.

Sentry

You can opt for the free tier through their website and continue with the next part of the tutorial, but we are open source enthusiastic here and we would do a local deployment of Sentry in docker-compose. Fortunately, Sentry provided a one-time setup with a shell script that get the software up and running in a matter of minutes.

The instruction for this is available here. Prerequisites

Docker

Let's clone the repo and run shell script



$ git clone https://github.com/getsentry/onpremise
$ cd onpremise
$ ./install.sh

It will prompt for a user account and password in the midway. Please provide relevant details and continue. The script will pull all the images and build them properly. Next we run



docker-compose up -d

Aafter waiting for a bit if you head over to localhost:9000, you should see Sentry running and asking for some details

Then, let's create a project

We see Sentry supports a variety of languages and frameworks. Let's choose Rails, there's one for Ruby but the Rails one would ease our life much better.

The next page shows us getting started guide for how to integrate into Rails project. We will go through that in the next section. For now, just copy the DSN to the clipboard since that would be needed for us to connect to the Sentry server from Rails.

Ruby on Rails Setup

Preparation

We will need a few tools installed in the system to get started with Rails development

Ruby 2.7
Rails 6.0
Docker With these installed, let's generate our API only project ```bash

rails new sentry-rails \
--skip-action-mailbox \
--skip-action-text \
--skip-active-storage \
--skip-javascript \
--skip-spring -T \
--skip-turbolinks \
--skip-sprockets \
--skip-test \
--api

We are skipping most of the goodies needed for frontend development since we are going to write API only codes here.
Let's create a `.env` file at the root of the project and paste the `DSN` that we copied from the last section and a release number since with sentry we can track releases as well.
```bash


SENTRY_DSN=http://1d0405d68ff04d47bcc61b1e146dea6b@localhost:9000/2
RELEASE_NO=1.0.17

Replace it with the value you got from Sentry. Next, we install Sentry gem and dotenv gem to load these environment variables into Rails application



$ bundle add dotenv-rails
$ bundle add sentry-raven

Let's do one bundle install after that and we are set to start coding.

Basic Setup

So, we will start with a very basic setup and that's as simple as, putting the DSN in config



# config/application.rb
Bundler.require(*Rails.groups)
Dotenv::Railtie.load
Raven.configure do |config|
  config.dsn = ENV['SENTRY_DSN']
end

We make sure our environment variables are loaded and then we configure Raven which is all we need to do setup Sentry in Rails.

Let's configure routes to send requests



# config/routes.rb
Rails.application.routes.draw do
    scope :api do
        get '/track/:id', action: :track, controller: 'track'
    end
end

We scope the routes with api and then write a get route. Let's write the controller now.



# app/controllers/track_controller.rb
class TrackController < ApplicationController
    def track
        1 / 0
        render json: params[:id]
    end
end

This is obviously a runtime error. Let's run the app with rails s and hit the route with localhost:3000/api/track/1 curl or Postman.

When we click, we see

This has pinpointed the error to the exact location of the error. This is good. We have setup error tracking with Sentry.

Beyond The Basics

Handle Errors Asynchronously

Next, we improve our basic setup. With this setup, whenever an error occurs, it sends to Sentry right away. Which is not optimal for production and we do not need it. It's better to send them asynchronously to not hurt our main application performance. We can leverage job runners like Sidekiq, Resque but for this setup, we would use Active Job that's baked in Rails itself. Add a new job in jobs folder.



# app/jobs/sentry_job.rb
class SentryJob < ActiveJob::Base
    queue_as :sentry

    def perform(event)
        Raven.send_event(event)
    end
end

Then modify Raven config with



# config/application.rb
Raven.configure do |config|
  config.dsn = ENV['SENTRY_DSN']
  config.async = lambda { |event| SentryJob.perform_later(event) }
end

Now if Sentry catches an error, it would send it through Active job rather than sending synchronously.

Capture Exception Manually

While Sentry can catch errors automatically, there are handy methods to capture it ourselves. Let's explore that,



# app/controller/track_controller.rb
class TrackController < ApplicationController
    def track
        id = params[:id].to_i
        case id
            when 1
                1 / 0
            when 2
                Raven.capture do
                    1 / 0
                end
            when 3
                begin
                    1 / 0
                rescue ZeroDivisionError => exception
                    Raven.capture_exception(exception)
                end
            end
        render json: params[:id]
    end
end

We can wrap our codes in Raven.capture block or do begin..rescue block and catch exception with Raven.capture_exception, both works the same way and sends error to Sentry server.

Sending Logs

Sentry is not a traditional logger, we would explore logging in depth in another article, but it supports sending messages with extra fields if wanted. Let's change the controller action to see that



# app/controller/track_controller.rb
when 4
    Raven.capture_message('Something went very wrong', {
        extra: { 'key' => 'value' },
        backtrace: caller
    })
end

This captures the message and a hash as extra. We can also send backtrace with it for additional context. But we should not put it everywhere as logs, this should serve a specific purpose.

Additional Tracking Data

We are tracking a lot of data about the nature of the error and host machine by default in Sentry. But sometimes we want to add additional context so that it's easier for us to debug. Let's add a service class for this.



# app/services/sentry_service.rb
class SentryService
    attr_accessor :user_id, :_request, :_params, :error, :payload

    def initialize(user_id, request, params, error, payload = {})
        @user_id = user_id
        @_request = request
        @_params = params
        @error = error
        @payload = payload
    end

    def register
        Raven.user_context(user_context)
        Raven.tags_context(tags_context)
        Raven.extra_context(extra_context)
    end

    private

    def user_context
        return {} if @user_id.blank?

        { id: @user_id }
    end

    def tags_context
        {
            component: 'api',
            action: @_params[:action],
            controller: @_params[:controller],
            environment: Rails.env,
            error_class: error.class
        }
    end

    def extra_context
        extra = {
            params: @_params.to_enum.to_h.with_indifferent_access,
            url: @_request.try(:url),
            uuid: @_request.try(:uuid),
            ip: @_request.try(:ip),
            fullpath: @_request.try(:fullpath),
            error_message: error.message
        }

        @payload.each do |k, v|
            extra[k] = v.try(:as_json).try(:with_indifferent_access)
        end

        extra
    end
end

A lot is happening here, so let's go through this one by one.
We are initializing SentryService with user info, request info, parameters, error and any payload if passed. Next there are three context methods user_context, tags_context, extra_context.

user_context returns user_id. tags_context returns some values like the action, the controller, the environment and error class. This would help us group errors into tags in sentry later for better debugging. extra_context returns some values extracted from request and params. Extra context values would be displayed on the error page.

We then register these contexts in Raven inside the register method. Now we have to use this class. Let's modify ApplicationController class so that all errors are handled through this service



# app/controller/application_controller.rb
class ApplicationController < ActionController::API
    around_action :set_sentry_context

    protected

    # capture any unspecified internal errors
    def set_sentry_context
        begin
            yield
        rescue StandardError => e
            SentryService.new('12', request, params, e, context_variables).register
            raise e
        end
    end

    # override this in any other controllers to customize payload
    def context_variables
        {}
    end
end

All other controllers inherit from ApplicationController. So here we make a around_action method set_sentry_context which yields any controller method but captures the error thrown and wraps the error into SentryService with register call before rethrowing the error again which would then be caught by Sentry itself. We send dummy user_id but if you use a gem like devise, you should have the user's id in current_user. We also send request and params

Now let's test it. If we curl localhost:3000/api/track/1 again, we should see

We see that more tags are visible in error dashboard which would let us filter by tags. Also, we have tracked additional data along with the User ID

Track Release

It's very important to know from which version of the application the error is generating from. So Sentry has that configuration as well. Remember we kept our release number inside .env. Now update the config method like this



# config/application.rb
Raven.configure do |config|
    config.release = ENV['RELEASE_NO']
    config.dsn = ENV['SENTRY_DSN']
    config.async = lambda { |event| SentryJob.perform_later(event) }
end

Now, in the error dashboard, we should see which error is coming from which version of the application.

Conclusion

We have covered how we can track Rails errors in Sentry as well as how we can do a lot more with it. You can configure Sentry to send you alerts in various channels as well. Monitoring errors are a critical part of having a healthy application running. We would also explore how logging can help us find bugs in our application. Til then, keep coding!

Resources

Deploy a Ruby on Rails API only application in Docker with PostgreSQL

Rafi — Tue, 24 Nov 2020 19:36:53 +0000

Project Link: https://github.com/Joker666/rails-api-docker

Docker allows packaging an application or service with all its dependencies into a single image which then can be hosted into different platforms like Docker Hub or Github Container Registry. These images can pulled and shared with teammates for easy development or deployed to production with container orchestration tools like Kubernetes.

When we look at the current state of development, there are endless installation instructions on how to install and configure an application as well as all its dependencies. And even then it doesn't work, the Ruby version doesn't match or some dependency's version upgrade broke the installation. It is where Docker comes in handy, the image created once, can run in any platform as long as Docker is installed. Today we are going to deploy a Ruby on Rails application in Docker.

Preparation

We will need a few tools installed in the system to get started with Rails development

Ruby 2.7
Rails 6.0
Docker

With these installed, let's generate our API only project

rails new docker-rails \
  --database=postgresql \
  --skip-action-mailbox \
  --skip-action-text \
  --skip-spring -T \
  --skip-turbolinks \
  --api

Since we are creating API only project, we are skipping the installation of few Rails web specific tools like action-text or turbolinks. And we are making sure --api flag is there to create an API only Rails project.

Making APIs

We are going to make two APIs. Authors and articles

rails g resource Author name:string --no-test-framework
rails g resource Article title:string body:text author:references --no-test-framework

Let's add has_many macro in Author model

# app/models/author.rb
class Author < ApplicationRecord
    has_many :articles
end

And populate DB with some seed data. Install faker first

bundle add faker

Then do bundle install and then update seeds file with

# db/seeds.rb
require 'faker'

Author.delete_all
Article.delete_all

10.times do
    Author.create(name: Faker::Book.unique.author)
end

50.times do
    Article.create({
        title: Faker::Book.title,
        body: Faker::Lorem.paragraphs(number: rand(5..7)),
        author: Author.limit(1).order('RANDOM()').first # sql random
    })
end

Now, we do not have PostgreSQL running, so we cannot run the migrations or seed data. We would do that when we deploy in docker. Now lets update the controller files

# app/controllers/articles_controller.rb
class ArticlesController < ApplicationController
    before_action :find_article, only: :show
    def index
        @articles = Article.all
        render json: @articles
    end

    def show
        render json: @article
    end

    private
        def find_article
            @article = Article.find(params[:id])
        end
end

# app/controllers/author_controller.rb
class AuthorsController < ApplicationController
    before_action :find_author, only: :show
    def index
        @authors = Author.all
        render json: @authors
    end

    def show
        render json: @author
    end

    private
        def find_author
            @author = Author.find(params[:id])
        end
end

Let's prefix our routes with api

Rails.application.routes.draw do
    scope :api do
        resources :articles
        resources :authors
    end
end

We are all set now to hit some endpoints.

Enter Docker

To build a docker image we have to write a Dockerfile. What is a Dockerfile through? Dockerfile is where all the dependencies are bundled and additional commands or steps are written to be executed before building the image. There are built in images for Ruby. We will start with an image of Ruby 2.7. Let's write the Dockerfile first and then we will explain what is happening there.

FROM ruby:2.7

RUN apt-get update -qq && apt-get install -y postgresql-client
# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1

WORKDIR /app

COPY Gemfile Gemfile.lock ./
RUN bundle install

COPY . .

ENTRYPOINT ["./entrypoint.sh"]
EXPOSE 3000

# Start the main process.
CMD ["rails", "server", "-b", "0.0.0.0"]

So we start from Ruby 2.7 pre-built image and then install PostgreSQL client into the system, this is a Debian based system so we use apt-get. Next, we freeze the bundle config which will actually help us maintain consistency over the dockerized system and host system. If Gemfile was modified but we did not run bundle install, this is where it will throw an error

Now we set our working directory inside the docker system to be /app. We copy over the Gemfile and Gemfile.lock over to the docker's app directory and run bundle install inside docker. After bundle install finishes we copy over all the files from our host system to the docker system.

After that, we execute a shell script which we will come back shortly and then we expose port 3000 and start the rails server binding it to 0.0.0.0.

The entrypoint.sh file

#!/bin/bash

set -e

if [ -f tmp/pids/server.pid ]; then
    rm tmp/pids/server.pid
fi

exec "$@"

This helps fix a Rails-specific issue that prevents the server from restarting when a certain server.pid file pre-exists, this needs to be run on every docker start.

We are done with our docker file. Now let's build it.

docker build -t rails-docker .

This builds the image pulling all the dependencies and saves it with a tag rails-docker:latest

Now, we can run this image, but that won't necessarily help us since we need PostgreSQL running as well. We could use a local installation of the DB but here we will run the DB inside docker.

Environment Setup

Let's add .env file the root of the directory

DBHOST=localhost
DBUSER=postgres
DBPASS=password

This is essentially our DB environment variables which we will overwrite with docker.
Now, let's update the database.yml file inside config so that it the Rails app can read from the environment variables to connect to PostgreSQL.

# config/database.yml
default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

development:
  <<: *default
  database: docker_rails_development
  username: <%= ENV['DBUSER'] %>
  password: <%= ENV['DBPASS'] %>
  host: <%= ENV['DBHOST'] %>

Docker-Compose

Docker compose is a handy way to write all the docker images as services dependent on each other and running inside one internal network where they can talk to each other. We are going to run PostgreSQL as a service in docker-compose along with the image of Rails API we just built

version: '3.8'

services:
    web:
        build: .
        image: rails-docker
        restart: "no"
        environment:
            - DBHOST=postgresql
            - DBUSER=postgres
            - DBPASS=password
        ports:
            - 3000:3000
        depends_on:
            - postgresql

    postgresql:
        image: postgres
        restart: "no"
        ports:
            - 5432:5432
        environment:
            POSTGRES_DB: docker_rails_development
            POSTGRES_USER: postgres
            POSTGRES_PASSWORD: password
        volumes:
            - postgresdb:/var/lib/postgresql/data/

volumes:
    postgresdb:

So there are two services here. In the postgresql service, we are using the official postgresql image and passing some values for environment variables and exposing the internal 5432 port to the host machine. We add a docker volume with it so that it stores data there and data can survive a restart.

The web service, runs the image we just built for the API and depends on postgresql service. That means the postgresql service needs to be up and running first for web service to start running. This is cool. Since we specified the POSTGRES_DB environment in the postgresql service, if the database doesn't exist when running the PostGreSQL server for the first time, it will create the database. Great, now let's run the services.

docker-compose -f docker-compose.yml up --build

This will build the images first if they are not built already and then run them. We would see that both images are running in the console. Now let's do our migration and seeding.

Stop the services with ctrl+c and run

docker-compose run web rails db:migrate
docker-compose run web rails db:seed

This will run the rails commands from inside the web service container. We now have data populated.

Now let's run the services again with

docker-compose up

Let's hit some endpoints to check

curl localhost:3000/api/authors

[
   {
      "id":1,
      "name":"Lakendra Bergnaum",
      "created_at":"2020-11-24T13:25:29.507Z",
      "updated_at":"2020-11-24T13:25:29.507Z"
   },
   ...
]

Sure it fetches the authors from the API and prints in the console. And if hit the articles endpoint it would print the articles as well.

Conclusion

That was a lot to grasp if you are starting with docker for the first time. But we covered how we can deploy a Rails API only app in docker along with PostgreSQL. This is a good starting point to build something awesome.

Resouce

The Complete Microservice Tutorial — [Part 1] Building User Service With GRPC, Node.JS, and MongoDB

Rafi — Fri, 20 Nov 2020 17:19:45 +0000

Project Link: https://github.com/Joker666/microservice-demo/

The Complete Microservice Tutorial — Part 0

Microservices are about decoupling your system. In a monolith, you build all the components of the software in one large codebase and deploy the software at once. But in the microservice world, we build each component of a large system decoupled from each other.

In Part - 0, we explored the project’s architecture and now we are going to build it. We are going to build the authentication and user service module of the application. We will use NodeJS for this logic and MongoDB for the data layer.

Prerequisites

Since we are going to build this service in NodeJS, you would need NodeJS and NPM installed. Also, we would need GRPC installed along with Protobuf for the transport layer.

NodeJS with NPM
ProtoBuf
GRPC
Docker

Essentially, we will build a GRPC server in NodeJS that accepts incoming RPC requests. We would need GRPC tools installed globally in NPM, so let’s do that first.

npm install -g grpc-tools

Proto

Let’s make a new directory Microservice-Demo and cd into it. The directory structure we will follow

MicroService-Demo
├── userService
│   ├── proto
│   │   ├── **/*.js
│   ├── node_modules
│   ├── api.js
│   ├── auth.js
|   ├── .env
|   ├── Dockerfile
│   ├── index.js
|   ├── package.json
│   └── testClient.js
├── protos
│   ├── user
│   │   ├── user.proto
|   docker-compose.yml

We are going to keep all our proto files outside of our NodeJS application so that it’s easier for us to use those in other services. If you are wondering what is a proto file, it is a new format introduced by Google to serialize data for API usage that needs to be compiled with protoc compiler. The compiler outputs the language generated files in the desired language and GRPC uses them to communicate between services. So let’s see the user.proto file.

// protos/user/user.proto

syntax = "proto3";

package demo_user;

option go_package = "github.com/Joker666/microservice-demo/protos/user";

service UserSvc {
    rpc register (RegisterRequest) returns (UserResponse);
    rpc login (LoginRequest) returns (UserResponse);
    rpc verify (VerifyRequest) returns (VerifyResponse);
    rpc getUser (GetUserRequest) returns (VerifyResponse);
}

message VerifyRequest {
    string token = 1;
}

message GetUserRequest {
    string user_id = 1;
}

message LoginRequest {
    string email = 1;
    string password = 2;
}

message RegisterRequest {
    string name = 1;
    string email = 2;
    string password = 3;
}

message UserResponse {
    string id = 1;
    string name = 2;
    string email = 3;
    string token = 4;
}

message VerifyResponse {
    string id = 1;
    string name = 2;
    string email = 3;
}

The proto file is using proto3 syntax. We see that there are a couple of messages in this file representing request and response data. Then there is a service UserSvc defined that has four methods that leverage these messages. Essentially, these are four APIs that we would be building today. There is a way to load the proto file’s definition in runtime without compiling the file, but we are going to compile the file here because that would make our life much easier when we build other services. Let’s compile this proto file and store the results in userService/proto directory. Run the next command from the root Microservice-Demo directory.

grpc_tools_node_protoc \
    --js_out=import_style=commonjs,binary:userService/proto/ \
    --grpc_out=grpc_js:userService/proto \
    --proto_path=./protos/user ./protos/user/*.proto

Running this command will output two files in userService/proto directory, one is user.pb.js and another user_grpc.pb.js . We would need to require them in code next to build our APIs.

Building the service

Sowe have some APIs we are going to build, let’s start with user registration. We would install bcrypt for password hashing and jsonwebtoken for generating a JWT token for authentication.

// userService/index.js

require('dotenv').config();
const grpc = require('@grpc/grpc-js');
const { MongoClient } = require("mongodb");
const services = require('./proto/user_grpc_pb');
const API = require("./api");

// Mongo Connection
const dbClient = new MongoClient(process.env.DB_URI, { useUnifiedTopology: true });
let api = null;

async function connectDB() {
    try {
        await dbClient.connect();
        let db = await dbClient.db(process.env.DB_NAME);
        db.command({ ping: 1 });
        console.log("Connected successfully to mongo server");
        // Create index
        await db.collection("users").createIndex({ email: 1 });

        // Init api
        api = new API(db, grpc);
    } catch (e) {
        console.error(e);
    }
}

async function main() {
    await connectDB().catch(console.dir);
    let server = new grpc.Server();
    server.addService(services.UserSvcService, {
        register: api.register,
        login: api.login,
        verify: api.verify,
        getUser: api.getUser,
    });
    let address = process.env.HOST + ":" + process.env.PORT;
    server.bindAsync(address, grpc.ServerCredentials.createInsecure(), () => {
        server.start();
        console.log("Server running at " + address);
    });
}

main();

This is a very basic NodeJS setup. Here we are importing the generated user_grpc.pb.js file. That gives us access to UserSvcService that we defined earlier in the proto file. We initialize a new GRPC service and add our API methods to it as services. Next, we bind the address that we get from .env and start the server. There’s some boilerplate code to connect to MongoDB and pass the db and grpc instance to API class. Let’s code out API class.

// userService/api.js

const bcrypt = require('bcrypt');
const auth = require("./auth");
const messages = require('./proto/user_pb');
const ObjectId = require('mongodb').ObjectID;

module.exports = class API {
    constructor(db, grpc) {
        this.db = db;
        this.grpc = grpc;
    }

    register = (call, callback) => {
        const users = this.db.collection("users");

        bcrypt.hash(call.request.getPassword(), 10, (err, hash) => {
            let user = { name: call.request.getName(), email: call.request.getEmail(), password: hash }
            users.insertOne(user).then(r => {
                let resp = new messages.UserResponse();
                resp.setId(user._id.toString());
                resp.setName(user.name);
                resp.setEmail(user.email);
                resp.setToken(auth.generateToken(user));
                callback(null, resp);
            });
        });
    }

    // See the rest of the methods in
    // https://github.com/Joker666/microservice-demo/blob/main/userService/api.js
};

In the API class, we implement the register method. There are two parameters that have been passed to us by GRPC service definition, call and callback . The call parameter contains request information that we can access with call.get{ParamName} and callback is what gets returned from the method. It has two parameters, the first parameter takes error object and the second one response object.

We hash the password user has provided and then save the user to MongoDB. We then create the UserResponse message we made in the proto file earlier and set the necessary fields. The callback then returns the message. You can explore the token generation code here and the rest of the APIs of this service here. The full code is available here.

So we have coded our first API and now let's test it.

Docker Deploy

We have coded the application, now let’s write the Dockerfile to deploy it.

# userService/Dockerfile

FROM node:15

WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .

EXPOSE 50051
CMD [ "node", "index.js" ]

We are copying everything from the service directory and installing the packages here. Since we would also need MongoDB, running only this in docker would not be enough. Let’s write the docker-compose.yml file.

# docker-compose.yml

version: '3.8'

services:
    user:
        build:
            context: ./userService
        image: microservice/demo/user
        restart: "no"
        environment:
            - DB_URI=mongodb://mongo:27017/
            - DB_NAME=Microservice-demo-user
        ports:
            - 8080:50051
        depends_on:
            - mongo

    mongo:
        image: mongo
        restart: always
        environment:
            MONGO_INITDB_DATABASE: Microservice-demo-user
        ports:
            - 27017:27017
        volumes:
            - mongodb:/data/db
            - mongodb_config:/data/configdb

volumes:
    postgresdb:
    mysqldb:
    mongodb:
    mongodb_config:

Let’s run this with docker-compose.yml up --build . We should see both MongoDB and our service is running successfully.

Testing

Since we have written a GRPC service, we cannot test it directly with any tool like Postman, well not yet. There are some tools out there that somewhat ease the process like BloomRPC but I like to test the service with real code.

So, we have a server and now we have to write a client to test it.

// userService/testClient.js

const messages = require('./proto/user_pb');
const services = require('./proto/user_grpc_pb');
const grpc = require('@grpc/grpc-js');

function main() {
    const client = new services.UserSvcClient('localhost:8080', grpc.credentials.createInsecure());

    let registerReq = new messages.RegisterRequest();
    registerReq.setName("Hello");
    registerReq.setEmail("hello@world.com");
    registerReq.setPassword("Password");
    client.register(registerReq, function(err, response) {
        console.log(response);
    });
}

main();

Here, we are importing the message and service files and creating a client by connecting to port 8080 since we port-forwarded it in the docker-compose file. When we run this client with node testClient.js we would see that the user is being registered and a new user entry getting created in MongoDB. It should print in the console the response that contains the created user information.

Whoa! That was a lot. But now we have a full functioning microservice written in NodeJS that is running a GRPC server that can accept incoming RPC requests and interact with the database.

Conclusion

Here we have explored user registration/authentication, in the next article, we will build the project service with Python and MySQL. Til then, stay tuned.

Resources

The Complete Microservice Tutorial with GRPC, NodeJS, Go, Python & Ruby — [Part 0] Introduction

Rafi — Mon, 16 Nov 2020 18:55:52 +0000

Project Link: https://github.com/Joker666/microservice-demo/

Part 1 - Building User Service with GRPC, Node.JS, and MongoDB

If you have not been living under the rock for a while, microservice is the defacto architecture to make large scale applications these days. A lot of companies have moved from monolithic architecture to microservice-based architecture like Uber, Netflix, Etsy, and so on. Companies adopt this structure as it simplifies their development and reduces complexity.

This is a 6 part tutorial where we will build a complete microservices-based application from scratch. You can see what has been built already in the project link.

Motivation

I have been writing microservices for a while. But I remember having little to no resource on the web for how to stick all the different parts together. I had to struggle a lot and find solutions with trial and error. It was a daunting experience. I wish the reduce some pain for the readers of this blog.

This is going to be a multipart series where I structure each article with specific technology and different service. This is going to be an introductory article that will demonstrate what we are going to build, the required tools and knowledge, and how to set up the environment for you to get started.

Architecture

I won’t go in-depth about what is a microservice and how big should it be. It is an organizational choice to create small/large services. We will architect the services with different programming languages for demonstration, note that it is not required in practice.

We are going to build the world’s simplest task management software. A user can register, create projects/tags, add tasks to the projects and tag the tasks into categories. So we have divided the responsibilities into 3 services. We are also going to make an API Gateway service that routes outside API calls to microservices.

╔═════════════════╦═════════════════╦══════════════════════════════╗
║     Service     ║  Technologies   ║         Description          ║
╠═════════════════╬═════════════════╬══════════════════════════════╣
║ User Service    ║ NodeJS, MongoDB ║ Authentication service       ║
║ Project Service ║ Python, MySQL   ║ Project/Tag creation service ║
║ Task Service    ║ Ruby, PosgreSQL ║ Task assignment service      ║
║ API Service     ║ Go              ║ API Gateway service          ║
╚═════════════════╩═════════════════╩══════════════════════════════╝

Tools

We are going to use some cloud-native tools to bootstrap our project. The most important being the choice of the transport layer. While JSON based APIs were used to communicate within the services, now RPC based frameworks are more popular. We are going to use GRPC. There are resources online why this is the better approach, I will link some below. The main ones being, reduced latency between service calls, and it is built on language-agnostic Protocol Buffers. The main tools are

Docker
Protobuf
GRPC

We will also need different programming languages and platforms installed in the system for developing each service

NodeJS and NPM
Python 3 and Pipenv
Ruby and Bundler
Golang

The built application is tested Unix based environments, in windows, you can use wsl2 for running the services but that has not been tested yet.

Getting started on some of the tools can be daunting if you are not already familiar with it like protocol buffers and GRPC or docker. I encourage you to go through some online resources first and get a basic understanding of what these tools do. You do not have to be a veteran to get started, just basic knowledge would do. I plan to write more about these tools. If I do, I will backlink them here.

Approach

There are many ways you can write microservices. The approach I will take will be a simple approach for beginners. Generally, microservices are small pieces of a large software that is glued together and it makes the whole application at the end. The services when deployed either independently or in a cluster, usually communicate among them in a private network that cannot be intercepted from outside or even exposed to the public. There is usually an API Gateway that is written which sits as the router for public API calls that routes them to relevant service or services and returns the response.

There are many open-source API gateways, but here we will write our own since the added complexity of these API gateways are not suitable for beginners I believe. Also not all API gateways offer JSON to RPC transcoding so that it can accept HTTP 1.0 requests, our one will.

Extension

While not the main target, I plan to add more tools to the project and write about it. Tools for

Tracing
Monitoring
Logging

which are the pillars of observability.

Conclusion

I will publish the next article, which gets started on building the first service with NodeJS and MongoDB very soon. Till then stay tuned.

Part 1 - Building User Service with GRPC, Node.JS, and MongoDB